Information processing device, information processing method, and program recording medium

Abstract

To provide an information processing device that prevents needless analysis from being performed based on an estimation result with a low reliability.SOLUTION: An information processing device includes: a moving image acquisition unit that obtains a moving image which is obtained by capturing the image of an exercise of an object person; an estimation unit that estimates the exercise of the object person from the moving image which is obtained by the moving image acquisition unit; a reliability determination unit that determines the reliability of the exercise of the object person which is estimated by the estimation unit; and a notification unit that notifies, to the object person, an improvement of an image capturing environment where the image of the exercise is captured, according to the reliability determined by the reliability determination unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program recording medium. [Background technology]

[0002] Videos of subjects' movements are analyzed. In the device described in Patent Document 1, in a video recording of a golf swing performed by a subject, the frame rate is maintained in an analysis section in which the swing motion is analyzed, and the frame rate is reduced in a non-analysis section in which the swing motion is not analyzed (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-118501 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, since the reliability of the analysis result obtained based on the acquired video is not judged, for example, the accuracy of the analysis result may be insufficient due to the influence of the shooting environment of the video. For example, when analyzing the skeletal information of a subject based on the acquired video and determining the evaluation of a swing based on the skeletal information, the accuracy of the skeletal information may be insufficient compared to the intention due to the influence of the shooting environment of the video, and an erroneous result may be output as the evaluation. [Means for solving the problem]

[0005] In order to solve the above problem, one aspect is an information processing device that includes a video acquisition unit that acquires a video of a subject's movements, an estimation unit that estimates the subject's movements from the video acquired by the video acquisition unit, a reliability determination unit that determines the reliability of the subject's movements estimated by the estimation unit, and a notification unit that notifies the subject of improvements to the shooting environment in which the movements are filmed in accordance with the reliability determined by the reliability determination unit.

[0006] In order to solve the above problem, one aspect of the present invention is an information processing method in which an acquisition unit acquires a video capturing a subject's movement, an estimation unit estimates the subject's movement from the video captured by the acquisition unit, a reliability determination unit determines a reliability of the subject's movement estimated by the estimation unit, and a notification unit notifies the subject of an improvement to the shooting environment in which the movement is captured according to the reliability determined by the reliability determination unit.

[0007] In order to solve the above problem, one aspect is a program recording medium that records a program, the program causing a computer to realize an acquisition function that acquires a video of a subject's exercise, an estimation function that estimates the subject's exercise from the video acquired by the acquisition function, a reliability determination function that determines the reliability of the subject's exercise estimated by the estimation function, and a notification function that notifies the subject of improvements to the shooting environment in which the exercise is filmed based on the reliability determined by the reliability determination function. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating a schematic configuration example of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of functional blocks of the information processing device according to the embodiment. [Figure 3A] FIG. 4 is a diagram showing an example of the configuration of first pattern information according to the embodiment. [Figure 3B] FIG. 11 is a diagram showing an example of the configuration of second pattern information according to the embodiment. [Figure 3C] FIG. 11 is a diagram showing an example of the configuration of third pattern information according to the embodiment. [Figure 3D] FIG. 13 is a diagram showing an example of the configuration of fourth pattern information according to the embodiment. [Figure 4] FIG. 13 is a diagram illustrating an example of a reliability table according to the embodiment. [Diagram 5] FIG. 4 is a diagram showing an example of a procedure of a process performed by the information processing device according to the embodiment. [Figure 6A] FIG. 2 is a diagram showing an example of frame a1 of a video with a high frame rate according to an embodiment. [Figure 6B] FIG. 13 is a diagram showing an example of frame a2 of a video with a high frame rate according to the embodiment. [Figure 6C] FIG. 13 is a diagram showing an example of frame a3 of a video with a high frame rate according to the embodiment. [Figure 6D] FIG. 13 is a diagram showing an example of frame a4 of a video with a high frame rate according to an embodiment. [Figure 6E] FIG. 13 is a diagram showing an example of frame a5 of a video with a high frame rate according to the embodiment. [Figure 6F] FIG. 13 is a diagram showing an example of frame a6 of a video with a high frame rate according to the embodiment. [Figure 6G] FIG. 13 is a diagram showing an example of frame b1 of a video with a low frame rate according to the embodiment. [Figure 6H] FIG. 13 is a diagram showing an example of frame b2 of a video with a low frame rate according to an embodiment. [Figure 6I] FIG. 13 is a diagram showing an example of frame b3 of a video with a low frame rate according to the embodiment. [Figure 7A] FIG. 4 is a diagram showing an example of first moving image information having a high resolution according to the embodiment. [Figure 7B] 11 is a diagram showing an example of second moving image information having a low resolution according to the embodiment. FIG. [Figure 8A] FIG. 17 is a diagram showing an example of eleventh moving image information in which the brightness is not constant outdoors on a sunny day according to the embodiment. [Figure 8B]FIG. 13 is a diagram showing an example of twelfth moving image information in which the brightness is not constant outdoors on a cloudy day according to the embodiment. [Figure 9] FIG. 13 is a diagram showing an example of 21st moving image information taken outdoors on a rainy day according to an embodiment. [Figure 10A] FIG. 13 is a diagram showing an example of 31st moving image information outdoors in the morning according to the embodiment. [Figure 10B] FIG. 13 is a diagram showing an example of 32nd moving image information taken outdoors during the day in an embodiment. [Figure 10C] A figure showing an example of 33rd moving image information taken outdoors in the evening according to the embodiment. [Figure 11A] FIG. 13 is a diagram showing an example of the 41st moving image information in which the subject's clothing and the background are the same color in the embodiment. [Figure 11B] A figure showing an example of 42nd moving image information in which the subject's clothing and the background are different colors in an embodiment. [Figure 12] FIG. 13 is a diagram showing an example of the 51st moving image information in which a person other than the subject is captured in the image according to the embodiment. [Figure 13A] FIG. 13 is a diagram showing an example of 61st moving image information manually captured by a person according to the embodiment. [Figure 13B] A figure showing an example of 62nd video information shot using a tripod according to the embodiment. [Figure 14] FIG. 13 is a diagram showing an example of a display using a human skeleton according to the embodiment. [Figure 15] FIG. 11 is a diagram illustrating an example of a process during learning according to the embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of analysis content in a model under learning according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments will be described with reference to the drawings.

[0010] FIG. 1 is a diagram showing a schematic configuration example of an information processing system 1 according to an embodiment. The information processing system 1 includes an information processing device 11, an imaging device 21, a sensor unit 22, and a display device 31. 1 also shows a first subject B1. The first subject B1 is an example of a subject. The first subject B1 is, for example, a human being. In this embodiment, the first subject B1 is not included in the information processing system 1.

[0011] The photographing device 21 photographs the first subject B1 as a moving image. As a result, the photographing device 21 acquires a moving image in which the first subject B1 appears. The first subject B1 becomes a subject of the moving image. Here, the photographing device 21 is configured using, for example, a camera having a function of photographing a moving image. As an example, a camera such as a smartphone carried by the first subject B1 or another person may be used as the photographing device 21.

[0012] In this embodiment, moving images are treated as information in a predetermined format and processed in the information processing system 1. In this embodiment, the information is also called image information. Furthermore, in this embodiment, the information processing device 11 processes information such as moving images in the form of digital data. However, as another example, a part or all of the processing target may be in the form of an analog signal.

[0013] The sensor unit 22 includes one or more sensors that detect a predetermined index value. The predetermined index value may be, for example, an index value related to the movement of the first subject B1, or may be an index value related to another event. Here, some or all of the sensors may be attached to the body of the first subject B1. In this embodiment, the sensor unit 22 is used during learning of the analysis model, but may be used during other processing.

[0014] The information processing device 11 receives video information acquired by the image capturing device 21 as necessary. The information processing device 11 receives as input the detection results from one or more sensors included in the sensor unit 22 as necessary. The information processing device 11 outputs information to be displayed to the display device 31 as necessary.

[0015] In this embodiment, the information processing device 11 learns an analysis model based on video information input from the imaging device 21 and detection results input from the sensor unit 22. The analysis model is a machine learning model that performs calculations related to the movement of the first subject B1. The information processing device 11 performs an analysis of the movement of the first subject B1 based on the video information input from the imaging device 21 and the trained analysis model. At this time, for example, a detection result input from the sensor unit 22 may be used in the analysis.

[0016] The display device 31 has a screen and outputs and displays information on the screen. In this embodiment, the display device 31 outputs and displays the information to be displayed that is input from the information processing device 11 on the screen.

[0017] Here, in this embodiment, the photographing device 21, the sensor unit 22 and the display device 31 are provided as separate devices from the information processing device 11. However, as another example, some or all of the functions of the photographing device 21, the sensor unit 22 and the display device 31 may be integrated into the information processing device 11.

[0018] FIG. 2 is a diagram showing an example of the functional block configuration of the information processing device 11 according to the embodiment. In this embodiment, the information processing device 11 is configured using a computer. The information processing device 11 is, for example, a desktop computer, but may be, for example, a notebook personal computer, a tablet terminal, a smartphone, or a computer of another type. For example, the information processing device 11 may be a device that is fixedly installed at a specified location such as a sports facility and processes video of subjects using the facility, or a smartphone or the like that can be carried by the user.

[0019] The information processing device 11 includes an input unit 111 , an output unit 112 , a communication unit 113 , a storage unit 114 , and a control unit 115 . The control unit 115 includes an acquisition unit 151 , an estimation unit 152 , a reliability determination unit 153 , an analysis unit 154 , a notification unit 155 , an output control unit 156 , and a learning control unit 157 . The acquisition unit 151 includes a video acquisition unit 171 .

[0020] The input unit 111 performs a process of inputting information. The input unit 111 performs, for example, a process of inputting information in response to a user's operation, or a process of inputting information from another device. 1, the other devices are an image capturing device 21 and a sensor unit 22. In addition, the other devices may be, for example, a portable recording device. The input unit 111 may include an operation unit. The operation unit receives the contents of an operation performed by a user and performs a process of inputting information in accordance with the user's operation. The operation unit may be, for example, a keyboard and a mouse. It should be noted that the user of the information processing device 11 may be, for example, the first subject B1, or may be another person.

[0021] The output unit 112 performs a process of outputting information. The output unit 112 performs, for example, a process of outputting information to a user, or a process of outputting information to another device. 1, the other device is a display device 31. The other device may also be, for example, a portable recording device. As a mode for outputting information, for example, a mode for outputting sound such as voice may be used together with the display output or independently of the display output.

[0022] The communication unit 113 communicates with other devices. In this embodiment, the communication unit 113 is shown separately from the input unit 111 and the output unit 112 . However, for example, the functions of the communication unit 113 may be included in the functions of the input unit 111 and the output unit 112 .

[0023] The storage unit 114 stores information. In this embodiment, the memory unit 114 may store any information, such as a program that defines the processing, parameters used in the processing, a machine learning model, information before processing, information during processing, or information after processing. The information may be in any format, and in this embodiment includes video. The video may also be called a moving image.

[0024] In this embodiment, the storage unit 114 stores an estimation model A1, a reliability determination model A2, video information C1, and subject information C2. The video information C1 includes one or more pieces of video information. The subject information C2 includes information about the first subject B1.

[0025] Here, in this embodiment, for convenience of explanation, the estimation model A1 is one machine learning model, but as another example, a configuration may be used in which multiple different estimation models are prepared and the estimation unit 152 selects an appropriate estimation model from among these multiple estimation models. In addition, in this embodiment, for convenience of explanation, the reliability determination model A2 is one machine learning model.

[0026] The control unit 115 performs various types of processing and control. The control unit 115 has a processor such as a CPU (Central Processing Unit), and performs various types of processing and control by executing a predetermined program using the processor. The program may be stored in the storage unit 114, for example.

[0027] The acquisition unit 151 acquires information. In the present embodiment, the acquisition unit 151 may acquire, for example, one or more of information input by the input unit 111, information received by the communication unit 113, or information stored in the storage unit 114. The video acquisition unit 171 acquires a video. In this embodiment, the video is treated as video information. The acquisition unit 151 may acquire, for example, imaging environment information that is information about the imaging environment of a video during learning. In this embodiment, the image capturing environment information includes video characteristic information and environmental status information.

[0028] In this embodiment, for the sake of convenience of explanation, the function of the video acquisition unit 171 possessed by the acquisition unit 151 is shown, but for example, this function may be inseparably integrated as the acquisition unit 151, and it may be considered that the function of the acquisition unit 151 acquires various information required.

[0029] The estimation unit 152 uses the estimation model A1 to estimate the motion of a first subject B1 appearing in a video. In the present embodiment, the estimation unit 152 may estimate the position of a joint of the first subject B1. Note that the movement of the position is estimated by acquiring the change in the position of the same joint in time series. The reliability determination unit 153 determines the reliability of the estimation result by the estimation unit 152 using the reliability determination model A2.

[0030] 2, the estimation model A1 and the reliability judgment model A2 are shown as separate models, but for example, the estimation model A1 and the reliability judgment model A2 may be realized by a common machine learning model. For example, the estimation model A1 and the reliability judgment model A2 may be configured as an integrated model, and may be configured as a two-class model or a model with three or more classes. The model may be called, for example, a class classifier. In this case, the function of the estimation unit 152 and the function of the reliability determination unit 153 may be regarded as being integrated.

[0031] As an example, when the estimation model A1 and the reliability determination model A2 are configured as an integrated two-class model, the two-class model calculates the probability that the estimation result is the first class and the probability that the estimation result is the second class for the input video information. In this case, the probability of the first class can be used as the reliability that the estimation result is the first class, and similarly, the probability of the second class can be used as the reliability that the estimation result is the second class. Note that each probability is expressed, for example, as a value between 0 and 1, and the sum of the probability of the first class and the probability of the second class is 1. Also, the higher the probability, the higher the reliability is considered to be. As a specific example, it may be determined that the confidence that the estimation result is the first class is higher when the probability of the first class is 0.9 and the probability of the second class is 0.1 than when the probability of the first class is 0.6 and the probability of the second class is 0.4. Similarly, it may be determined that the confidence that the estimation result is the class with the higher probability is higher when the difference between the probabilities of the respective classes is large.

[0032] The analysis unit 154 performs analysis using the estimation result by the estimation unit 152 according to the reliability determined by the reliability determination unit 152. In this embodiment, the analysis unit 154 analyzes the movement of the first subject B1 appearing in the video. In this analysis, the movement of the first subject B1 during the movement is analyzed. Furthermore, the analysis unit 154 may perform the analysis using, for example, the estimation result by the estimation unit 152, and either or both of information on the shooting environment of the video and information on the first subject B1.

[0033] Here, various analyses may be performed as the analysis of the movement, for example, analysis of information on one or more joints in the body of the first subject B1 may be performed. As the information on the joint, for example, information on one or more of the position of the joint, a predetermined angle of the joint, the movement of the joint, the moving speed of the joint, or the rotation speed of the joint may be used. The analysis of the motion may involve, for example, identifying a person corresponding to the subject to be analyzed from a video and extracting a number of feature points representing the posture of the person. The feature points may include, for example, joints. Furthermore, the analysis of the movement may involve, for example, analysis of information relating to the skeleton of the body of the first subject B1.

[0034] In addition, the analysis unit 154 has a function of performing further analysis based on the estimation result obtained by the estimation model A1, but for example, there may be cases where the estimation result obtained by the estimation model A1 is used as the analysis result as it is. In this embodiment, for example, the reliability of the estimation result by the estimation unit 152 and the reliability of the analysis result by the analysis unit 154 that performs analysis using the estimation result may be correlated with each other.

[0035] The notification unit 155 makes suggestions for improving the shooting environment. The notification unit 155 may make a suggestion for improving the imaging environment by, for example, outputting information proposing improvement of the imaging environment from the output unit 112 via the output control unit 156. Here, as the proposal for the shooting environment, for example, when the shooting environment is determined to be bad, a proposal for improving the shooting environment is used.

[0036] The output control unit 156 controls the information output by the output unit 112 . In this embodiment, the output control unit 156 controls the process of displaying information about the analysis result by the analysis unit 154 on the screen of the display device 31 via the output unit 112. In addition, in this embodiment, the output control unit 156 controls the process of displaying information by the notification unit 155 suggesting improvements to the shooting environment on the screen of the display device 31 via the output unit 112.

[0037] The learning control unit 157 controls the learning process of the estimation model A1. The learning process is a machine learning process. Furthermore, the learning control unit 157 controls the learning process of the reliability determination model A2. The learning process is a machine learning process. When a machine learning model in which the estimation model A1 and the reliability determination model A2 are integrated is used, the learning control unit 157 controls the learning process of the machine learning model.

[0038] In this embodiment, the learning control unit 157 controls the learning process of the estimation model A1 and the learning process of the reliability determination model A2, for example, based on a video acquired by the imaging device 21 and a detection result acquired by the sensor unit 22. In this case, for example, information on the video may be used as input information for a machine learning model, and information based on the detection result may be used as correct answer data in the learning process of the machine learning model. The correct answer data may be called, for example, teacher data.

[0039] Four different examples of pattern information are shown using FIG. 3A to FIG. 3D. FIG. 3A is a diagram showing an example of the configuration of the first pattern information T1 according to the embodiment. The first pattern information T1 includes photographing environment information and subject information. The shooting environment information includes video characteristic information and shooting situation information. The names of each piece of information are given for the convenience of explanation, and may be called by other names. In the present embodiment, the information on the image capturing environment and the subject information C2 correspond to the image capturing environment information and the subject information in the first pattern information T1.

[0040] The first pattern information T1 includes, as moving image characteristic information, an evaluation value of the frame rate and an evaluation value of the resolution. Here, the evaluation value of the frame rate may be, for example, high, normal, or low. In the example of Fig. 3A, the frame rate is managed as high. For example, a frame rate less than 30 fps is managed as low, a frame rate between 30 fps and 60 fps is managed as normal, and a frame rate higher than 60 fps is managed as high. Similarly, the evaluation value of the resolution may be, for example, high, normal, or low. In the example of Fig. 3A, the resolution is managed as high. For example, a resolution less than 720 is managed as low resolution, a resolution equal to or greater than 720 and less than 1080 is managed as normal resolution, and a resolution equal to or greater than 1080 is managed as high resolution. As another example, the frame rate may be expressed as a numerical value, and similarly, the resolution may be expressed as a numerical value.

[0041] The first pattern information T1 includes, as shooting situation information, information indicating whether the shooting is indoors or outdoors, information indicating the weather, information indicating the time, information indicating clothing, information indicating reflections, and information indicating blur.

[0042] The information indicating whether the location is indoors or outdoors can be either "indoors" or "outdoors." In the example of FIG. 3A, the location is managed as being indoors.

[0043] The information representing the weather is, for example, information such as sunny, cloudy, rainy, etc. In addition, when the user is indoors and is not affected by the weather, the information representing the weather may be information representing that the user is indoors and therefore does not have any influence. In the example of Fig. 3A, the weather outdoors is rainy, but the user is indoors and therefore does not have any influence.

[0044] The information representing the time is, for example, information such as morning, daytime, and evening. In addition, in the case where the location is indoors and is not affected by time, the information representing the time may be information representing that the location is indoors and therefore irrelevant. For example, the period from sunrise to four hours after sunrise is considered to be morning, the period from four hours after sunrise to ten hours after sunrise is considered to be daytime, and the period from ten hours after sunrise to sunset is considered to be evening. In the example of FIG. 3A, although it is daytime, it is managed that the location is indoors and therefore irrelevant.

[0045] The information representing the clothes is, for example, information indicating whether the predetermined clothes of the subject are the same color as the background or a different color. In the example of Fig. 3A, it is managed that the predetermined clothes of the subject are a different color from the background.

[0046] The information representing the reflection is, for example, information indicating whether or not an object other than the subject is reflected. In the example of Fig. 3A, it is managed that there is no reflection. Here, the object other than the subject may be, for example, any object including a living being such as a human, or may be limited to a human by having the acquisition unit 151 or the like perform image recognition to determine whether an object appearing in the video is a human or not.

[0047] The information representing the blur is, for example, information indicating whether or not there is blur when shooting a moving image. In the example of Fig. 3A, it is managed that there is no blur.

[0048] The first pattern information T1 includes, as subject information, information indicative of the subject's age and information indicative of the subject's exercise experience.

[0049] The information indicating the age of the subject is, for example, numerical information indicating the age of the subject. In the example of Fig. 3A, the age of the subject is managed as 30 years old. The information indicating the subject's exercise experience is, for example, information indicating the subject's exercise experience in the form of a numerical value, etc. In the example of Fig. 3A, the subject's exercise experience is managed as 10 years.

[0050] FIG. 3B is a diagram showing an example of the configuration of the second pattern information T2 according to the embodiment. As the second pattern information T2, information for each item similar to that of the first pattern information T1 is stored.

[0051] In the example of FIG. 3B, the frame rate is managed as low. In the example of FIG. 3B, the resolution is managed as high. In the example of FIG. 3B, the information indicating whether the location is indoors or outdoors is managed as being outdoors. In the example of FIG. 3B, the information indicating the weather indicates that the weather outdoors is sunny. In the example of FIG. 3B, the information indicating the time is managed as being daytime. In the example of FIG. 3B, the information representing the clothing is managed such that a specific clothing item of the subject is a different color from the background. In the example of FIG. 3B, the information representing the reflection indicates that there is no reflection. In the example of FIG. 3B, the information representing the blurring is managed as "no blurring."

[0052] In the example of FIG. 3B, the information indicating the subject's age indicates that the subject is 40 years old. In the example of FIG. 3B, the information indicating the exercise experience of the subject is managed such that the subject has 5 years of exercise experience.

[0053] FIG. 3C is a diagram showing an example of the configuration of the third pattern information T3 according to the embodiment. As the third pattern information T3, information for each item similar to that of the first pattern information T1 is stored.

[0054] In the example of FIG. 3C, the frame rate is managed as high. In the example of FIG. 3C, the resolution is managed as low. In the example of FIG. 3C, the information indicating whether the location is indoors or outdoors is managed as being outdoors. In the example of FIG. 3C, the information indicating the weather indicates that the weather outdoors is sunny. In the example of FIG. 3C, the information indicating the time is managed as being daytime. In the example of FIG. 3C, the information representing the clothing is managed such that a specific clothing item of the subject is a different color from the background. In the example of FIG. 3C, the information representing the reflection indicates that there is no reflection. In the example of FIG. 3C, the information representing the blurring is managed as "no blurring."

[0055] In the example of FIG. 3C, the information indicating the subject's age indicates that the subject is 20 years old. In the example of Fig. 3C, the information representing the exercise experience of the subject is managed as "no experience in exercise." Note that "no experience" may be managed as "0 years of experience."

[0056] FIG. 3D is a diagram showing an example of the configuration of the fourth pattern information T4 according to the embodiment. As the fourth pattern information T4, information for each item similar to that of the first pattern information T1 is stored.

[0057] In the example of FIG. 3D, the frame rate is managed as low. In the example of FIG. 3D, the resolution is managed as low. In the example of FIG. 3D, the information indicating whether the location is indoors or outdoors is managed as being outdoors. In the example of FIG. 3D, the information representing the weather indicates that the weather outside is sunny. In the example of FIG. 3D, the information indicating the time is that it is morning. In the example of FIG. 3D, the information representing the clothing is managed such that the subject's specific clothing is the same color as the background. In the example of FIG. 3D, the information representing the reflection is managed as the fact that there is a reflection. In the example of FIG. 3D, the fact that there is blur is managed as information representing blur.

[0058] In the example of FIG. 3D, the information indicating the subject's age indicates that the subject is 50 years old. In the example of FIG. 3D, the information representing the exercise experience of the subject is managed such that the subject has one year of exercise experience.

[0059] Here, in this embodiment, the reliability determination unit 153 determines the reliability using the reliability determination model A2, which is a machine learning model. However, as another example, the reliability determination unit 153 may determine the reliability using a table or the like without using machine learning. In this case, a machine learning model is used as the estimation model A1, and a model using a table or the like is used as the reliability determination model A2 instead of the machine learning model.

[0060] FIG. 4 is a diagram showing an example of a reliability table T11 according to the embodiment. The reliability table T11 stores the correspondence between conditions and reliability. The condition is, for example, a condition related to an estimation result by the estimation unit 152 that inputs a moving image, and is specified based on the estimation result. The condition represents, for example, a condition related to a shooting environment of the moving image, and as a specific example, represents a condition related to a part or all of the shooting environment information. The condition may be, for example, a condition related to the shooting environment and other information. The other information may be, for example, information related to the first subject B1, or, as a specific example, a part or all of the subject information. In the example of Fig. 4, values ​​from 0% to 100% are used as the reliability value, and a lower value indicates a lower reliability, and a higher value indicates a higher reliability. 0% corresponds to a probability of 0, and 100% corresponds to a probability of 1. In the example of FIG. 4, "aaa" and the like written in the "Condition" column of the reliability table T11 represent the respective conditions, but the description of the specific contents of the conditions is omitted.

[0061] When the reliability table T11 shown in FIG. 4 is used, the reliability determination unit 153 determines the reliability according to the estimation result by the estimation unit 152 by referring to the reliability table T11. Here, the reliability table T11 may be stored in advance in, for example, the storage unit 114. Note that the contents of the reliability table T11 may be rewritable by, for example, a user at any time.

[0062] FIG. 5 is a diagram showing an example of a procedure of processing performed by the information processing device 11 according to the embodiment.

[0063] In step S1, the acquisition unit 151 acquires predetermined information in the information processing device 11. In this embodiment, the moving image acquisition unit 171 acquires moving image information. Then, the information processing device 11 proceeds to the process of step S2. In addition, when other information is used together with the video information, the acquisition unit 151 further acquires the other information. The other information is, for example, one or more pieces of information included in the shooting environment information and the subject information.

[0064] In step S2, in the information processing device 11, the estimation unit 152 estimates the motion of a subject appearing in the video of the acquired video information by using the estimation model A1. Here, as the estimation of the subject's motion, for example, estimation of the movement of the position of the subject's joints may be performed. Then, the information processing device 11 proceeds to the process of step S3.

[0065] In step S3, in the information processing device 11, the reliability determination unit 153 determines the reliability of the estimation result by the estimation model A1 using the reliability determination model A2. Then, the information processing device 11 proceeds to the process of step S4.

[0066] Here, for example, when a machine learning model in which the estimation model A1 and the reliability determination model A2 are integrated is used, the process of step S2 and the process of step S3 may be integrated.

[0067] In step S4, in the information processing device 11, the notification unit 155 determines whether or not to make a suggestion for improving the shooting environment based on the determined reliability. As a result of this determination, in the information processing device 11, if a proposal for improving the shooting environment is to be made, the result becomes YES in step S4, and the process proceeds to step S5. On the other hand, if the result of this determination is that the information processing device 11 does not make any suggestions for improving the shooting environment, the result becomes NO in step S4, and the process proceeds to step S6. The function of determining whether or not to make a suggestion for improving the shooting environment may be provided in the reliability determination unit 153, instead of the notification unit 155, for example.

[0068] Here, any method may be used as a method for determining whether or not to suggest improvements to the shooting environment based on the reliability. For example, a method may be used in which, if the reliability is below a predetermined threshold, it is determined to suggest improvements to the shooting environment, and, if the reliability is equal to or greater than the threshold, it is determined not to suggest improvements to the shooting environment.

[0069] In this embodiment, when the reliability is low, it is assumed that the reliability is low due to the poor shooting environment of the video estimated by the estimation model A1. In addition, in this embodiment, it is possible to specify what kind of poor shooting environment is causing the reliability to be low based on the estimation result of the estimation model A1, etc. Such specification is possible, for example, by machine learning performed in advance or information determined in advance.

[0070] In step S5, in the information processing device 11, the notification unit 155 performs control to output the contents of a proposal for improving the shooting environment. Then, the information processing device 11 ends the process of this flow.

[0071] In step S6, in the information processing device 11, the analysis unit 154 analyzes the movement of the first subject B1 based on the estimation result by the estimation unit 152. In addition, in the information processing device 11, the output control unit 156 controls the output of information related to the analysis result. Then, the information processing device 11 ends the process of this flow.

[0072] A specific example of the shooting environment will be given. In this embodiment, the captured video shows the motion of a specific exercise performed by the subject. The motion is not particularly limited, and may be, for example, a motion that has a set sequence of motion but may vary from person to person, such as a golf swing. In this embodiment, the moving image to be analyzed may be specified by, for example, a user of the information processing device 11, or may be automatically specified by the information processing device 11 or the like. In the golf course example, a round of golf is played outdoors, and facilities such as retail stores are located indoors. In the case of golf, the subject plays the game with a certain tool, which is a golf club.

[0073] Furthermore, the video to be analyzed may be, for example, a video based on multiple actions of a subject filmed in a similar shooting environment, or a video that is the result of averaging, etc., multiple similar actions performed by the same subject. As a specific example, when a subject performs a golf swing motion multiple times in a similar shooting environment, a video based on the results of these multiple swings may be used as an analysis target. Processing such as averaging the images may be performed by, for example, the acquisition unit 151 in the control unit 115.

[0074] In the analysis process according to the present embodiment, for example, information on a plurality of different parts, such as the positions or angles of a plurality of joints of a human, may be identified based on a video in which a specific action is captured. Note that the process of identifying a specific part from the video may utilize, for example, a known image recognition process. Here, the video includes, for example, time-series change data of a body part involved in at least one specific movement.

[0075] In this embodiment, when illustrating a video of a subject's exercise, the subject's body is illustrated, but the tools used by the subject are not illustrated. In this embodiment, the subject holds a golf club and performs a golf swing.

[0076] The frame rate of a moving image, which is an example of a shooting environment, will be described with reference to FIGS. 6A to 6F and 6G to 6I. In this example, the frame rate of the moving image shown in FIGS. 6A to 6F is higher than the frame rate of the moving image shown in FIGS. 6G to 6I.

[0077] 6A to 6F show an example of six consecutive frames that make up one video. Each of the frames shown in FIG. 6A to FIG. 6F, or a group of two or more frames, may be used as information to be displayed.

[0078] FIG. 6A is a diagram showing an example of a1-th frame Fa1 of a moving image with a high frame rate according to the embodiment. FIG. 6B is a diagram showing an example of a2-th frame Fa2 of a moving image with a high frame rate according to the embodiment. FIG. 6C is a diagram showing an example of a3-th frame Fa3 of a moving image with a high frame rate according to the embodiment. FIG. 6D is a diagram showing an example of a4th frame Fa4 of a moving image with a high frame rate according to the embodiment. FIG. 6E is a diagram showing an example of a5th frame Fa5 of a moving image with a high frame rate according to the embodiment. FIG. 6F is a diagram showing an example of a6th frame Fa6 of a moving image with a high frame rate according to the embodiment.

[0079] The a1 frame Fa1 to the a6 frame Fa6 capture the movement of the eleventh subject B11. The eleventh subject B11 is an example of a subject. In the example of Figures 6A to 6F, the movement of the position of a predetermined joint of the 11th subject B11 is shown as the a1 joint Ha1, the a2 joint Ha2, the a3 joint Ha3, the a4 joint Ha4, the a5 joint Ha5, and the a6 joint Ha6 in the order of Figures 6A to 6F. In this example, the a1 joint Ha1 to the a6 joint Ha6 represent the same joint, and Figures 6A to 6F represent the change in the position of the joint.

[0080] 6G to 6I show an example of three consecutive frames that make up one video. Each of the frames shown in FIGS. 6G to 6I, or a group of two or more frames, may be used as information to be displayed.

[0081] FIG. 6G is a diagram showing an example of the b1-th frame Fb1 of a moving image having a low frame rate according to the embodiment. FIG. 6H is a diagram showing an example of the b2-th frame Fb2 of a moving image with a low frame rate according to the embodiment. FIG. 6I is a diagram showing an example of the b3-th frame Fb3 of a moving image with a low frame rate according to the embodiment.

[0082] The b1 frame Fb1 to the b3 frame Fb3 capture the movement of the 21st subject B21. The 21st subject B21 is an example of a subject. In the example of Figures 6G to 6I, the movement of the position of a predetermined joint of the 21st subject B21 is shown as the b1 joint Hb1, the b2 joint Hb2, and the b3 joint Hb3 in the order of Figures 6G to 6I. In this example, the b1 joint Hb1 to the b3 joint Hb3 represent the same joint, and Figures 6G to 6I show the change in the position of that joint.

[0083] It should be noted that the a1 joint Ha1 to the a6 joint Ha6 in the example of FIGS. 6A to 6F and the b1 joint Hb1 to the b3 joint Hb3 in the example of FIGS. 6G to 6I are the same joints. In the example of Figure 6A and the example of Figure 6G, the subject is in a position where he is holding a golf club and is ready to go, and in the example of Figure 6D, the subject is in a position where he is swinging the golf club over his shoulder.

[0084] In this example, for a similar series of motions, such as a golf swing, a video consisting of six frames is acquired in the examples of Figures 6A to 6F, and a video consisting of three frames is acquired in the examples of Figures 6G to 6I. In other words, the frame rate of the video in the examples of Figures 6A to 6F is twice as high as the frame rate of the video in the examples of Figures 6G to 6I. For example, when motion analysis is performed from a video in a real environment, various types of filming equipment may be used. In general, the frame rate can have a significant effect on the performance of an inference device that processes a time series system.

[0085] Here, when the frame rate of the video information is high, it is suitable for processing in which analysis is performed by taking detailed analysis points, for example. In other words, in this case, it is possible to secure a sufficient number of frames for analysis, and if the analysis is performed without taking detailed points, for example, the analysis accuracy may decrease. On the other hand, when the frame rate of video information is low, it is suitable for processing to perform analysis without taking detailed analysis points, for example. In other words, in this case, compared to video information with a high frame rate, taking detailed analysis points may result in taking analysis points at overlapping positions, which may result in erroneous analysis results and reduced analysis accuracy.

[0086] For example, multiple machine learning models for analyzing the subject's motion may be prepared according to the frame rate of the video information. As a specific example, a machine learning model suitable for video information with a high frame rate and a machine learning model suitable for video information with a low frame rate may be prepared separately. Also, for example, when video information with a low frame rate is processed, the reliability may be determined to be lower than when video information with a high frame rate is processed. In this case, for example, a process may be performed to suggest improving the shooting environment so as to obtain video information with a higher frame rate. Whether the frame rate is high or low may be determined using a threshold value, for example. Furthermore, three or more classifications may be used as the level of the frame rate. As a specific example, three classifications of frame rate such as high, medium, and low may be used.

[0087] Here, the acquisition unit 151 may use the imaging environment acquisition function to determine the frame rate of the video based on, for example, information on the properties of the video, or may determine the frame rate of the video based on information about the video.

[0088] The resolution of a moving image, which is an example of a shooting environment, will be described with reference to FIGS. 7A and 7B. In this example, the resolution of the moving image shown in FIG. 7A is higher than the resolution of the moving image shown in FIG. 7B. In this example, for the sake of simplicity, each of Figures 7A and 7B shows a group of six consecutive frames of a moving image arranged horizontally as moving image information of the moving image. Each frame of the moving image shown in each of Figures 7A to 7B, or a group of two or more frames, may be used as information to be displayed.

[0089] FIG. 7A is a diagram showing an example of first moving image information G1 having a high resolution according to the embodiment. In the first video information G1, the exercise of the 31st subject B31 is captured. The 31st subject B31 is an example of a subject. FIG. 7A shows an example of a change in movement of the c-th joint Hc, which is a predetermined joint of the 31st subject B31.

[0090] FIG. 7B is a diagram showing an example of the second moving image information G2 having a low resolution according to the embodiment. In the second video information G2, the exercise of the 41st subject B41 is captured. The 41st subject B41 is an example of a subject. FIG. 7B shows an example of a change in movement of the d-th joint Hd, which is a predetermined joint of the 41st subject B41. It should be noted that the c-th joint Hc in the example of FIG. 7A and the d-th joint Hd in the example of FIG. 7B are the same joint.

[0091] Here, when the resolution of the video information is high, for example, it is suitable for processing to perform analysis by taking fine analysis points. In other words, in this case, since the edges of people appearing in the video can be clearly detected, it is possible to take fine analysis points, and for example, if analysis is performed without taking fine points, the analysis accuracy may decrease. On the other hand, when the resolution of video information is low, for example, it is suitable for processing to perform analysis without taking many analysis points. In other words, in this case, compared to video information with high resolution, the edges of people appearing in the video cannot be detected clearly, so if there are many analysis points, there is a possibility that non-human parts will be mistakenly determined to be human movements. For example, taking many analysis points may increase noise, which may reduce the accuracy of the analysis.

[0092] For example, multiple machine learning models for analyzing the subject's motion may be prepared according to the resolution of the video information. As a specific example, a machine learning model suitable for video information with high resolution and a machine learning model suitable for video information with low resolution may be prepared separately. Also, for example, when low-resolution video information is processed, the reliability may be determined to be lower than when high-resolution video information is processed. In this case, for example, a process may be performed to suggest improving the shooting environment so as to obtain video information with a higher resolution. Whether the resolution is high or low may be determined using a threshold value, for example. Furthermore, three or more divisions may be used as the degree of resolution. As a specific example, three divisions such as high, medium, and low resolution may be used.

[0093] Here, the acquisition unit 151 may determine the resolution of the video based on, for example, information on the properties of the video, or may determine the resolution of the video based on information about the video, using the imaging environment acquisition function.

[0094] With reference to FIG. 8A and FIG. 8B, a description will be given of whether a photographing location, which is an example of a photographing environment, is outdoors or indoors. In this example, a video shot indoors is, for example, similar to the video shown in FIG. 7A. In this example, for the sake of simplicity, each of Fig. 8A and Fig. 8B shows a group of six consecutive frames of a moving image arranged horizontally as moving image information of the moving image. Each frame of the moving image shown in each of Fig. 8A to Fig. 8B, or a group of two or more frames, may be used as information to be displayed.

[0095] FIG. 8A is a diagram showing an example of eleventh moving image information G11 in which the brightness is not constant outdoors on a sunny day according to the embodiment. In the eleventh video information G11, the exercise of the 51st subject B51 is captured. The 51st subject B51 is an example of a subject. FIG. 8A shows an example of a change in movement of a specific joint, the e-th joint He, of the 51st subject B51. In the example of FIG. 8A, the sun K is shown in each frame.

[0096] FIG. 8B is a diagram showing an example of the twelfth moving image information G12 in which the brightness is not constant outdoors on a cloudy day according to the embodiment. In the twelfth video information G12, the exercise of the 61st subject B61 is captured. The 61st subject B61 is an example of a subject. FIG. 8B shows an example of a change in movement of the f-th joint Hf, which is a predetermined joint of the 61st subject B61. It should be noted that the e-th joint He in the example of FIG. 8A and the f-th joint Hf in the example of FIG. 8B are the same joint.

[0097] Here, when a video is shot indoors, for example, the brightness is considered to be constant or nearly constant, and it is suitable for processing to perform analysis by taking detailed analysis points. In other words, in this case, for example, if analysis is performed without taking detailed points, the analysis accuracy may decrease. When the brightness during shooting is constant or nearly constant, for example, analysis accuracy can be ensured even if analysis is performed without adjusting the contrast or brightness of the image. On the other hand, when a video is shot outdoors, a shooting environment may occur in which the brightness is not constant throughout a series of videos due to, for example, the weather or the time of shooting. If the brightness during shooting is not constant, for example, it may be necessary to adjust the contrast or brightness of the image before performing analysis. The adjustment may also be called correction.

[0098] For example, multiple machine learning models for analyzing the subject's movement may be prepared depending on whether the shooting location is indoors or outdoors. As a specific example, a machine learning model suitable for video information shot indoors and a machine learning model suitable for video information shot indoors may be prepared separately. For example, multiple machine learning models for analyzing the subject's movement may be prepared depending on whether the brightness at the time of shooting is constant or not. As a specific example, a machine learning model suitable for video information shot under constant brightness and a machine learning model suitable for video information shot under non-constant brightness may be prepared separately.

[0099] Also, for example, when video information shot outdoors is processed, the reliability may be determined to be lower than when video information shot indoors is processed. Also, for example, when processing moving image information in which the brightness at the time of shooting is not constant, the reliability may be determined to be lower than when processing moving image information in which the brightness at the time of shooting is constant. In this case, for example, processing may be performed to suggest improvements to the shooting environment so as to obtain moving image information in which the brightness is more constant. Whether the brightness is constant or not may be determined, for example, by using an upper threshold value and a lower threshold value that define a range in which the brightness is considered to be constant. As a specific example, if the brightness that can vary falls within a predetermined range, it may be considered to be constant, and if it deviates from the predetermined range, it may be considered not to be constant.

[0100] Here, even if the photograph was taken indoors, if the lighting in the room is too bright, too dark, or flashing, for example, the reliability may be determined to be low compared to when the lighting in the room is appropriate. As a specific example, if the shooting location is indoors and the lighting in the room is bright, causing the image to be blown out, the reliability may be determined to be low and a suggestion may be made to improve the shooting environment by lowering the brightness when shooting. As another specific example, if the shooting location is indoors and the lighting in the room is dim, making it difficult to see the subject, etc., the reliability may be determined to be low and a suggestion may be made to improve the shooting environment, such as increasing the brightness when taking the photo.

[0101] With reference to FIG. 9, a description will be given of weather, which is one example of an image capturing environment, when an image capturing location is outdoors, which is one example of an image capturing environment. In this example, when the shooting location is outdoors, a video shot on a sunny day and a video shot on a cloudy day are similar to the video shown in FIG. 8A and the video shown in FIG. 8B, respectively. In this example, for the sake of simplicity, a group of six consecutive frame images in a moving image is shown as the moving image information of the moving image in Fig. 9. Each frame in the moving image shown in Fig. 9, or a group of two or more frames, may be used as the information to be displayed.

[0102] FIG. 9 is a diagram showing an example of the twenty-first moving image information G21 captured outdoors on a rainy day according to the embodiment. In the 21st video information G21, the exercise of the 71st subject B71 is captured. The 71st subject B71 is an example of a subject. FIG. 9 shows an example of a change in the movement of the g-th joint Hg, which is a predetermined joint of the 71st subject B71. In the example of FIG. 9, in order to simplify the illustration, a reference symbol is attached to an a-th raindrop Qa, which is an example of a raindrop, and the reference symbols of the other raindrops are omitted. It should be noted that the g-th joint Hg in the example of FIG. 9 is the same joint as the e-th joint He in the example of FIG. 8A and the f-th joint Hf in the example of FIG. 8B.

[0103] Here, when the weather at the shooting location is fine, for example, it is suitable for processing to adjust the brightness of the video to be lowered and perform analysis. In other words, when the weather is fine, the brightness is higher than indoors and is prone to whiteout, so it may be necessary to adjust the brightness to be lowered and perform analysis. Also, when the weather at the shooting location is cloudy, for example, it is suitable for processing to adjust the brightness of the video to be increased and perform analysis. In other words, when the weather is cloudy, the brightness is lower than indoors, making it difficult to detect the boundary between the person in the video and the background, so it may be necessary to adjust the brightness to be increased and perform analysis. Also, when the weather at the shooting location is rainy, for example, the brightness of the video is adjusted to be increased, and analysis is suitable for processing without taking many analysis points. In other words, when it is raining, as when it is cloudy, the brightness is lower than indoors, making it difficult to detect the boundary between the person in the video and the background, and furthermore, raindrops in the video become noise during analysis, so the more analysis points there are, the more noise there will be, and the accuracy of the analysis will decrease.

[0104] For example, multiple machine learning models for analyzing the subject's movement may be prepared according to the weather at the shooting location. As a specific example, a machine learning model suitable for video information shot at a sunny location, a machine learning model suitable for video information shot at a cloudy location, and a machine learning model suitable for video information shot at a rainy location may be prepared separately. Also, for example, when video information shot in a rainy location is processed, the reliability may be determined to be lower than when video information shot in a sunny or cloudy location is processed. In this case, for example, a process may be performed to suggest improving the shooting environment so that video information is acquired when the shooting location is sunny or cloudy. In this example, the weather conditions such as sunny, cloudy, and rainy are exemplified, but other weather conditions such as snow may be used.

[0105] As a specific example, if the shooting location is outdoors, it is sunny and there is backlighting, the reliability may be determined to be low, and a suggestion may be made to improve the shooting environment by taking the photo from the opposite direction to avoid backlighting. As another specific example, if the shooting location is outdoors and it is sunny and too bright, causing blown-out highlights, the reliability may be determined to be low and a suggestion may be made to improve the shooting environment by shooting in the shade. As another specific example, if the shooting location is outdoors and it is raining, the reliability may be determined to be low, and a suggestion may be made to improve the shooting environment by taking the photo in a place where there is no rain. As another specific example, if the shooting location is outdoors and it is cloudy, the reliability may be determined to be low, and a suggestion may be made to improve the shooting environment, such as increasing the brightness by using a light or the like.

[0106] An example of an imaging environment in which the times are morning, midday, and evening will be described with reference to FIGS. 10A to 10C. In this example, for the sake of simplicity, each of Figures 10A to 10C shows a group of six consecutive frame images arranged horizontally as the video information of the video. Each frame or a group of two or more frames in the video shown in each of Figures 10A to 10C may be used as information to be displayed.

[0107] FIG. 10A is a diagram showing an example of the thirty-first moving image information G31 taken outdoors in the morning according to the embodiment. In the 31st video information G31, the exercise of the 81st subject B81 is captured. The 81st subject B81 is an example of a subject. FIG. 10A shows an example of a change in movement of the i-th joint Hi, which is a predetermined joint of the 81st subject B81. In the example of FIG. 10A, the first sun K1, which is the sun located in the morning position, appears in the background of the video.

[0108] FIG. 10B is a diagram showing an example of the thirty-second moving image information G32 taken outdoors in the daytime according to the embodiment. In the 32nd video information G32, the exercise of the 91st subject B91 is captured. The 91st subject B91 is an example of a subject. FIG. 10B shows an example of a change in movement of the j-th joint Hj, which is a predetermined joint of the 91st subject B91. In the example of FIG. 10B, the second sun K2, which is the sun in the daytime position, appears in the background of the video.

[0109] FIG. 10C is a diagram showing an example of the thirty-third moving image information G33 captured outdoors in the evening according to the embodiment. In the 33rd video information G33, the exercise of the 111th subject B111 is captured. The 111th subject B111 is an example of a subject. FIG. 10C shows an example of a change in movement of the k-th joint Hk, which is a predetermined joint of the 111th subject B111. In addition, in the example of FIG. 10C, the third sun K3, which is the sun in an evening position, appears in the background of the video. It should be noted that the i-th joint Hi in the example of FIG. 10A, the j-th joint Hj in the example of FIG. 10B, and the k-th joint Hk in the example of FIG. 10C are the same joint.

[0110] Here, if the photograph was taken in the morning, for example, it is suitable for analysis without taking detailed analysis points. In other words, in this case, when the sun is low in the sky, sunlight enters from an angle and the image is easily backlit, resulting in overexposure, so taking detailed analysis points will result in noise. Also, when the image is taken during the day, it is suitable for processing to adjust the brightness to be lowered and perform analysis. In other words, in this case, for example, when it is sunny, the brightness is higher than indoors and is prone to whiteout, so it may be necessary to adjust the brightness to be lowered and perform analysis. Also, when the shooting time is in the evening, for example, it is suitable for processing to adjust brightness and contrast and perform analysis without taking detailed analysis points. That is, in this case, the color of the video shot in the evening is redder than the color of the video shot during the day, so it is necessary to adjust the contrast. Furthermore, the brightness of the video shot in the evening is lower than the brightness of the video shot during the day, so it may be necessary to adjust the brightness to be higher. Also, when the contrast and brightness of the video are adjusted, it becomes difficult to detect the boundary between the human and the background, and taking detailed analysis points may result in noise.

[0111] For example, multiple machine learning models for analyzing the subject's motion may be prepared according to the outdoor shooting time. As a specific example, a machine learning model suitable for video information shot outdoors in the morning, a machine learning model suitable for video information shot outdoors in the daytime, and a machine learning model suitable for video information shot outdoors in the evening may be prepared separately. In addition, for example, when video information in which the outdoor shooting time is a predetermined time is processed, the reliability may be determined to be lower than when video information in which the outdoor shooting time is another predetermined time is processed. In this case, for example, a process of suggesting an improvement of the shooting environment may be performed, such as changing the outdoor shooting time to acquire video information.

[0112] In this example, it is assumed that the morning period is from sunrise to 10:00, the afternoon period is from 10:00 to 16:00, and the evening period is from 16:00 to sunset, but other time periods may be used for each period. Furthermore, in this example, three time divisions, namely, morning, daytime, and evening, are shown, but other divisions, such as night, may also be used. Additionally, the time segments may be defined, for example, by the start and end times of each segment. Furthermore, as time divisions, different time zone divisions may be used for each season, for example. Furthermore, the time divisions may be, for example, divisions that are unrelated to the movement of the sun.

[0113] As a specific example, if the shooting location is outdoors, in the morning or evening, and the lighting is backlit, the reliability may be determined to be low, and a suggestion may be made to improve the shooting environment by taking the photo from the opposite direction so that there is no backlighting. As another specific example, if the shooting location is outdoors, during the day, and it is too bright and the image is blown out, the reliability may be determined to be low and a suggestion may be made to improve the shooting environment by taking the photo in the shade.

[0114] The relationship between the color of the subject's clothing and the background, which is an example of a shooting environment, will be described with reference to FIGS. 11A and 11B. In this example, for the sake of simplicity, each of Fig. 11A and Fig. 11B shows a group of six consecutive frames of a moving image arranged horizontally as moving image information of the moving image. Each frame of the moving image shown in each of Fig. 11A to Fig. 11B, or a group of two or more frames, may be used as information to be displayed.

[0115] FIG. 11A is a diagram showing an example of the 41st moving image information G41 in which the clothing of the subject and the background are the same color according to the embodiment. In the 41st video information G41, the exercise of the 121st subject B121 is captured. The 121st subject B121 is an example of a subject. FIG. 11A shows an example of a change in movement of the oth joint Ho, which is a predetermined joint of the 121st subject B121. 11A, the color of the first glove U1 of the 121st subject B121 is white, which is the same color as the white background part in the video. The part in question is, for example, a whiteboard.

[0116] FIG. 11B is a diagram showing an example of the 42nd moving image information G42 in which the clothing of the subject and the background are different colors according to the embodiment. In the 42nd video information G42, the exercise of the 131st subject B131 is captured. The 131st subject B131 is an example of a subject. FIG. 11B shows an example of a change in movement of the p-th joint Hp, which is a predetermined joint of the 131st subject B131. In the example of FIG. 11B, the color of the second glove U2 of the 131st subject B131 is a color other than white, which is different from the white background color in the video. It should be noted that the oth joint Ho in the example of FIG. 11A and the pth joint Hp in the example of FIG. 11B are the same joint.

[0117] Here, the first glove U1 in the example of FIG. 11A is an example of clothing of the 121st subject B121, and the second glove U2 in the example of FIG. 11B is an example of clothing of the 131st subject B131. In this example, a glove, such as the first glove U1 or the second glove U2, is part of the subject's overall outfit.

[0118] Here, when the clothing of the subject and the background in the video are the same color, for example, it is suitable for processing to perform analysis without taking detailed analysis points. In other words, in this case, it is difficult to detect the boundary between the person and the background in the video in the same color area, so analysis errors are likely to occur and it may be impossible to detect the boundary at all. In this case, even if analysis points are taken finely, noise points will be detected, and the analysis accuracy will decrease. On the other hand, when the clothing of the subject in the video is a different color from the background, for example, it is suitable for processing to perform analysis by selecting analysis points in detail. In other words, in this case, the boundary between the person in the video and the background can be clearly detected, making it possible to perform analysis with sufficient accuracy.

[0119] For example, multiple machine learning models for analyzing the subject's movement may be prepared depending on whether the subject's clothes and the background in the video are the same color or different colors. As a specific example, a machine learning model suitable for video information in which the subject's clothes and the background in the video are the same color and a machine learning model suitable for video information in which the subject's clothes and the background in the video are different colors may be prepared separately. In addition, for example, when video information is processed in which the subject's clothing and the background are the same color, the reliability may be determined to be lower than when video information is processed in which the subject's clothing and the background are different colors. In this case, for example, a process of suggesting an improvement to the shooting environment may be performed to obtain video information in which the subject's clothing and the background are different colors.

[0120] As a specific example, if the color of the subject's clothing and the background color in a recorded video are similar and the reliability is determined to be low, a suggestion may be made to improve the shooting environment by changing at least one of the subject's clothing or the shooting location. In this example, cases where the color of the subject's clothing and the background color are similar include cases where these colors are the same color and cases where these colors are not the same color but are close enough to be indistinguishable. In this way, similar processing may be performed on similar colors that are not the same color. In other words, even if a color is strictly different, it may be excluded from the different colors if it is similar.

[0121] Referring to FIG. 12, a description will be given of an example of a photographing environment, in which a person is photographed. In this example, a moving image without a person in it is, for example, similar to the moving image shown in FIG. 7A. In this example, for the sake of simplicity, a group of six consecutive frame images in a moving image is shown as moving image information of the moving image in Fig. 12. Each frame in the moving image shown in Fig. 12, or a group of two or more frames, may be used as information to be displayed.

[0122] FIG. 12 is a diagram showing an example of the 51st moving image information G51 in which a person other than the target person according to the embodiment is captured. In the 51st video information G51, the exercise of the 141st subject B141 is captured. The 141st subject B141 is an example of a subject. FIG. 12 shows an example of a change in the movement of the q-th joint Hq, which is a predetermined joint of the 141st subject B141. In the example of FIG. 12, a first person V1, who is a person other than the 141st subject person B141, appears in the video.

[0123] Here, when a person other than the target person is captured in the video, for example, it is suitable for processing to perform analysis without taking detailed analysis points. In other words, in this case, if analysis points are taken in detail, there is a possibility that a person other than the target person will be erroneously analyzed, and the analysis accuracy will decrease. On the other hand, if the video does not include people other than the target person, it is suitable for processing to perform analysis by selecting detailed analysis points. In other words, in this case, since the video does not include people other than the target person, analysis with sufficient accuracy is possible.

[0124] For example, multiple machine learning models for analyzing the motion of a subject may be prepared depending on whether or not a person other than the subject is captured in the video. As a specific example, a machine learning model suitable for video information in which a person other than the subject is captured and a machine learning model suitable for video information in which a person other than the subject is not captured may be prepared separately. Also, for example, when video information in which people other than the target person are captured is processed, the reliability may be determined to be lower than when video information in which people other than the target person are not captured is processed. In this case, for example, a process of suggesting an improvement to the shooting environment may be performed so as to obtain video information in which people other than the target person are not captured.

[0125] Here, in this example, a person other than the subject has been given as an example of an object that appears in the video, but for example, if it is determined that the person is not moving in the video, the impact on the analysis of the subject's movement may be smaller than when it is determined that the person is moving, so processing may be performed assuming that the person is not visible.

[0126] Although the above description has been given of a case where a person other than the subject moves as an example of a case where the background other than the subject moves in a video, the same processing may be performed when other moving objects are captured in the video. The other objects may be, for example, animals, vehicles such as automobiles or airplanes, smoke, raindrops, snow, etc.

[0127] As a specific example, if a person other than the target person is captured in the captured video and the reliability is determined to be low, a suggestion may be made to improve the shooting environment by shooting in a different location where there are no people other than the target person.

[0128] An example of a photographing environment, ie, blurring, will be described with reference to FIGS. 13A and 13B. In this example, for the sake of simplicity, each of Fig. 13A and Fig. 13B shows a group of six consecutive frames of a moving image arranged horizontally as moving image information of the moving image. Each frame of the moving image shown in each of Fig. 13A to Fig. 13B, or a group of two or more frames, may be used as information to be displayed.

[0129] FIG. 13A is a diagram showing an example of the 61st moving image information G61 manually captured by a person according to the embodiment. In the 61st video information G61, the exercise of the 151st subject B151 is captured. The 151st subject B151 is an example of a subject. FIG. 13A shows an example of a change in the movement of the r-th joint Hr, which is a predetermined joint of the subject B151. In the example of FIG. 13A, a first line W1 serving as a reference for the position of the shoes of the 151st subject B151 is shown near the feet of the 151st subject B151. In the example of FIG. 13A, the image is captured while the photographing device 21 is being manually held by a person, and blurring occurs at the position where the 151st subject B151 is captured in multiple frames of the video.

[0130] FIG. 13B is a diagram showing an example of the 62nd moving image information G62 captured using a tripod according to the embodiment. In the 62nd video information G62, the exercise of the 161st subject B161 is captured. The 161st subject B161 is an example of a subject. FIG. 13B shows an example of a change in movement of the s-th joint Hs, which is a predetermined joint of the 161st subject B161. In the example of FIG. 13B, an eleventh line W11 serving as a reference for the position of the shoes of the 161st subject B161 is shown near the feet of the 161st subject B161. In the example of FIG. 13B, the image is captured with the image capturing device 21 fixed in position by a tripod, and no blurring occurs in the position where the 161st subject B161 is captured in multiple frames of the video. It should be noted that the r-th joint Hr in the example of FIG. 13A and the s-th joint Hs in the example of FIG. 13B are the same joint.

[0131] Here, when a video is manually shot by a person and is blurred, for example, it is suitable for processing to perform analysis without taking detailed analysis points. In other words, in this case, if analysis points are taken finely, positional changes of people in the shot video due to camera shake or the like will be detected, and there is a possibility that erroneous results will be output. On the other hand, if the video is shot using a tripod and there is no shaking, it is suitable for processing to perform analysis by selecting small analysis points. In other words, in this case, the pivot foot does not shake, and the position of the person in the shot video does not change, making it possible to perform analysis with less noise.

[0132] For example, multiple machine learning models for analyzing the motion of a subject may be prepared depending on whether or not the video is blurred. As a specific example, a machine learning model suitable for video information with blur and a machine learning model suitable for video information without blur may be prepared separately. Also, for example, when blurred video information is processed, the reliability may be determined to be lower than when non-blurred video information is processed. In this case, for example, a process may be performed to suggest improvements to the shooting environment so as to obtain non-blurred video information.

[0133] In this example, it is assumed that there will be blur in the video when the video is shot manually by a person, and there will be no blur when the video is shot using a tripod, but it is not necessary to use a method that determines the presence or absence of blur based on whether the video is shot manually by a person or using a tripod; for example, a method that determines the presence or absence of blur based on the video may be used. Although a tripod is used in this example, other devices may be used to fix the imaging device 21.

[0134] As a specific example, if the captured video is blurred and is determined to have low reliability, a suggestion may be made to improve the shooting environment, such as by fixing the device, such as a camera, used to capture the video.

[0135] Here, the shooting environments shown in the above examples are illustrative and not limited to these, and various other shooting environments may be taken into consideration. For example, a case may be taken into consideration where the area in which the subject is captured is too small compared to the angle of view of the video, where the subject's body does not fit within the angle of view of the video and part of the subject's body extends beyond the angle of view, or where the subject's orientation is not appropriate for the direction in which the video is shot. Furthermore, in each of the above examples, a mode in which the position of one joint in the human body is displayed has been shown, but a mode in which the positions of two or more joints are displayed may also be used. Furthermore, in each of the above examples, a mode in which the positions of the joints on the human body are displayed has been shown, but as another example, a mode in which the positions of the joints on the human body are not displayed may be used. Furthermore, the human body does not necessarily have to be displayed precisely, but may be displayed, for example, roughly. Also, a representation of the human skeleton may be used instead of or in addition to the human body.

[0136] FIG. 14 is a diagram illustrating an example of a display using a human skeleton according to the embodiment. 14, the body of the 211th subject B211 is diagrammatically shown, and the skeleton of the 211th subject B211 is diagrammatically shown. The 211th subject B211 is an example of a subject. In the example of Fig. 14, the skeleton is composed of a plurality of bones L. In the example of Fig. 14, a reference symbol is attached to one bone L, and the reference symbols of the other bones are omitted. In addition, in the example of Fig. 14, a joint H is shown. Note that in the example of Fig. 14, a reference symbol is attached to one joint H, and the reference symbols of the other joints are omitted.

[0137] For example, as a form of diagrammatically displaying a human being, a skeletal display as shown in FIG. 14 may be used. 14 shows a mode in which both the human body and skeleton are displayed, but as another example, a mode in which the human skeleton is displayed without displaying the human body may be used. Also, the human skeleton does not necessarily need to be displayed precisely, and may be displayed roughly. Furthermore, in the example of FIG. 14, a mode in which the positions of the joints are displayed on the skeleton is shown, but as another example, a mode in which the positions of the joints are not displayed may be used.

[0138] Learning of the reliability determination process will be described with reference to FIGS.

[0139] FIG. 15 is a diagram illustrating an example of a process during learning according to the embodiment. The example of Fig. 15 shows a case where a machine learning reliability determination model A2 is trained. In the example of Fig. 15, machine learning labeling is performed. Note that Fig. 15 is a schematic diagram for explanation, and is not necessarily a strict diagram.

[0140] FIG. 15 shows an acquisition process r1, an estimation process r2, a labeling process r3, a reliability determination process r4, and a presentation process r5. 15 also shows a 311th subject B 311. The 311th subject B 311 is an example of a subject. 15 also shows a first user ua1 who is an example of a user who operates the information processing device 11. The first user ua1 may be, for example, the first target person B311, or another person.

[0141] Nine sensors, a1 sensor a1 to a9 sensor a9, are attached to different predetermined locations on the body of the subject B311. The predetermined locations may be, for example, the positions of the joints. In this example, machine learning correct answer data b1 is generated from some or all of the information of the detection values ​​of the a1 sensor a1 to the a9 sensor a9, or information obtained by calculation based on the information. The correct answer data b1 is used as training data. In the example of FIG. 1, the a1-th sensor a1 to the a9-th sensor a9 are included in the sensor unit 22.

[0142] In this example, the image capturing device 21 captures the motion of the subject B311. In this example, in the information processing device 11, each process shown in FIG.

[0143] In the acquisition process r1, the video acquisition unit 171 acquires a video capturing the exercise of the 311th subject B311. Furthermore, when information other than the video is used for machine learning, in the acquisition process r1, the acquisition unit 151 also acquires the other information. The other information may be, for example, one or both of information on the shooting environment of the video and information on the 311th subject B311.

[0144] In the estimation process r2, the estimation unit 152 estimates the motion of the 311th subject B311 from the acquired video. Here, in this estimation, for example, estimation may be performed regarding the movement of the joints of the subject B311.

[0145] In the labeling process r3, the learning control unit 157 performs machine learning labeling processing. Here, the labeling process may be performed based on, for example, the result of the estimation process and the supervised answer data b1. Furthermore, the labeling process may be performed, for example, according to instructions given by the a1 user ua1.

[0146] In the reliability determination process r4, the reliability determination unit 153 determines the reliability of the result of the estimation process of the motion of the 311th subject B311. In this example, the process of determining the reliability refers to the labeling information.

[0147] In the presentation process r5, under the control of the output control unit 156, the reliability determined in the reliability determination process r4 and the result of the estimation process in the estimation process r2 are presented. Here, the presentation may be in any form, and for example, the information to be presented may be displayed on the screen of a display unit.

[0148] The a1 user ua1 gives labeling instructions as necessary based on the presented information, thereby realizing feedback for machine learning.

[0149] Here, for example, when video information captured in various shooting environments is input to the estimation model A1 used in the estimation process r2, the reliability may change for each different shooting environment. In other words, there may be a tendency for shooting environments to be suitable for the estimation model A1, and a tendency for shooting environments to be unsuitable for the estimation model A1. For this reason, when the reliability is high, the estimation result by the estimation model A1 is used for subsequent analysis, but when the reliability is low, it may be necessary to suggest improving the shooting environment of the video. In other words, it may be better to increase the reliability of the estimation result by re-shooting the video, rather than performing subsequent analysis using an estimation result with low reliability.

[0150] For example, when a classifier is used, it is possible to label a first imaging environment to a first class and a second imaging environment to a second class. The first imaging environment and the second imaging environment are, for example, different imaging environments, and various imaging environments may be labeled, such as an imaging environment that is too bright, an imaging environment that is too dark, an imaging environment in which a person other than the subject is captured, and the like. Also, a classifier with three or more classes may be used.

[0151] FIG. 16 is a diagram illustrating an example of analysis content in a learning model according to the embodiment. In this example, the model under training is the reliability determination model A2 under training. 16, the horizontal axis represents a predetermined index, and the vertical axis represents the degree of agreement with correct answer data b1, which indicates the degree of effect, as a value from 0% to 100%. On the vertical axis, 0% indicates no effect, the effect increases toward the top, and the effect is maximum at 100%. 16 shows a distribution characteristic 711 that indicates the analysis result when a large amount of pattern information is analyzed for the model under training. Note that the distribution characteristic 711 is an example for the purpose of explanation, and is not an actually obtained result. 16, the predetermined index is, for example, the predetermined parameter itself or a value obtained from the predetermined parameter. In this example, the predetermined index is a plurality of pieces of pattern information.

[0152] In the example of Figure 16, the point corresponding to the data of the d1 pattern corresponds to the d1 point d1, the point corresponding to the data of the d2 pattern corresponds to the d2 point d2, and the point corresponding to the data of the d3 pattern corresponds to the d3 point d3. Here, the d1 pattern to the d3 pattern may be patterns of any shooting environment, respectively. In this example, the d1 pattern to the d3 pattern are different patterns among a plurality of pieces of pattern information. From the distribution characteristic 711 shown in FIG. 16, the imaging environment that matches the model being trained can be understood.

[0153] As described above, the information processing device 11 according to this embodiment includes the video acquisition unit 171, the estimation unit 152, the reliability determination unit 153, and the notification unit 155. The video acquisition unit 171 acquires a video of the subject's exercise. The estimation unit 152 estimates the motion of the subject from the video acquired by the video acquisition unit 171. The reliability determination unit 153 determines the reliability of the subject's motion estimated by the estimation unit 152. The notification unit 155 notifies the subject of an improvement in the imaging environment for imaging the exercise, depending on the reliability determined by the reliability determination unit 153.

[0154] Therefore, in the information processing device 11 according to this embodiment, when the reliability of the estimation result is low, it is possible to determine that the video shooting environment is poor and to suggest improvements to the shooting environment, thereby making it possible to prevent, for example, unnecessary analysis from being performed based on an estimation result with low reliability.

[0155] For example, in the information processing device 11 according to this embodiment, the reliability of the joint positions of a subject estimated from a video capturing the subject's movements is calculated, and if the calculated reliability is good, further analysis is performed using the joint positions, whereas if the calculated reliability is poor, analysis using the joint positions is not performed and a suggestion is made to improve the shooting environment of the video. As a result, in the information processing device 11 according to the present embodiment, for example, when an evaluation of the subject's exercise quality is obtained as an analysis result, it is possible to prevent the subject, etc. from being notified of the exercise evaluation using an estimation result with low reliability. In the present embodiment, as a specific example, the evaluation of the subject's exercise is an evaluation of the subject's golf swing.

[0156] In this way, the information processing device 11 according to this embodiment can prevent the failure of the estimation result or the presentation of an unintended analysis result to the user, which may occur in an environment in which the device is actually expected to be used. Note that, for example, by continuously learning the reliability determination model A2 based on feedback from the user, it is possible to reduce unknown failures.

[0157] In addition, as a device for capturing video of the subject's exercise, for example, a specific external device may be used. The external device may be, for example, the photographing device 21 shown in Fig. 1. The photographing device 21 may be, for example, a smartphone carried by the subject, that is, the camera function of the smartphone may be used as the photographing function of the photographing device 21. When the photographing device 21 is integrated with the information processing device 11, the information processing device 11 may be, for example, a smartphone carried by the subject.

[0158] In the information processing device 11 according to this embodiment, for example, the reliability determining unit 153 has a learning model using a classifier. Therefore, in the information processing device 11 according to this embodiment, the reliability of the estimation result based on the moving image can be determined by using a classifier.

[0159] In the information processing device 11 according to this embodiment, for example, the estimation unit 152 estimates the joint positions of the subject from a moving image. Therefore, in the information processing device 11 according to the present embodiment, when an analysis is performed on the movement of the joint positions of a subject, it is possible to prevent unnecessary analysis from being performed based on an estimation result with low reliability.

[0160] In the information processing device 11 according to the present embodiment, for example, improvement of the shooting environment includes a proposal for a case where a moving image is shot indoors and the reliability is low. Therefore, in the information processing device 11 according to this embodiment, when a video is shot indoors and the shooting environment is poor, for example, the lighting is too bright or too dark, it is possible to suggest improving the shooting environment.

[0161] In the information processing device 11 according to the present embodiment, for example, improvement of the shooting environment includes a proposal for a case where a moving image is shot outdoors and the reliability is low. Therefore, in the information processing device 11 according to this embodiment, when a video is shot outdoors and the shooting environment is poor, for example, when it is too bright or too dark due to the weather or the time, it can suggest improving the shooting environment.

[0162] In the information processing device 11 according to the present embodiment, for example, improvement of the shooting environment includes a proposal in the case where the background color of the location where the video was shot and the color of the subject's clothes are similar and the reliability is low. Therefore, in the information processing device 11 according to this embodiment, when the color of the subject's clothing is similar to the background color, it is possible to suggest improving the shooting environment by changing at least one of the subject's clothing or the shooting location before taking the photograph.

[0163] In the information processing device 11 according to the present embodiment, for example, improvement of the shooting environment includes a proposal for a case where a person other than the subject appears in the video and the reliability is low. Therefore, in the information processing device 11 according to this embodiment, when people other than the target person appear in the video, it is possible to suggest improvements to the shooting environment so that the video is shot in an environment in which people other than the target person do not appear in the video.

[0164] In the information processing device 11 according to the present embodiment, for example, improvement of the shooting environment includes a proposal for a case where a video is blurred and has low reliability. Therefore, in the information processing device 11 according to the present embodiment, when a moving image is blurred, it is possible to suggest improvements to the shooting environment so that the moving image is shot in a shooting environment in which blurring does not occur.

[0165] The information processing device 11 according to the present embodiment includes an analysis unit 154 and an output unit 112, for example. The analysis unit 154 performs analysis using the subject's movement estimated by the estimation unit 152 according to the reliability. The output unit 112 outputs the analysis result obtained by the analysis unit 154 . Therefore, the information processing device 11 according to this embodiment can perform analysis using the estimation result of the subject's motion and display the analysis result.

[0166] In addition, the analysis unit 154 may analyze, for example, the movement of the skeleton during the exercise of the subject. Here, as the skeletal information, for example, information based on a general human skeleton may be used, or skeletal information of the subject's body may be measured and used. Furthermore, the skeleton information does not necessarily have to be strictly accurate, and for example, general skeleton information may be used.

[0167] Furthermore, the output unit 112 may output the analysis result to, for example, an external device of the subject. Here, the external device may be, for example, a smartphone carried by the subject. In this case, for example, a screen display function of the smartphone may be used as a display function of the display device 31.

[0168] In the information processing device 11 according to this embodiment, for example, the analysis unit 154 performs analysis based on information related to the shooting environment. Therefore, in the information processing device 11 according to the present embodiment, the analysis is performed based on the estimation result of the subject's motion as well as information on the shooting environment, thereby making it possible to improve the accuracy of the analysis.

[0169] In the information processing device 11 according to this embodiment, for example, the analysis unit 154 performs analysis based on information on the subject. Therefore, in the information processing device 11 according to the present embodiment, by performing analysis based on information about the subject together with the estimation result of the subject's motion, it is possible to improve the accuracy of the analysis.

[0170] In the information processing device 11 according to the present embodiment, for example, the information on the subject includes information indicating the age of the subject and information indicating the exercise experience of the subject. Therefore, in the information processing device 11 of this embodiment, for example, when performing analysis, one or both of the subject's age and the subject's exercise experience can be used, and analysis results suitable for the subject can be obtained.

[0171] Here, the information indicating age may be, for example, a value indicating the age itself, or information indicating a classification such as young, middle-aged, or elderly, or information indicating a classification as either an adult or a child. In addition, various information may be used as information indicating exercise experience. For example, information indicating the number of years of exercise experience may be used without specifying the type of exercise, or information indicating whether or not a specified type of exercise has experience may be used, or information indicating the number of years of experience of a specified type of exercise may be used. Furthermore, as information indicating the exercise experience, information indicating the proficiency of the exercise technique, such as amateur or professional, may be used. In addition, various other information may be used as the information about the subject.

[0172] Here, the subject information acquisition function of the acquisition unit 151 may acquire, for example, information input by a user as information about the subject. As another example, the subject information acquisition function of the acquisition unit 151 may receive and acquire information about the subject stored in a smartphone or the like carried by the subject from the smartphone or the like.

[0173] As a specific example, this embodiment can provide a body motion analysis system that estimates the positions of body joints in 2D or 3D. In the body motion analysis system, the reliability of the joint position estimation result is automatically calculated, and the analysis result using the estimation result is variably presented. In other words, in the body motion analysis system, when the reliability is low, the analysis result is not presented, and a suggestion for improving the shooting environment is made. In the body motion analysis system, a reliability calculation model is trained in advance using output data of an estimator that estimates joint positions and training data used to train the estimator, where the estimator is a machine learning estimation model and the training data is ground truth data for the estimator.

[0174] As a specific example, a two-class classifier may be trained with the output data of the estimator as the first class and the training data as the second class. In this example, the input of the two-class classifier is the joint position, and the output of the two-class classifier is a continuous value of the reliability of each class. In this case, the output is a value between 0 and 1, and the sum of the values ​​of both classes is 1. By incorporating such a two-class classifier as a reliability calculation model into the body motion analysis system and calculating the reliability of the estimation result, for example, it is possible to automatically distinguish between failures of the estimation result and unintended estimation results, and to notify the user of the occurrence of an abnormality, etc. Also, for example, by continuously training the two-class classifier based on feedback from the user, it is possible to build a system that can automatically respond to unknown failures.

[0175] Furthermore, with this body movement analysis system, even if the environment in which the video will be shot is not specified in detail before the video is shot, suggestions for improving the shooting environment can be made, if necessary, depending on the reliability of the estimation results based on the video after shooting. In addition, in the case where the body movement analysis system, for example, collects analysis result data using a specified device and stores it in a memory area, it can help reduce communication charges or storage area by confirming or presenting the necessity of the analysis results depending on the reliability at a stage before the data is collected.

[0176] An example of a method for acquiring information about the shooting environment will be described below. The information may be used, for example, when training the reliability determination model A2.

[0177] For example, location information of an external device of the subject may be acquired. The function of the external device to acquire location information may be, for example, a function of a Global Navigation Satellite System (GNSS) such as a Global Positioning System (GPS). In this case, the location of the external device may be regarded as the location of a person carrying the external device. Whether the location where the video was taken is indoors or outdoors may be determined based on information such as a GPS, or may be determined based on the contents of a manual input by a user. Such a determination may be performed by, for example, a shooting environment acquisition function of the acquisition unit 151.

[0178] For example, the weather information may be weather information provided by the Japan Meteorological Agency or a private company. The photographing environment acquisition function of the acquisition unit 151 may receive weather information via the communication unit 113, for example, in a wired or wireless manner.

[0179] The weather at the location where the video was shot may be determined based on, for example, information such as a GPS indicating the location of the location and weather information, or may be determined based on the contents of a manual input by a user. Such a determination may be performed, for example, by a shooting environment acquisition function of the acquisition unit 151. For example, information such as a GPS indicating the location of the location may be information such as a GPS indicating the location of a subject present at the location. As another example, the photographing environment acquisition function of the acquisition unit 151 may determine the weather by performing a predetermined image recognition process on a video.

[0180] For example, the time information may be information of a time signal, or information of time based on a GPS signal. The photographing environment acquisition function of the acquisition unit 151 may receive the time information by wire or wirelessly via the communication unit 113. In general, the time information has a time difference depending on the country or region. The time at the location where the video was taken may be determined based on, for example, information such as a GPS indicating the location of the location and time information, or may be determined based on the contents of a manual input by a user. Such a determination may be performed, for example, by a shooting environment acquisition function of the acquisition unit 151. For example, information such as a GPS indicating the location of the location may be information such as a GPS indicating the location of a subject present at the location.

[0181] For example, the photographing environment acquisition function of the acquisition unit 151 may determine whether the background color and the clothing color are the same color based on whether the boundary between the subject and the surroundings can be detected. In other words, if the boundary can be detected, the background color and the clothing color are considered to be different colors, and if the boundary cannot be detected, the background color and the clothing color are considered to be the same color. The boundary between a person and the background in a video may be called, for example, an edge.

[0182] Here, the subject's clothing may be, for example, all of the clothing worn by the subject, or a predetermined part of the clothing. In addition, the shooting environment acquisition function of the acquisition unit 151 may determine whether the background color and the color of the clothing are the same color by performing a predetermined image recognition process based on, for example, a video in which the subject appears or an image of a frame of the video.

[0183] For example, the movement of the background may be the movement of an object other than the subject person. The object may be, for example, a person other than the subject person, an animal, a vehicle such as a car or an airplane, smoke, raindrops, snow, or the like. Here, the shooting environment acquisition function of the acquisition unit 151 may perform a predetermined image recognition process based on, for example, a video in which the subject appears or an image of a frame of the video, thereby determining whether there is movement in the background or identifying objects moving in the background.

[0184] For example, the photographing environment acquisition function of the acquisition unit 151 may determine blur based on whether the position of the subject's supporting foot has changed. In other words, if the position of the subject's supporting foot has changed, it is deemed that there is blur, and if the position of the subject's supporting foot has not changed, it is deemed that there is no blur. Here, the shooting environment acquisition function of the acquisition unit 151 may determine whether the position of the subject's pivot foot has changed by performing a predetermined image recognition process based on, for example, a video in which the subject appears or an image of a frame of the video.

[0185] In this embodiment, the subject performs a golf swing motion as an example of a sport, but the sport may be any sport, and may be various sports such as baseball, soccer, tennis, etc. As a specific example, the information processing device 11 may analyze a baseball batting form or pitching form, a soccer kicking form, or a tennis swing form. Furthermore, the information processing device 11 according to the present embodiment and the like may be applied to things other than sports, for example, when a subject performs a motion other than in sports.

[0186] In the present embodiment, a case has been shown in which information on the joints of a subject appearing in a video is obtained based on video information, but as another example, information obtained by collecting a plurality of still image frames may be used. Even when such information obtained by collecting a plurality of still image frames is used, it is possible to perform substantially the same processing as when video information is used. Such information obtained by collecting a plurality of still image frames may be used as information on a video.

[0187] In addition, in this embodiment, a case has been shown in which information regarding the joints of a subject appearing in a video is obtained based on information about the video. However, as another example, a configuration may be used in which information regarding the joints of a subject is obtained based on information detected by a sensor. In other words, the movement of the subject's joints, etc. may be identified based on the detection results of a sensor attached to the subject's body, and the movement of the subject's joints, etc. may be identified based on the movement of a tool used by the subject, which is detected by a sensor attached to the tool.

[0188] For example, a configuration may be used in which one or both of the following are performed: information regarding the subject's joints, etc. is obtained based on information detected by a sensor attached to the subject's body, and information regarding the subject's joints, etc. is obtained based on information detected by a sensor attached to a specified location outside the subject. Here, the predetermined part on the outside of the subject is not particularly limited and may be, for example, a predetermined part of a tool used by the subject. The tool may be a tool for a sport played by the subject, such as a golf club, a baseball bat or glove, a soccer ball, or a tennis racket. Additionally, the number of sensors used may be any number greater than or equal to one. As the sensor, any sensor may be used, for example, one or more of an acceleration sensor, a position sensor, an angular velocity sensor, and the like may be used.

[0189] In this way, instead of a camera that captures video of the subject, a sensor that detects information about the subject may be used. Also, for example, a camera that captures a video of a subject may be used together with a sensor that detects information about the subject. In this case, various information about the subject may be acquired based on one or both of the information of the video captured by the camera and the information detected by the sensor. It should be noted that the sensor here refers to a device other than a camera, and various sensors may be used instead of or together with a camera.

[0190] A program for implementing the functions of any of the components of any of the above-described devices may be recorded in a computer-readable recording medium, and the program may be read into a computer system and executed. The term "computer system" as used herein includes hardware such as an operating system or peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CDs (Compact Discs)-ROMs, and storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also refers to storage devices that hold a program for a certain period of time, such as volatile memory in a computer system that serves as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. The volatile memory may be a RAM (Random Access Memory). The recording medium may be a non-transitory recording medium.

[0191] The above program may be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. The "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network such as the Internet or a communication line such as a telephone line. The above program may be for realizing a part of the above-mentioned functions. The above program may be a so-called differential file that can realize the above-mentioned functions in combination with a program already recorded in the computer system. The differential file may be called a differential program.

[0192] The function of any of the components in any of the above-described devices may be realized by a processor. Each process in the embodiment may be realized by a processor that operates based on information such as a program and a computer-readable recording medium that stores information such as a program. The functions of each part of the processor may be realized by individual hardware, or the functions of each part may be realized by integrated hardware. The processor includes hardware, and the hardware may include at least one of a circuit that processes digital signals and a circuit that processes analog signals. The processor may be configured using one or more circuit devices mounted on a circuit board, or one or both of one or more circuit elements. An IC (Integrated Circuit) or the like may be used as the circuit device, and a resistor or a capacitor or the like may be used as the circuit element.

[0193] The processor may be a CPU. However, the processor is not limited to a CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) may be used. The processor may be a hardware circuit using an ASIC (Application Specific Integrated Circuit). The processor may be configured with multiple CPUs, or may be configured with a hardware circuit using multiple ASICs. The processor may be configured with a combination of multiple CPUs and a hardware circuit using multiple ASICs. The processor may include one or more of an amplifier circuit or a filter circuit that processes an analog signal.

[0194] Although the embodiments have been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of this disclosure are also included.

[0195] [Note] Below, <Configuration Example 1> to <Configuration Example 14> are shown. In addition, the lower configuration example may or may not be applied to the higher configuration example. Furthermore, a lower configuration example that is applicable to any of the two or more higher configuration examples may be applied to any of the two or more higher configuration examples, i.e., two or more application examples are generated, and a configuration example that is even lower than the lower configuration example may be applied to any of these two or more application examples.

[0196] <Configuration example 1> A video acquisition unit that acquires a video of the subject's exercise; an estimation unit that estimates the motion of the subject from the video acquired by the video acquisition unit; a reliability determination unit that determines a reliability of the motion of the subject estimated by the estimation unit; a notification unit that notifies the subject of an improvement to a photographing environment for photographing the exercise according to the reliability determined by the reliability determination unit; An information processing device comprising:

[0197] <Configuration example 2> The reliability determination unit has a learning model using a classifier. The information processing device described in <Configuration Example 1>.

[0198] <Configuration example 3> The estimation unit estimates joint positions of the subject from the video. The information processing device according to <Configuration Example 1> or <Configuration Example 2>.

[0199] <Configuration Example 4> The improvement of the shooting environment includes a suggestion when the video was shot indoors and the reliability is low. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 3>.

[0200] <Configuration Example 5> The improvement of the shooting environment includes a suggestion when the video was shot outdoors and the reliability is low. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 4>.

[0201] <Configuration Example 6> The improvement of the shooting environment includes a suggestion when the reliability is low because the background color of the location where the video was shot and the color of the subject's clothes are similar. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 5>.

[0202] <Configuration Example 7> The improvement of the shooting environment includes a suggestion when a person other than the subject is included in the video and the reliability is low. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 6>.

[0203] <Configuration Example 8> The improvement of the shooting environment includes a suggestion when the video is blurred and the reliability is low. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 7>.

[0204] <Configuration Example 9> an analysis unit that performs analysis using the motion of the subject estimated by the estimation unit according to the reliability; an output unit that outputs an analysis result obtained by the analysis unit; The information processing device according to any one of <Configuration Example 1> to <Configuration Example 8>, comprising:

[0205] <Configuration Example 10> The analysis unit performs the analysis based on information about the shooting environment. The information processing device according to <Configuration Example 9>.

[0206] <Configuration Example 11> The analysis unit performs the analysis based on information about the subject. The information processing device according to <Configuration Example 9> or <Configuration Example 10>.

[0207] <Configuration Example 12> The information about the subject includes information indicating the subject's age and information indicating the subject's exercise experience; The information processing device according to <Configuration Example 11>.

[0208] It is also possible to provide a method of processing performed by the information processing device described above. <Configuration Example 13> The acquisition unit acquires a video of the subject's exercise, an estimation unit estimating the motion of the subject from the video acquired by the acquisition unit; a reliability determination unit that determines a reliability of the motion of the subject estimated by the estimation unit; a notification unit notifying the subject of an improvement of a photographing environment for photographing the exercise according to the reliability determined by the reliability determination unit; Information processing methods.

[0209] It is also possible to provide a program recording medium for recording a program executed by the processor in the information processing device as described above. <Configuration Example 14> A program recording medium for recording a program, The program includes: An acquisition function for acquiring a video of the subject's exercise; an estimation function that estimates the motion of the subject from the video acquired by the acquisition function; a reliability determination function that determines reliability of the motion of the subject estimated by the estimation function; A notification function that notifies the subject of an improvement in a shooting environment for shooting the exercise according to the reliability determined by the reliability determination function; To achieve this, Program recording medium. [Explanation of symbols]

[0210] 1...information processing system, 11...information processing device, 21...imaging device, 22...sensor unit, 31...display device, 111...input unit, 112...output unit, 113...communication unit, 114...storage unit, 115...control unit, 151...acquisition unit, 152...estimation unit, 153...reliability determination unit, 154...analysis unit, 155...notification unit, 156...output control unit, 157...learning control unit, 171...video acquisition unit, 711...analysis characteristic, A1...estimation model, A2...reliability determination model, B1...first subject, B11...eleventh subject, B21...twenty-first subject, B31...thirty-first subject, B41...forty-first subject, B51...fifth subject 1 subject, B61...61st subject, B71...71st subject, B81...81st subject, B91...91st subject, B111...111th subject, B121...121st subject, B131...131st subject, B141...141st subject, B151...151st subject, B161...161st subject, B211...211th subject, B311...311th subject, C1...video information, C2...subject information, Fa1...a1th frame, Fa2...a2th frame, Fa3...a3th frame, Fa4...a4th frame, Fa5...a5th frame, Fa6...a6th frame, Fb1...b1st Frame, Fb2...b2 frame, Fb3...b3 frame, G1...1st video information, G2...2nd video information, G11...11th video information, G12...12th video information, G21...21st video information, G31...31st video information, G32...32nd video information, G33...33rd video information, G41...41st video information, G42...42nd video information, G51...51st video information, G61...61st video information, G62...62nd video information, H...joint, Ha1...a1st joint, Ha2...a2nd joint, Ha3...a3th joint, Ha4...a4th joint, Ha5...a5th joint, Ha6...a6th joint, Hb1 ...b1 joint, Hb2...b2 joint, Hb3...b3 joint, Hc...c joint, Hd...d joint, He...e joint, Hf...f joint, Hg...g joint, Hi...i joint, Hj...j joint, Hk...k joint, Ho...o joint, Hp...p joint, Hq...q joint, Hr...r joint, Hs...s joint, K...sun, K1...1st sun, K2...2nd sun, K3...3rd sun, L...bone, Qa...ath raindrop, T1...1st pattern information, T2...2nd pattern information, T3...3rd pattern information, T4...4th pattern information, T11...reliability table, U1...1st glove, U2...2nd glove,V1: first person; a1: a1 sensor; a2: a2 sensor; a3: a3 sensor; a4: a4 sensor; a5: a5 sensor; a6: a6 sensor; a7: a7 sensor; a8: a8 sensor; a9: a9 sensor; b1: correct answer data; d1: d1 point; d2: d2 point; d3: d3 point; r1: acquisition process; r2: estimation process; r3: labeling process; r4: confidence determination process; r5: presentation process; ua1: a1 user;

Claims

1. A video acquisition unit that acquires a video of the subject's exercise; an estimation unit that estimates the motion of the subject from the video acquired by the video acquisition unit; a reliability determination unit that determines a reliability of the motion of the subject estimated by the estimation unit; a notification unit that notifies the subject of an improvement to a photographing environment for photographing the exercise according to the reliability determined by the reliability determination unit; An information processing device comprising:

2. The reliability determination unit has a learning model using a classifier. The information processing device according to claim 1 .

3. The estimation unit estimates joint positions of the subject from the video. The information processing device according to claim 1 .

4. The improvement of the shooting environment includes a suggestion when the video was shot indoors and the reliability is low. The information processing device according to claim 1 .

5. The improvement of the shooting environment includes a suggestion when the video was shot outdoors and the reliability is low. The information processing device according to claim 1 .

6. The improvement of the shooting environment includes a suggestion when the reliability is low because the background color of the location where the video was shot and the color of the subject's clothes are similar. The information processing device according to claim 1 .

7. The improvement of the shooting environment includes a suggestion when a person other than the subject is included in the video and the reliability is low. The information processing device according to claim 1 .

8. The improvement of the shooting environment includes a suggestion when the video is blurred and the reliability is low. The information processing device according to claim 1 .

9. an analysis unit that performs analysis using the motion of the subject estimated by the estimation unit according to the reliability; an output unit that outputs an analysis result obtained by the analysis unit; The information processing device according to claim 1 .

10. The analysis unit performs the analysis based on information about the shooting environment. The information processing device according to claim 9.

11. The analysis unit performs the analysis based on information about the subject. The information processing device according to claim 9.

12. The information about the subject includes information indicating the subject's age and information indicating the subject's exercise experience; The information processing device according to claim 11.

13. The acquisition unit acquires a video of the subject's exercise, an estimation unit estimating the motion of the subject from the video acquired by the acquisition unit; a reliability determination unit that determines a reliability of the motion of the subject estimated by the estimation unit; a notification unit notifying the subject of an improvement of a photographing environment for photographing the exercise according to the reliability determined by the reliability determination unit; Information processing methods.

14. A program recording medium for recording a program, The program includes: An acquisition function for acquiring a video of the subject's exercise; an estimation function that estimates the motion of the subject from the video acquired by the acquisition function; a reliability determination function that determines reliability of the motion of the subject estimated by the estimation function; A notification function that notifies the subject of an improvement in a shooting environment for shooting the exercise according to the reliability determined by the reliability determination function; To achieve this, Program recording medium.

Images