Information processing device, information processing method, and program recording medium

Abstract

To provide an information processing device capable of increasing the accuracy of an analysis result.SOLUTION: An information processing device includes: a moving image acquisition unit that obtains a moving image which is obtained by capturing the image of an exercise of an object person; an image capturing environment acquisition unit that obtains image capturing environment about an image capturing environment of the moving image which is obtained by the moving image acquisition unit; a selection unit that selects an analysis model based on the image capturing environment information obtained by the image capturing environment acquisition unit; a motion analysis unit that analyzes the exercise of the object person from the moving image by using the analysis model selected by the selection unit; and an output unit that outputs an analysis result of the exercise of the object person which is obtained by the operation analysis unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program recording medium. [Background technology]

[0002] Videos of subjects' movements are analyzed. In the device described in Patent Document 1, in a video recording of a golf swing performed by a subject, the frame rate is maintained in an analysis section in which the swing motion is analyzed, and the frame rate is reduced in a non-analysis section in which the swing motion is not analyzed (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-118501 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional technology, when analyzing a video using an estimation model, which is a machine learning model, the estimation model is not changed according to information about the acquired video, so depending on the acquired video, the conditions required by the estimation model may not be met.In such cases, it may not be possible to properly analyze the swing movement captured on video. [Means for solving the problem]

[0005] In order to solve the above problem, one aspect is an information processing device comprising a video acquisition unit that acquires a video of a subject's movements, a shooting environment acquisition unit that acquires shooting environment information related to the shooting environment of the video acquired by the video acquisition unit, a selection unit that selects an analysis model based on the shooting environment information acquired by the shooting environment acquisition unit, a motion analysis unit that analyzes the subject's movements from the video using the analysis model selected by the selection unit, and an output unit that outputs an analysis result of the subject's movements obtained by the motion analysis unit.

[0006] In order to solve the above problem, one aspect of this information processing method is one in which a video acquisition unit acquires a video of a subject's movements, a shooting environment acquisition unit acquires shooting environment information related to the shooting environment of the video acquired by the video acquisition unit, a selection unit selects an analysis model based on the shooting environment information acquired by the shooting environment acquisition unit, a motion analysis unit analyzes the subject's movements from the video using the analysis model selected by the selection unit, and an output unit outputs the analysis result of the subject's movements obtained by the motion analysis unit.

[0007] In order to solve the above problem, one aspect is a program recording medium for recording a program, the program causing a computer to realize a video acquisition function for acquiring a video of a subject's movements, a shooting environment acquisition function for acquiring shooting environment information related to the shooting environment of the video acquired by the video acquisition function, a selection function for selecting an analysis model based on the shooting environment information acquired by the shooting environment acquisition function, a motion analysis function for analyzing the subject's movements from the video using the analysis model selected by the selection function, and an output function for outputting the analysis results of the subject's movements obtained by the motion analysis function. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating a schematic configuration example of an information processing system according to an embodiment. [Diagram 2]FIG. 2 is a diagram illustrating an example of the configuration of functional blocks of the information processing device according to the embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of the configuration of an analysis model group according to the embodiment. [Figure 4A] FIG. 4 is a diagram showing an example of the configuration of first pattern information according to the embodiment. [Figure 4B] FIG. 11 is a diagram showing an example of the configuration of second pattern information according to the embodiment. [Figure 4C] FIG. 11 is a diagram showing an example of the configuration of third pattern information according to the embodiment. [Figure 4D] FIG. 13 is a diagram showing an example of the configuration of fourth pattern information according to the embodiment. [Diagram 5] 5A and 5B are diagrams illustrating an example of a selection process performed by a selection unit according to the embodiment. [Figure 6] FIG. 13 is a diagram illustrating an example of the configuration of a correspondence table of a selection unit according to the embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of a process performed by an analysis model according to the embodiment. [Figure 8] FIG. 4 is a diagram showing an example of a procedure of a process performed by the information processing device according to the embodiment. [Figure 9A] FIG. 2 is a diagram showing an example of frame a1 of a video with a high frame rate according to an embodiment. [Figure 9B] FIG. 13 is a diagram showing an example of frame a2 of a video with a high frame rate according to the embodiment. [Figure 9C] FIG. 13 is a diagram showing an example of frame a3 of a video with a high frame rate according to the embodiment. [Figure 9D] FIG. 13 is a diagram showing an example of frame a4 of a video with a high frame rate according to an embodiment. [Figure 9E] FIG. 13 is a diagram showing an example of frame a5 of a video with a high frame rate according to the embodiment. [Figure 9F] FIG. 13 is a diagram showing an example of frame a6 of a video with a high frame rate according to the embodiment. [Figure 9G] FIG. 13 is a diagram showing an example of frame b1 of a video with a low frame rate according to the embodiment. [Figure 9H] FIG. 13 is a diagram showing an example of frame b2 of a video with a low frame rate according to an embodiment. [Figure 9I] FIG. 13 is a diagram showing an example of frame b3 of a video with a low frame rate according to the embodiment. [Figure 10A] FIG. 4 is a diagram showing an example of first moving image information having a high resolution according to the embodiment. [Figure 10B] 11 is a diagram showing an example of second moving image information having a low resolution according to the embodiment. FIG. [Figure 11A] FIG. 17 is a diagram showing an example of eleventh moving image information in which the brightness is not constant outdoors on a sunny day according to the embodiment. [Figure 11B] FIG. 13 is a diagram showing an example of twelfth moving image information in which the brightness is not constant outdoors on a cloudy day according to the embodiment. [Figure 12] FIG. 13 is a diagram showing an example of 21st moving image information taken outdoors on a rainy day according to an embodiment. [Figure 13A] FIG. 13 is a diagram showing an example of 31st moving image information outdoors in the morning according to the embodiment. [Figure 13B] FIG. 13 is a diagram showing an example of 32nd moving image information taken outdoors during the day in an embodiment. [Figure 13C] A figure showing an example of 33rd moving image information taken outdoors in the evening according to the embodiment. [Figure 14A] FIG. 13 is a diagram showing an example of the 41st moving image information in which the subject's clothing and the background are the same color in the embodiment. [Figure 14B] A figure showing an example of 42nd moving image information in which the subject's clothing and the background are different colors in an embodiment. [Figure 15] FIG. 13 is a diagram showing an example of the 51st moving image information in which a person other than the subject is captured in the image according to the embodiment. [Figure 16A] FIG. 13 is a diagram showing an example of 61st moving image information manually captured by a person according to the embodiment. [Figure 16B] A figure showing an example of 62nd video information shot using a tripod according to the embodiment. [Figure 17] FIG. 13 is a diagram showing an example of a display using a human skeleton according to the embodiment. [Figure 18] FIG. 11 is a diagram illustrating an example of a process during learning according to the embodiment. [Figure 19] FIG. 10 is a diagram illustrating an example of analysis content in a model under learning according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments will be described with reference to the drawings.

[0010] FIG. 1 is a diagram showing a schematic configuration example of an information processing system 1 according to an embodiment. The information processing system 1 includes an information processing device 11, an imaging device 21, a sensor unit 22, and a display device 31. 1 also shows a first subject B1. The first subject B1 is an example of a subject. The first subject B1 is, for example, a human being. In this embodiment, the first subject B1 is not included in the information processing system 1.

[0011] The photographing device 21 photographs the first subject B1 as a moving image. As a result, the photographing device 21 acquires a moving image in which the first subject B1 appears. The first subject B1 becomes a subject of the moving image. Here, the photographing device 21 is configured using, for example, a camera having a function of photographing a moving image. As an example, a camera such as a smartphone carried by the first subject B1 or another person may be used as the photographing device 21.

[0012] In this embodiment, moving images are treated as information in a predetermined format and processed in the information processing system 1. In this embodiment, the information is also called image information. Furthermore, in this embodiment, the information processing device 11 processes information such as moving images in the form of digital data. However, as another example, a part or all of the processing target may be in the form of an analog signal.

[0013] The sensor unit 22 includes one or more sensors that detect a predetermined index value. The predetermined index value may be, for example, an index value related to the movement of the first subject B1, or may be an index value related to another event. Here, some or all of the sensors may be attached to the body of the first subject B1. In this embodiment, the sensor unit 22 is used during learning of the analysis model, but may be used during other processing.

[0014] The information processing device 11 receives video information acquired by the image capturing device 21 as necessary. The information processing device 11 receives as input the detection results from one or more sensors included in the sensor unit 22 as necessary. The information processing device 11 outputs information to be displayed to the display device 31 as necessary.

[0015] In this embodiment, the information processing device 11 learns an analysis model based on video information input from the imaging device 21 and detection results input from the sensor unit 22. The analysis model is a machine learning model that performs calculations related to the movement of the first subject B1. The information processing device 11 performs an analysis of the movement of the first subject B1 based on the video information input from the imaging device 21 and the trained analysis model. At this time, for example, a detection result input from the sensor unit 22 may be used in the analysis.

[0016] The display device 31 has a screen and outputs and displays information on the screen. In this embodiment, the display device 31 outputs and displays the information to be displayed that is input from the information processing device 11 on the screen.

[0017] Here, in this embodiment, the photographing device 21, the sensor unit 22 and the display device 31 are provided as separate devices from the information processing device 11. However, as another example, some or all of the functions of the photographing device 21, the sensor unit 22 and the display device 31 may be integrated into the information processing device 11.

[0018] FIG. 2 is a diagram showing an example of the functional block configuration of the information processing device 11 according to the embodiment. In this embodiment, the information processing device 11 is configured using a computer. The information processing device 11 is, for example, a desktop computer, but may be, for example, a notebook personal computer, a tablet terminal, a smartphone, or a computer of another type. For example, the information processing device 11 may be a device that is fixedly installed at a specified location such as a sports facility and processes video of subjects using the facility, or a smartphone or the like that can be carried by the user.

[0019] The information processing device 11 includes an input unit 111 , an output unit 112 , a communication unit 113 , a storage unit 114 , and a control unit 115 . The control unit 115 includes an acquisition unit 151 , a selection unit 152 , a motion analysis unit 153 , a shooting environment proposal unit 154 , an output control unit 155 , and a learning control unit 156 . The acquisition unit 151 includes a video acquisition unit 171 and a shooting environment acquisition unit 172 .

[0020] The input unit 111 performs a process of inputting information. The input unit 111 performs, for example, a process of inputting information in response to a user's operation, or a process of inputting information from another device. 1, the other devices are an image capturing device 21 and a sensor unit 22. In addition, the other devices may be, for example, a portable recording device. The input unit 111 may include an operation unit. The operation unit receives the contents of an operation performed by a user and performs a process of inputting information in accordance with the user's operation. The operation unit may be, for example, a keyboard and a mouse. It should be noted that the user of the information processing device 11 may be, for example, the first subject B1, or may be another person.

[0021] The output unit 112 performs a process of outputting information. The output unit 112 performs, for example, a process of outputting information to a user, or a process of outputting information to another device. 1, the other device is a display device 31. The other device may also be, for example, a portable recording device. As a mode for outputting information, for example, a mode for outputting sound such as voice may be used together with the display output or independently of the display output.

[0022] The communication unit 113 communicates with other devices. In this embodiment, the communication unit 113 is shown separately from the input unit 111 and the output unit 112 . However, for example, the functions of the communication unit 113 may be included in the functions of the input unit 111 and the output unit 112 .

[0023] The storage unit 114 stores information. In this embodiment, the memory unit 114 may store any information, such as a program that defines the processing, parameters used in the processing, a machine learning model, information before processing, information during processing, or information after processing. The information may be in any format, and in this embodiment includes video. The video may also be called a moving image.

[0024] In the present embodiment, the storage unit 114 stores an analytical model group A1, video information C1, imaging environment information C2, and subject information C3. The analytical model group A1 includes a plurality of analytical models. The video information C1 includes one or more pieces of video information. The image capturing environment information C2 includes information related to the image capturing environment. The subject information C3 includes information about the first subject B1.

[0025] The control unit 115 performs various types of processing and control. The control unit 115 has a processor such as a CPU (Central Processing Unit), and performs various types of processing and control by executing a predetermined program using the processor. The program may be stored in the storage unit 114, for example.

[0026] The acquisition unit 151 acquires information. In the present embodiment, the acquisition unit 151 may acquire, for example, one or more of information input by the input unit 111, information received by the communication unit 113, or information stored in the storage unit 114. The video acquisition unit 171 acquires a video. In this embodiment, the video is treated as video information. The image-capturing environment acquisition unit 172 acquires image-capturing environment information, which is information related to the image-capturing environment of a video. In this embodiment, the image capturing environment information includes video characteristic information and environmental status information.

[0027] In this embodiment, for the sake of convenience of explanation, the functions of the video acquisition unit 171 and the shooting environment acquisition unit 172 possessed by the acquisition unit 151 are shown. However, for example, these functions may be inseparably integrated as the acquisition unit 151, and it may be considered that the functions of the acquisition unit 151 acquire various pieces of information required.

[0028] The selection unit 152 selects one analysis model to be used for analyzing the moving image acquired by the moving image acquisition unit 171, from among the multiple analysis models included in the analysis model group A1. In this embodiment, the selection unit 152 selects the analysis model based on the shooting environment information acquired by the shooting environment acquisition unit 172. Here, the selection unit 152 may select the analysis model based on, for example, the image capture environment information and the subject information acquired by the image capture environment acquisition unit 172. That is, the selection unit 152 selects one analysis model from among a plurality of analysis models based on one or more of the video characteristic information, the environmental situation information, and the subject information. Furthermore, in this embodiment, there may be cases where the selection unit 152 selects to suggest an improvement to the shooting environment without selecting an analysis model.

[0029] The motion analysis unit 153 analyzes the motion of the first subject B1 appearing in the video, using the analysis model selected by the selection unit 152. In this analysis, the motion of the first subject B1 during the motion is analyzed.

[0030] Here, various analyses may be performed as the analysis of the movement, for example, analysis of information on one or more joints in the body of the first subject B1 may be performed. As the information on the joint, for example, information on one or more of the position of the joint, a predetermined angle of the joint, the movement of the joint, the moving speed of the joint, or the rotation speed of the joint may be used. The analysis of the movement may involve, for example, identifying a person corresponding to the subject to be analyzed from a video and extracting a number of feature points representing the posture of the person. The feature points may include, for example, joints. Furthermore, the analysis of the movement may involve, for example, analysis of information relating to the skeleton of the body of the first subject B1. Note that the motion analysis section 153 may take the analysis result obtained by the analytical model as the final analysis result, or may perform further analysis based on the analysis result obtained by the analytical model.

[0031] The imaging environment suggestion unit 154 makes suggestions for improving the imaging environment. The photographing environment suggestion unit 154 may make a suggestion for improving the photographing environment by, for example, outputting information proposing improvement of the photographing environment from the output unit 112 via the output control unit 155. Here, as the proposal for the shooting environment, for example, when the shooting environment is determined to be bad, a proposal for improving the shooting environment is used.

[0032] The output control unit 155 controls the information output by the output unit 112 . In this embodiment, the output control unit 155 controls the process of displaying information relating to the analysis result by the motion analysis unit 153 on the screen of the display device 31 via the output unit 112. Moreover, in this embodiment, the output control unit 155 controls a process of displaying information proposing improvements to the shooting environment by the shooting environment proposing unit 154 on the screen of the display device 31 via the output unit 112.

[0033] The learning control unit 156 controls the learning process of each analytical model included in the analytical model group A1. The learning process is a machine learning process. In this embodiment, the learning control unit 156 controls the learning process of the analytical model, for example, based on a video acquired by the imaging device 21 and a detection result acquired by the sensor unit 22. In this case, for example, information on the video may be used as input information for the analytical model, and information based on the detection result may be used as correct answer data in the learning process of the analytical model. The correct answer data may be called, for example, teacher data.

[0034] FIG. 3 is a diagram showing an example of the configuration of the analysis model group A1 according to the embodiment. The analytical model group A1 includes a plurality of analytical models, that is, n analytical models. In the example of FIG. 3, a first analytical model D1 to an n-th analytical model Dn are shown as these n analytical models.

[0035] Here, various analytical models may be used as the multiple analytical models, and any analytical model may be used, such as the α1 analytical model which has the most analysis points among the multiple analytical models, the α2 analytical model which performs analysis at fewer analysis points than the α1 analytical model by adjusting the brightness of the video to lower the brightness, the α3 analytical model which performs analysis at fewer analysis points than the α1 analytical model and the α2 analytical model, the α4 analytical model which adjusts the contrast of the video to perform analysis at fewer analysis points than the α1 analytical model, and the α5 analytical model which adjusts the brightness of the video to lower the brightness and also adjusts the contrast of the video to perform analysis at fewer analysis points than the α1 analytical model.

[0036] Four different examples of pattern information are shown using FIG. 4A to FIG. 4D. FIG. 4A is a diagram showing an example of the configuration of the first pattern information T1 according to the embodiment. The first pattern information T1 includes photographing environment information and subject information. The shooting environment information includes video characteristic information and shooting situation information. The names of each piece of information are given for the convenience of explanation, and may be called by other names. In the example of FIG. 2, the image-taking environment information C2 and the subject information C3 correspond to the image-taking environment information and the subject information in the first pattern information T1.

[0037] The first pattern information T1 includes, as moving image characteristic information, an evaluation value of the frame rate and an evaluation value of the resolution. Here, the evaluation value of the frame rate may be, for example, high, normal, or low. In the example of Fig. 4A, the frame rate is managed as high. For example, a frame rate less than 30 fps is managed as low, a frame rate between 30 fps and 60 fps is managed as normal, and a frame rate higher than 60 fps is managed as high. Similarly, the evaluation value of the resolution may be, for example, high, normal, or low. In the example of Fig. 4A, the resolution is managed as high. For example, a resolution less than 720 is managed as low resolution, a resolution equal to or greater than 720 and less than 1080 is managed as normal resolution, and a resolution equal to or greater than 1080 is managed as high resolution. As another example, the frame rate may be expressed as a numerical value, and similarly, the resolution may be expressed as a numerical value.

[0038] The first pattern information T1 includes, as shooting situation information, information indicating whether the shooting is indoors or outdoors, information indicating the weather, information indicating the time, information indicating clothing, information indicating reflections, and information indicating blur.

[0039] The information indicating whether the location is indoors or outdoors can be either "indoors" or "outdoors." In the example of FIG. 4A, the location is managed as being indoors.

[0040] The information representing the weather is, for example, information such as sunny, cloudy, rainy, etc. In addition, when the user is indoors and is not affected by the weather, the information representing the weather may be information representing that the user is indoors and therefore does not have any influence. In the example of Fig. 4A, the weather outdoors is rainy, but the user is indoors and therefore does not have any influence.

[0041] The information representing the time is, for example, information such as morning, daytime, and evening. Note that, in the case of being indoors and not affected by time, the information representing the time may be information representing that it is irrelevant since it is indoors. For example, the period from sunrise to 4 hours after sunrise is considered to be morning, the period from 4 hours after sunrise to 10 hours after sunrise is considered to be daytime, and the period from 10 hours after sunrise to sunset is considered to be evening. In the example of FIG. 4A, although it is daytime, it is managed that it is irrelevant since it is indoors.

[0042] The information representing the clothes is, for example, information indicating whether the predetermined clothes of the subject are the same color as the background or a different color. In the example of Fig. 4A, it is managed that the predetermined clothes of the subject are a different color from the background.

[0043] The information representing the reflection is, for example, information indicating whether or not an object other than the subject is reflected in the image. In the example of Fig. 4A, it is managed that there is no reflection. Here, the object other than the subject may be, for example, any object including a living thing such as a person, or may be limited to a person by performing image recognition by the shooting environment acquisition unit 172 or the like to determine whether an object appearing in the video is a person or not.

[0044] The information representing the blur is, for example, information indicating whether or not there is blur when shooting a moving image. In the example of Fig. 4A, it is managed that there is no blur.

[0045] The first pattern information T1 includes, as subject information, information indicative of the subject's age and information indicative of the subject's exercise experience.

[0046] The information indicating the age of the subject is, for example, numerical information indicating the age of the subject. In the example of Fig. 4A, the age of the subject is managed as 30 years old. The information indicating the subject's exercise experience is, for example, information indicating the subject's exercise experience in the form of a numerical value, etc. In the example of Fig. 4A, the subject's exercise experience is managed as 10 years.

[0047] FIG. 4B is a diagram showing an example of the configuration of the second pattern information T2 according to the embodiment. As the second pattern information T2, information for each item similar to that of the first pattern information T1 is stored.

[0048] In the example of FIG. 4B, the frame rate is managed as low. In the example of FIG. 4B, the resolution is managed as high. In the example of FIG. 4B, the information indicating whether the location is indoors or outdoors is managed as being outdoors. In the example of FIG. 4B, the information indicating the weather indicates that the weather outside is sunny. In the example of FIG. 4B, the information indicating the time is managed as being daytime. In the example of FIG. 4B, the information representing the clothing is managed such that a specific clothing item of the subject is a different color from the background. In the example of FIG. 4B, the information representing the reflection indicates that there is no reflection. In the example of FIG. 4B, the information representing the blurring is managed as "no blurring."

[0049] In the example of FIG. 4B, the information indicating the age of the subject indicates that the subject is 40 years old. In the example of FIG. 4B, the information indicating the exercise experience of the subject is managed such that the subject has 5 years of exercise experience.

[0050] FIG. 4C is a diagram showing an example of the configuration of the third pattern information T3 according to the embodiment. As the third pattern information T3, information for each item similar to that of the first pattern information T1 is stored.

[0051] In the example of FIG. 4C, the frame rate is managed as high. In the example of FIG. 4C, the resolution is managed as low. In the example of FIG. 4C, the information indicating whether the location is indoors or outdoors is managed as being outdoors. In the example of FIG. 4C, the information indicating the weather indicates that the weather outdoors is sunny. In the example of FIG. 4C, the information indicating the time is managed as being daytime. In the example of FIG. 4C, the information representing the clothing is managed such that a specific clothing item of the subject is a different color from the background. In the example of FIG. 4C, the information representing the reflection indicates that there is no reflection. In the example of FIG. 4C, the information representing the blurring is managed as "no blurring."

[0052] In the example of FIG. 4C, the information indicating the age of the subject is managed as being 20 years old. In the example of Fig. 4C, the information representing the exercise experience of the subject is managed as "no experience in exercise." Note that "no experience" may be managed as "0 years of experience."

[0053] FIG. 4D is a diagram showing an example of the configuration of the fourth pattern information T4 according to the embodiment. As the fourth pattern information T4, information for each item similar to that of the first pattern information T1 is stored.

[0054] In the example of FIG. 4D, the frame rate is managed as low. In the example of FIG. 4D, the resolution is managed as low. In the example of FIG. 4D, the information indicating whether the location is indoors or outdoors is managed as being outdoors. In the example of FIG. 4D, the information indicating the weather indicates that the weather outside is sunny. In the example of FIG. 4D, the information indicating the time is that it is morning. In the example of FIG. 4D, the information representing the clothing is managed such that the subject's specific clothing is the same color as the background. In the example of FIG. 4D, the information representing the reflection is managed as the fact that there is a reflection. In the example of FIG. 4D, the fact that there is blur is managed as information representing blur.

[0055] In the example of FIG. 4D, the information indicating the subject's age indicates that the subject is 50 years old. In the example of FIG. 4D, the information representing the exercise experience of the subject is managed such that the subject has one year of exercise experience.

[0056] FIG. 5 is a diagram illustrating an example of the selection performed by the selection unit 152 according to the embodiment. The selection unit 152 selects one analysis model based on one or more pieces of information included in the imaging environment information and the subject information. In the example of FIG. 5, the selection unit 152 selects a first analysis model D1 for a moving image that corresponds to the shooting environment information and subject information defined in the first pattern information T1. In the example of FIG. 5, the selection unit 152 selects the second analysis model D2 for a moving image that corresponds to the imaging environment information and the subject information defined in the second pattern information T2. In the example of FIG. 5, the selection unit 152 selects the third analysis model D3 for a moving image that corresponds to the shooting environment information and subject information defined in the third pattern information T3.

[0057] In addition, in the example of Figure 5, the selection unit 152 outputs a shooting environment change instruction E1, which is an instruction to change the shooting environment, to the shooting environment proposal unit 154 for a video that corresponds to the shooting environment information and subject information specified in the fourth pattern information T4. In this embodiment, when the photographing environment suggestion unit 154 receives a photographing environment change instruction E1 from the selection unit 152, the photographing environment suggestion unit 154 makes a suggestion for improving the photographing environment.

[0058] FIG. 6 is a diagram showing an example of the configuration of the correspondence table T11 of the selection unit 152 according to the embodiment. In this embodiment, the selection unit 152 selects an analysis model or the like by referring to the correspondence table T11. Here, the correspondence table T11 may be stored in advance in, for example, the storage unit 114. Note that the contents of the correspondence table T11 may be rewritable by, for example, a user at any time.

[0059] The correspondence table T11 stores feature information and selection numbers in association with each other. In this embodiment, the feature information is one or more pieces of information included in the imaging environment information and the subject information. The selection number is a number that indicates the selection result. In this embodiment, each selection number corresponds to one analysis model or to an instruction to change the shooting environment. In the example of FIG. 6, "aaa" and the like written in the "Characteristic Information" column of the correspondence table T11 represent each piece of characteristic information, but the description of the specific content of the characteristic information is omitted.

[0060] Note that, here, an example configuration has been shown in which the selection unit 152 selects an analysis model, etc., using the correspondence table T11. However, as another example configuration, the selection unit 152 may be configured using a machine learning model. For example, the selection unit 152 may be configured using a machine learning model that outputs a calculation result that specifies whether to perform a specified analysis model or an instruction to change the shooting environment in response to input of specified feature information. The machine learning model may be called by any name.

[0061] FIG. 7 is a diagram illustrating an example of a process performed by the analysis model D according to the embodiment. In the example of FIG. 7, one analytical model out of the n analytical models is shown as analytical model D. The analysis model D receives the video information C1 and analyzes the motion of the first subject B1 appearing in the video information C1. Here, in addition to the video information C1, the analysis model D may input, for example, a part or all of one or both of the imaging environment information C2 and the subject information C3. The analysis model D obtains the analysis result 211 .

[0062] FIG. 8 is a diagram showing an example of a procedure of processing performed by the information processing device 11 according to the embodiment.

[0063] In step S1, in the information processing device 11, the moving image acquisition section 171 acquires moving image information, and the imaging environment acquisition section 172 acquires imaging environment information. Then, the information processing device 11 proceeds to the process of step S2. In addition, in a case where subject information is used, the acquiring unit 151 may acquire the subject information.

[0064] In step S2, the selection unit 152 selects one analysis model in the information processing device 11. Here, in this embodiment, there may be a case where the selection unit 152 selects to issue an instruction to change the imaging environment. Then, the information processing device 11 proceeds to the process of step S3.

[0065] In step S3, if the information processing device 11 does not propose a shooting environment, the result of step S3 is NO, and the process proceeds to step S4. On the other hand, in the information processing device 11, if a shooting environment is to be proposed, the result is YES in step S3, and the process proceeds to step S6.

[0066] In step S4, in the information processing device 11, the motion analysis section 153 analyzes the motion of the first subject B1 using the selected analysis model. Then, the information processing device 11 proceeds to the process of step S5.

[0067] In step S5, in the information processing device 11, the output control unit 155 performs control to output information related to the analysis result. Then, the information processing device 11 ends the process of this flow.

[0068] In step S6, in the information processing device 11, the photographing environment proposal unit 154 performs control to output the contents of the proposal for the photographing environment. Then, the information processing device 11 ends the process of this flow.

[0069] Here, in the examples of FIGS. 1 to 8, the selection unit 152 may select to make a suggestion for improving the shooting environment, but a configuration in which such a selection is not made may be used. For example, the selection unit 152 may be configured to select one analysis model from among a plurality of analysis models. In this case, the selection unit 152 may not have a function of selecting to make a proposal for improving the shooting environment, may not have the processes of steps S3 and S6 in the processing flow shown in Fig. 8, and may perform the process of step S4 after the process of step S2.

[0070] A specific example of the shooting environment will be given. In this embodiment, the captured video shows the motion of a specific exercise performed by the subject. The motion is not particularly limited, and may be, for example, a motion that has a set sequence of motion but may vary from person to person, such as a golf swing. In this embodiment, the moving image to be analyzed may be specified by the user of the information processing device 11, or may be automatically specified by the information processing device 11, for example. In the golf course example, a round of golf is played outdoors, and facilities such as retail stores are located indoors. In the case of golf, the subject plays the game with a certain tool, which is a golf club.

[0071] Furthermore, the video to be analyzed may be, for example, a video based on multiple actions of a subject filmed in a similar shooting environment, or a video that is the result of averaging, etc., multiple similar actions performed by the same subject. As a specific example, when a subject performs a golf swing motion multiple times in a similar shooting environment, a video based on the results of these multiple swings may be used as an analysis target. Processing such as averaging the images may be performed by, for example, the acquisition unit 151 in the control unit 115.

[0072] In the analysis process according to the present embodiment, for example, information on a plurality of different parts, such as the positions or angles of a plurality of joints of a human, may be identified based on a video in which a specific action is captured. Note that the process of identifying a specific part from the video may utilize, for example, a known image recognition process. Here, the video includes, for example, time-series change data of a body part involved in at least one specific movement.

[0073] In this embodiment, when illustrating a video of a subject's exercise, the subject's body is illustrated, but the tools used by the subject are not illustrated. In this embodiment, the subject holds a golf club and performs a golf swing.

[0074] The frame rate of a moving image, which is an example of a shooting environment, will be described with reference to FIGS. 9A to 9F and 9G to 9I. In this example, the frame rate of the moving image shown in FIGS. 9A to 9F is higher than the frame rate of the moving image shown in FIGS. 9G to 9I.

[0075] 9A to 9F show an example of six consecutive frames that make up one video. Each of the frames shown in FIG. 9A to FIG. 9F, or a group of two or more frames, may be used as information to be displayed.

[0076] FIG. 9A is a diagram showing an example of a1-th frame Fa1 of a moving image with a high frame rate according to the embodiment. FIG. 9B is a diagram showing an example of a2-th frame Fa2 of a moving image with a high frame rate according to the embodiment. FIG. 9C is a diagram showing an example of a3-th frame Fa3 of a moving image with a high frame rate according to the embodiment. FIG. 9D is a diagram showing an example of a4th frame Fa4 of a moving image with a high frame rate according to the embodiment. FIG. 9E is a diagram showing an example of a5th frame Fa5 of a moving image with a high frame rate according to the embodiment. FIG. 9F is a diagram showing an example of a6th frame Fa6 of a moving image with a high frame rate according to the embodiment.

[0077] The a1 frame Fa1 to the a6 frame Fa6 capture the movement of the eleventh subject B11. The eleventh subject B11 is an example of a subject. 9A to 9F, the movement of the position of a predetermined joint of the 11th subject B11 is shown as the a1 joint Ha1, the a2 joint Ha2, the a3 joint Ha3, the a4 joint Ha4, the a5 joint Ha5, and the a6 joint Ha6 in the order of FIG. 9A to FIG. 9F. In this example, the a1 joint Ha1 to the a6 joint Ha6 represent the same joint, and FIG. 9A to FIG. 9F show the change in the position of the joint.

[0078] 9G to 9I show an example of three consecutive frames that make up one video. Each of the frames shown in FIG. 9G to FIG. 9I, or a group of two or more frames, may be used as information to be displayed.

[0079] FIG. 9G is a diagram showing an example of the b1-th frame Fb1 of a moving image having a low frame rate according to the embodiment. FIG. 9H is a diagram showing an example of the b2-th frame Fb2 of a moving image with a low frame rate according to the embodiment. FIG. 9I is a diagram showing an example of the b3-th frame Fb3 of a moving image with a low frame rate according to the embodiment.

[0080] The b1 frame Fb1 to the b3 frame Fb3 capture the movement of the 21st subject B21. The 21st subject B21 is an example of a subject. 9G to 9I, the movement of the position of a predetermined joint of the 21st subject B21 is shown as the b1 joint Hb1, the b2 joint Hb2, and the b3 joint Hb3 in the order of FIG. 9G to FIG. 9I. In this example, the b1 joint Hb1 to the b3 joint Hb3 represent the same joint, and FIG. 9G to FIG. 9I show the change in the position of the joint.

[0081] It should be noted that the a1 joint Ha1 to the a6 joint Ha6 in the example of FIGS. 9A to 9F and the b1 joint Hb1 to the b3 joint Hb3 in the example of FIGS. 9G to 9I are the same joints. In the example of Figure 9A and the example of Figure 9G, the subject is in a position where he is holding a golf club and is ready to go, and in the example of Figure 9D, the subject is in a position where he is swinging the golf club over his shoulder.

[0082] In this example, for a similar series of motions, such as a golf swing, a video consisting of six frames is acquired in the examples of Figures 9A to 9F, and a video consisting of three frames is acquired in the examples of Figures 9G to 9I. In other words, the frame rate of the video in the examples of Figures 9A to 9F is twice as high as the frame rate of the video in the examples of Figures 9G to 9I. For example, when motion analysis is performed from a video in a real environment, various types of filming equipment may be used. In general, the frame rate can have a significant effect on the performance of an inference device that processes a time series system.

[0083] Here, when the frame rate of the video information is high, it is suitable for processing in which analysis is performed by taking detailed analysis points, for example. In other words, in this case, it is possible to secure a sufficient number of frames for analysis, and if the analysis is performed without taking detailed points, for example, the analysis accuracy may decrease. On the other hand, when the frame rate of video information is low, it is suitable for processing to perform analysis without taking detailed analysis points, for example. In other words, in this case, compared to video information with a high frame rate, taking detailed analysis points may result in taking analysis points at overlapping positions, which may result in erroneous analysis results and reduced analysis accuracy.

[0084] For example, multiple machine learning models for analyzing the subject's motion may be prepared according to the frame rate of the video information. As a specific example, a machine learning model suitable for video information with a high frame rate and a machine learning model suitable for video information with a low frame rate may be prepared separately. Also, for example, when video information with a low frame rate is processed, the reliability may be determined to be lower than when video information with a high frame rate is processed. In this case, for example, a process may be performed to suggest improving the shooting environment so as to obtain video information with a higher frame rate. Whether the frame rate is high or low may be determined using a threshold value, for example. Furthermore, three or more classifications may be used as the level of the frame rate. As a specific example, three classifications of frame rate such as high, medium, and low may be used.

[0085] Here, the imaging environment acquisition section 172 may determine the frame rate of the video based on, for example, information on the properties of the video, or may determine the frame rate of the video based on information about the video.

[0086] The resolution of a moving image, which is an example of a shooting environment, will be described with reference to FIGS. 10A and 10B. In this example, the resolution of the moving image shown in FIG. 10A is higher than the resolution of the moving image shown in FIG. 10B. In this example, for the sake of simplicity, each of Fig. 10A and Fig. 10B shows a group of six consecutive frames of a moving image arranged horizontally as moving image information of the moving image. Each frame of the moving image shown in each of Fig. 10A to Fig. 10B, or a group of two or more frames, may be used as information to be displayed.

[0087] FIG. 10A is a diagram showing an example of first moving image information G1 having a high resolution according to the embodiment. In the first video information G1, the exercise of the 31st subject B31 is captured. The 31st subject B31 is an example of a subject. FIG. 10A shows an example of a change in movement of the c-th joint Hc, which is a predetermined joint of the 31st subject B31.

[0088] FIG. 10B is a diagram showing an example of the second moving image information G2 having a low resolution according to the embodiment. In the second video information G2, the exercise of the 41st subject B41 is captured. The 41st subject B41 is an example of a subject. FIG. 10B shows an example of a change in movement of the d-th joint Hd, which is a predetermined joint of the 41st subject B41. It should be noted that the c-th joint Hc in the example of FIG. 10A and the d-th joint Hd in the example of FIG. 10B are the same joint.

[0089] Here, when the resolution of the video information is high, for example, it is suitable for processing to perform analysis by taking fine analysis points. In other words, in this case, since the edges of people appearing in the video can be clearly detected, it is possible to take fine analysis points, and for example, if analysis is performed without taking fine points, the analysis accuracy may decrease. On the other hand, when the resolution of video information is low, for example, it is suitable for processing to perform analysis without taking many analysis points. In other words, in this case, compared to video information with high resolution, the edges of people appearing in the video cannot be detected clearly, so if there are many analysis points, there is a possibility that non-human parts will be mistakenly determined to be human movements. For example, taking many analysis points may increase noise, which may reduce the accuracy of the analysis.

[0090] For example, multiple machine learning models for analyzing the subject's motion may be prepared according to the resolution of the video information. As a specific example, a machine learning model suitable for video information with high resolution and a machine learning model suitable for video information with low resolution may be prepared separately. Also, for example, when low-resolution video information is processed, the reliability may be determined to be lower than when high-resolution video information is processed. In this case, for example, a process may be performed to suggest improving the shooting environment so as to obtain video information with a higher resolution. Whether the resolution is high or low may be determined using a threshold value, for example. Furthermore, three or more divisions may be used as the degree of resolution. As a specific example, three divisions such as high, medium, and low resolution may be used.

[0091] Here, the imaging environment acquisition section 172 may determine the resolution of the video based on, for example, information on the properties of the video, or may determine the resolution of the video based on information on the video.

[0092] With reference to FIGS. 11A and 11B, a description will be given of whether a photographing location, which is an example of a photographing environment, is outdoors or indoors. In this example, a video shot indoors is, for example, similar to the video shown in FIG. 10A. In this example, for the sake of simplicity, each of Fig. 11A and Fig. 11B shows a group of six consecutive frames of a moving image arranged horizontally as moving image information of the moving image. Each frame of the moving image shown in each of Fig. 11A to Fig. 11B, or a group of two or more frames, may be used as information to be displayed.

[0093] FIG. 11A is a diagram showing an example of eleventh moving image information G11 in which the brightness is not constant outdoors on a sunny day according to the embodiment. In the eleventh video information G11, the exercise of the 51st subject B51 is captured. The 51st subject B51 is an example of a subject. FIG. 11A shows an example of a change in movement of a specific joint, the e-th joint He, of the 51st subject B51. In the example of FIG. 11A, the sun K is shown in each frame.

[0094] FIG. 11B is a diagram showing an example of twelfth moving image information G12 in which the brightness is not constant outdoors on a cloudy day according to the embodiment. In the twelfth video information G12, the exercise of the 61st subject B61 is captured. The 61st subject B61 is an example of a subject. FIG. 11B shows an example of a change in movement of the f-th joint Hf, which is a predetermined joint of the 61st subject B61. It should be noted that the e-th joint He in the example of FIG. 11A and the f-th joint Hf in the example of FIG. 11B are the same joint.

[0095] Here, when a video is shot indoors, for example, the brightness is considered to be constant or nearly constant, and it is suitable for processing to perform analysis by taking detailed analysis points. In other words, in this case, for example, if analysis is performed without taking detailed points, the analysis accuracy may decrease. When the brightness during shooting is constant or nearly constant, for example, analysis accuracy can be ensured even if analysis is performed without adjusting the contrast or brightness of the image. On the other hand, when a video is shot outdoors, a shooting environment may occur in which the brightness is not constant throughout a series of videos due to, for example, the weather or the time of shooting. If the brightness during shooting is not constant, for example, it may be necessary to adjust the contrast or brightness of the image before performing analysis. The adjustment may also be called correction.

[0096] For example, multiple machine learning models for analyzing the subject's movement may be prepared depending on whether the shooting location is indoors or outdoors. As a specific example, a machine learning model suitable for video information shot indoors and a machine learning model suitable for video information shot indoors may be prepared separately. For example, multiple machine learning models for analyzing the subject's movement may be prepared depending on whether the brightness at the time of shooting is constant or not. As a specific example, a machine learning model suitable for video information shot under constant brightness and a machine learning model suitable for video information shot under non-constant brightness may be prepared separately.

[0097] Also, for example, when video information shot outdoors is processed, the reliability may be determined to be lower than when video information shot indoors is processed. Also, for example, when processing moving image information in which the brightness at the time of shooting is not constant, the reliability may be determined to be lower than when processing moving image information in which the brightness at the time of shooting is constant. In this case, for example, processing may be performed to suggest improvements to the shooting environment so as to obtain moving image information in which the brightness is more constant. Whether the brightness is constant or not may be determined, for example, by using an upper threshold value and a lower threshold value that define a range in which the brightness is considered to be constant. As a specific example, if the brightness that can vary falls within a predetermined range, it may be considered to be constant, and if it deviates from the predetermined range, it may be considered not to be constant.

[0098] Here, even if the photograph was taken indoors, if the lighting in the room is too bright, too dark, or flashing, for example, the reliability may be determined to be low compared to when the lighting in the room is appropriate. As a specific example, if the shooting location is indoors and the lighting in the room is bright, causing the image to be blown out, the reliability may be determined to be low and a suggestion may be made to improve the shooting environment by lowering the brightness when shooting. As another specific example, if the shooting location is indoors and the lighting in the room is dim, making it difficult to see the subject, etc., the reliability may be determined to be low and a suggestion may be made to improve the shooting environment, such as increasing the brightness when taking the photo.

[0099] With reference to FIG. 12, a description will be given of weather, which is one example of an image capturing environment, when an image capturing location is outdoors, which is one example of an image capturing environment. In this example, when the shooting location is outdoors, a video shot on a sunny day and a video shot on a cloudy day are similar to the video shown in FIG. 11A and the video shown in FIG. 11B, respectively. In this example, for the sake of simplicity, a group of six consecutive frame images in a moving image is shown as moving image information of the moving image in Fig. 12. Each frame in the moving image shown in Fig. 12, or a group of two or more frames, may be used as information to be displayed.

[0100] FIG. 12 is a diagram showing an example of the twenty-first moving image information G21 captured outdoors on a rainy day according to the embodiment. In the 21st video information G21, the exercise of the 71st subject B71 is captured. The 71st subject B71 is an example of a subject. FIG. 12 shows an example of a change in the movement of the g-th joint Hg, which is a predetermined joint of the 71st subject B71. In the example of FIG. 12, in order to simplify the illustration, a reference symbol is attached to an a-th raindrop Qa, which is an example of a raindrop, and the reference symbols of the other raindrops are omitted. It should be noted that the g-th joint Hg in the example of FIG. 12 is the same joint as the e-th joint He in the example of FIG. 11A and the f-th joint Hf in the example of FIG. 11B.

[0101] Here, when the weather at the shooting location is fine, for example, it is suitable for processing to adjust the brightness of the video to be lowered and perform analysis. In other words, when the weather is fine, the brightness is higher than indoors and is prone to whiteout, so it may be necessary to adjust the brightness to be lowered and perform analysis. Also, when the weather at the shooting location is cloudy, for example, it is suitable for processing to adjust the brightness of the video to be increased and perform analysis. In other words, when the weather is cloudy, the brightness is lower than indoors, making it difficult to detect the boundary between the person in the video and the background, so it may be necessary to adjust the brightness to be increased and perform analysis. Also, when the weather at the shooting location is rainy, for example, the brightness of the video is adjusted to be increased, and analysis is suitable for processing without taking many analysis points. In other words, when it is raining, as when it is cloudy, the brightness is lower than indoors, making it difficult to detect the boundary between the person in the video and the background, and furthermore, raindrops in the video become noise during analysis, so the more analysis points there are, the more noise there will be, and the accuracy of the analysis will decrease.

[0102] For example, multiple machine learning models for analyzing the subject's movement may be prepared according to the weather at the shooting location. As a specific example, a machine learning model suitable for video information shot at a sunny location, a machine learning model suitable for video information shot at a cloudy location, and a machine learning model suitable for video information shot at a rainy location may be prepared separately. Also, for example, when video information shot in a rainy location is processed, the reliability may be determined to be lower than when video information shot in a sunny or cloudy location is processed. In this case, for example, a process may be performed to suggest improving the shooting environment so that video information is acquired when the shooting location is sunny or cloudy. In this example, the weather conditions such as sunny, cloudy, and rainy are exemplified, but other weather conditions such as snow may be used.

[0103] As a specific example, if the shooting location is outdoors, it is sunny and there is backlighting, the reliability may be determined to be low, and a suggestion may be made to improve the shooting environment by taking the photo from the opposite direction to avoid backlighting. As another specific example, if the shooting location is outdoors and it is sunny and too bright, causing blown-out highlights, the reliability may be determined to be low and a suggestion may be made to improve the shooting environment by shooting in the shade. As another specific example, if the shooting location is outdoors and it is raining, the reliability may be determined to be low, and a suggestion may be made to improve the shooting environment by taking the photo in a place where there is no rain. As another specific example, if the shooting location is outdoors and it is cloudy, the reliability may be determined to be low, and a suggestion may be made to improve the shooting environment, such as increasing the brightness by using a light or the like.

[0104] An example of an imaging environment in which the times are morning, midday, and evening will be described with reference to FIGS. 13A to 13C. In this example, for the sake of simplicity, each of Figures 13A to 13C shows a group of six consecutive frame images in a moving image arranged horizontally as moving image information for that moving image. Each frame or a group of two or more frames in the moving image shown in each of Figures 13A to 13C may be used as information to be displayed.

[0105] FIG. 13A is a diagram showing an example of the thirty-first moving image information G31 outdoors in the morning according to the embodiment. In the 31st video information G31, the exercise of the 81st subject B81 is captured. The 81st subject B81 is an example of a subject. FIG. 13A shows an example of a change in movement of the i-th joint Hi, which is a predetermined joint of the 81st subject B81. In the example of FIG. 13A, the first sun K1, which is the sun located in the morning position, appears in the background of the video.

[0106] FIG. 13B is a diagram showing an example of the thirty-second moving image information G32 taken outdoors in the daytime according to the embodiment. In the 32nd video information G32, the exercise of the 91st subject B91 is captured. The 91st subject B91 is an example of a subject. FIG. 13B shows an example of a change in movement of the j-th joint Hj, which is a predetermined joint of the 91st subject B91. In the example of FIG. 13B, the second sun K2, which is the sun in the daytime position, appears in the background of the video.

[0107] FIG. 13C is a diagram showing an example of the thirty-third moving image information G33 captured outdoors in the evening according to the embodiment. In the 33rd video information G33, the exercise of the 111th subject B111 is captured. The 111th subject B111 is an example of a subject. FIG. 13C shows an example of a change in movement of the k-th joint Hk, which is a predetermined joint of the 111th subject B111. In addition, in the example of FIG. 13C, the third sun K3, which is the sun located in an evening position, appears in the background of the video. It should be noted that the i-th joint Hi in the example of FIG. 13A, the j-th joint Hj in the example of FIG. 13B, and the k-th joint Hk in the example of FIG. 13C are the same joint.

[0108] Here, if the photograph was taken in the morning, for example, it is suitable for analysis without taking detailed analysis points. In other words, in this case, when the sun is low in the sky, sunlight enters from an angle and the image is easily backlit, resulting in overexposure, so taking detailed analysis points will result in noise. Also, when the image is taken during the day, it is suitable for processing to adjust the brightness to be lowered and perform analysis. In other words, in this case, for example, when it is sunny, the brightness is higher than indoors and is prone to whiteout, so it may be necessary to adjust the brightness to be lowered and perform analysis. Also, when the shooting time is in the evening, for example, it is suitable for processing to adjust brightness and contrast and perform analysis without taking detailed analysis points. That is, in this case, the color of the video shot in the evening is redder than the color of the video shot during the day, so it is necessary to adjust the contrast. Furthermore, the brightness of the video shot in the evening is lower than the brightness of the video shot during the day, so it may be necessary to adjust the brightness to be higher. Also, when the contrast and brightness of the video are adjusted, it becomes difficult to detect the boundary between the human and the background, and taking detailed analysis points may result in noise.

[0109] For example, multiple machine learning models for analyzing the subject's motion may be prepared according to the outdoor shooting time. As a specific example, a machine learning model suitable for video information shot outdoors in the morning, a machine learning model suitable for video information shot outdoors in the daytime, and a machine learning model suitable for video information shot outdoors in the evening may be prepared separately. In addition, for example, when video information in which the outdoor shooting time is a predetermined time is processed, the reliability may be determined to be lower than when video information in which the outdoor shooting time is another predetermined time is processed. In this case, for example, a process of suggesting an improvement of the shooting environment may be performed, such as changing the outdoor shooting time to acquire video information.

[0110] In this example, it is assumed that the morning period is from sunrise to 10:00, the afternoon period is from 10:00 to 16:00, and the evening period is from 16:00 to sunset, but other time periods may be used for each period. Furthermore, in this example, three time divisions, namely, morning, daytime, and evening, are shown, but other divisions, such as night, may also be used. Additionally, the time segments may be defined, for example, by the start and end times of each segment. Furthermore, as time divisions, different time zone divisions may be used for each season, for example. Furthermore, the time divisions may be, for example, divisions that are unrelated to the movement of the sun.

[0111] As a specific example, if the shooting location is outdoors, in the morning or evening, and the lighting is backlit, the reliability may be determined to be low, and a suggestion may be made to improve the shooting environment by taking the photo from the opposite direction so that there is no backlighting. As another specific example, if the shooting location is outdoors, during the day, and it is too bright and the image is blown out, the reliability may be determined to be low and a suggestion may be made to improve the shooting environment by taking the photo in the shade.

[0112] The relationship between the color of the subject's clothing and the background, which is an example of a shooting environment, will be described with reference to FIGS. 14A and 14B. In this example, for the sake of simplicity, each of Fig. 14A and Fig. 14B shows a group of six consecutive frames of a moving image arranged horizontally as moving image information of the moving image. Each frame of the moving image shown in each of Fig. 14A to Fig. 14B, or a group of two or more frames, may be used as information to be displayed.

[0113] FIG. 14A is a diagram showing an example of the 41st moving image information G41 in which the clothing of the subject and the background are the same color according to the embodiment. In the 41st video information G41, the exercise of the 121st subject B121 is captured. The 121st subject B121 is an example of a subject. FIG. 14A shows an example of a change in movement of the oth joint Ho, which is a predetermined joint of the 121st subject B121. 14A, the color of the first glove U1 of the 121st subject B121 is white, which is the same color as the white background part in the video. The part in question is, for example, a whiteboard.

[0114] FIG. 14B is a diagram showing an example of the 42nd moving image information G42 in which the clothing of the subject and the background are different colors according to the embodiment. In the 42nd video information G42, the exercise of the 131st subject B131 is captured. The 131st subject B131 is an example of a subject. FIG. 14B shows an example of a change in movement of the p-th joint Hp, which is a predetermined joint of the 131st subject B131. In the example of FIG. 14B, the color of the second glove U2 of the 131st subject B131 is a color other than white, which is different from the white background color in the video. It should be noted that the oth joint Ho in the example of FIG. 14A and the pth joint Hp in the example of FIG. 14B are the same joint.

[0115] Here, the first glove U1 in the example of FIG. 14A is an example of clothing of the 121st subject B121, and the second glove U2 in the example of FIG. 14B is an example of clothing of the 131st subject B131. In this example, a glove, such as the first glove U1 or the second glove U2, is part of the subject's overall outfit.

[0116] Here, when the clothing of the subject and the background in the video are the same color, for example, it is suitable for processing to perform analysis without taking detailed analysis points. In other words, in this case, it is difficult to detect the boundary between the person and the background in the video in the same color area, so analysis errors are likely to occur and it may be impossible to detect the boundary at all. In this case, even if analysis points are taken finely, noise points will be detected, and the analysis accuracy will decrease. On the other hand, when the clothing of the subject in the video is a different color from the background, for example, it is suitable for processing to perform analysis by selecting analysis points in detail. In other words, in this case, the boundary between the person in the video and the background can be clearly detected, making it possible to perform analysis with sufficient accuracy.

[0117] For example, multiple machine learning models for analyzing the subject's movement may be prepared depending on whether the subject's clothes and the background in the video are the same color or different colors. As a specific example, a machine learning model suitable for video information in which the subject's clothes and the background in the video are the same color and a machine learning model suitable for video information in which the subject's clothes and the background in the video are different colors may be prepared separately. In addition, for example, when video information is processed in which the subject's clothing and the background are the same color, the reliability may be determined to be lower than when video information is processed in which the subject's clothing and the background are different colors. In this case, for example, a process of suggesting an improvement to the shooting environment may be performed to obtain video information in which the subject's clothing and the background are different colors.

[0118] As a specific example, if the color of the subject's clothing and the background color in a recorded video are similar and the reliability is determined to be low, a suggestion may be made to improve the shooting environment by changing at least one of the subject's clothing or the shooting location. In this example, cases where the color of the subject's clothing and the background color are similar include cases where these colors are the same color and cases where these colors are not the same color but are close enough to be indistinguishable. In this way, similar processing may be performed on similar colors that are not the same color. In other words, even if a color is strictly different, it may be excluded from the different colors if it is similar.

[0119] Referring to FIG. 15, a description will be given of an example of a photographing environment, in which a person is photographed. In this example, a moving image without a person in it is, for example, similar to the moving image shown in FIG. 10A. In this example, for the sake of simplicity, a group of six consecutive frame images in a moving image is shown as moving image information of the moving image in Fig. 15. Each frame in the moving image shown in Fig. 15, or a group of two or more frames, may be used as information to be displayed.

[0120] FIG. 15 is a diagram showing an example of the 51st moving image information G51 in which a person other than the target person according to the embodiment is captured. In the 51st video information G51, the exercise of the 141st subject B141 is captured. The 141st subject B141 is an example of a subject. FIG. 15 shows an example of a change in the movement of the q-th joint Hq, which is a predetermined joint of the 141st subject B141. In the example of FIG. 15, a first person V1, who is a person other than the first subject person B141, appears in the video.

[0121] Here, when a person other than the target person is captured in the video, for example, it is suitable for processing to perform analysis without taking detailed analysis points. In other words, in this case, if analysis points are taken in detail, there is a possibility that a person other than the target person will be erroneously analyzed, and the analysis accuracy will decrease. On the other hand, if the video does not include people other than the target person, it is suitable for processing to perform analysis by selecting detailed analysis points. In other words, in this case, since the video does not include people other than the target person, analysis with sufficient accuracy is possible.

[0122] For example, multiple machine learning models for analyzing the motion of a subject may be prepared depending on whether or not a person other than the subject is captured in the video. As a specific example, a machine learning model suitable for video information in which a person other than the subject is captured and a machine learning model suitable for video information in which a person other than the subject is not captured may be prepared separately. Also, for example, when video information in which people other than the target person are captured is processed, the reliability may be determined to be lower than when video information in which people other than the target person are not captured is processed. In this case, for example, a process of suggesting an improvement to the shooting environment may be performed so as to obtain video information in which people other than the target person are not captured.

[0123] Here, in this example, a person other than the subject has been given as an example of an object that appears in the video, but for example, if it is determined that the person is not moving in the video, the impact on the analysis of the subject's movement may be smaller than when it is determined that the person is moving, so processing may be performed assuming that the person is not visible.

[0124] Although the above description has been given of a case where a person other than the subject moves as an example of a case where the background other than the subject moves in a video, the same processing may be performed when other moving objects are captured in the video. The other objects may be, for example, animals, vehicles such as automobiles or airplanes, smoke, raindrops, snow, etc.

[0125] As a specific example, if a person other than the target person is captured in the captured video and the reliability is determined to be low, a suggestion may be made to improve the shooting environment by shooting in a different location where there are no people other than the target person.

[0126] With reference to FIGS. 16A and 16B, a description will be given of a case where blur is an example of a photographing environment. In this example, for the sake of simplicity, each of Fig. 16A and Fig. 16B shows a group of six consecutive frames of a moving image arranged horizontally as moving image information of the moving image. Each frame of the moving image shown in each of Fig. 16A to Fig. 16B, or a group of two or more frames, may be used as information to be displayed.

[0127] FIG. 16A is a diagram showing an example of the 61st moving image information G61 manually captured by a person according to the embodiment. In the 61st video information G61, the exercise of the 151st subject B151 is captured. The 151st subject B151 is an example of a subject. FIG. 16A shows an example of a change in the movement of the r-th joint Hr, which is a predetermined joint of the subject B151. In the example of FIG. 16A, a first line W1 serving as a reference for the position of the shoes of the 151st subject B151 is shown near the feet of the 151st subject B151. In the example of FIG. 16A, the image is captured by a person manually holding the photographing device 21, and blurring occurs at the position where the 151st subject B151 is captured in multiple frames of the video.

[0128] FIG. 16B is a diagram showing an example of the 62nd moving image information G62 captured using a tripod according to the embodiment. In the 62nd video information G62, the exercise of the 161st subject B161 is captured. The 161st subject B161 is an example of a subject. FIG. 16B shows an example of a change in movement of the s-th joint Hs, which is a predetermined joint of the 161st subject B161. In the example of FIG. 16B, an eleventh line W11 serving as a reference for the position of the shoes of the 161st subject B161 is shown near the feet of the 161st subject B161. In the example of FIG. 16B, the image is captured with the image capturing device 21 fixed in position by a tripod, and no blurring occurs in the position where the 161st subject B161 is captured in multiple frames of the video. It should be noted that the r-th joint Hr in the example of FIG. 16A and the s-th joint Hs in the example of FIG. 16B are the same joint.

[0129] Here, when a video is manually shot by a person and is blurred, for example, it is suitable for processing to perform analysis without taking detailed analysis points. In other words, in this case, if analysis points are taken finely, positional changes of people in the shot video due to camera shake or the like will be detected, and there is a possibility that erroneous results will be output. On the other hand, if the video is shot using a tripod and there is no shaking, it is suitable for processing to perform analysis by selecting small analysis points. In other words, in this case, the pivot foot does not shake, and the position of the person in the shot video does not change, making it possible to perform analysis with less noise.

[0130] For example, multiple machine learning models for analyzing the motion of a subject may be prepared depending on whether or not the video is blurred. As a specific example, a machine learning model suitable for video information with blur and a machine learning model suitable for video information without blur may be prepared separately. Also, for example, when blurred video information is processed, the reliability may be determined to be lower than when non-blurred video information is processed. In this case, for example, a process may be performed to suggest improvements to the shooting environment so as to obtain non-blurred video information.

[0131] In this example, it is assumed that there will be blur in the video when the video is shot manually by a person, and there will be no blur when the video is shot using a tripod, but it is not necessary to use a method that determines the presence or absence of blur based on whether the video is shot manually by a person or using a tripod; for example, a method that determines the presence or absence of blur based on the video may be used. Although a tripod is used in this example, other devices may be used to fix the imaging device 21.

[0132] As a specific example, if the captured video is blurred and is determined to have low reliability, a suggestion may be made to improve the shooting environment, such as by fixing the equipment, such as a camera, used to capture the video.

[0133] Here, the shooting environments shown in the above examples are illustrative and not limited to these, and various other shooting environments may be taken into consideration. For example, a case may be taken into consideration where the area in which the subject is captured is too small compared to the angle of view of the video, where the subject's body does not fit within the angle of view of the video and part of the subject's body extends beyond the angle of view, or where the subject's orientation is not appropriate for the direction in which the video is shot. Furthermore, in each of the above examples, a mode in which the position of one joint in the human body is displayed has been shown, but a mode in which the positions of two or more joints are displayed may also be used. Furthermore, in each of the above examples, a mode in which the positions of the joints on the human body are displayed has been shown, but as another example, a mode in which the positions of the joints on the human body are not displayed may be used. Furthermore, the human body does not necessarily have to be displayed precisely, but may be displayed, for example, roughly. Also, a representation of the human skeleton may be used instead of or in addition to the human body.

[0134] FIG. 17 is a diagram illustrating an example of a display using a human skeleton according to the embodiment. 17, the body of the 211th subject B211 is diagrammatically shown, and the skeleton of the 211th subject B211 is diagrammatically shown. The 211th subject B211 is an example of a subject. In the example of Fig. 17, the skeleton is composed of a plurality of bones L. In the example of Fig. 17, a reference symbol is attached to one bone L, and the reference symbols of the other bones are omitted. Also, in the example of Fig. 17, a joint H is shown. Note that in the example of Fig. 17, a reference symbol is attached to one joint H, and the reference symbols of the other joints are omitted.

[0135] For example, as a form of diagrammatically displaying a human being, a skeletal display as shown in FIG. 17 may be used. 17 shows a mode in which both the human body and skeleton are displayed, but as another example, a mode in which the skeleton of the human is displayed without displaying the human body may be used. Also, the skeleton of the human does not necessarily have to be displayed precisely, and may be displayed roughly. Furthermore, in the example of FIG. 17, a mode in which the positions of the joints are displayed on the skeleton is shown, but as another example, a mode in which the positions of the joints are not displayed may be used.

[0136] Learning of the analysis model selection process will be described with reference to FIGS.

[0137] FIG. 18 is a diagram illustrating an example of a process during learning according to the embodiment. Fig. 18 shows a schematic diagram of the learning process using an analytical model pattern 611. Note that Fig. 18 is a schematic diagram for the purpose of explanation, and is not necessarily a strict diagram. 18 shows one analytical model pattern 611, similar processing is performed for a plurality of different analytical model patterns. In the example of FIG. 3, these plurality of analytical model patterns correspond to the patterns of the first analytical model D1 to the n-th analytical model Dn, respectively.

[0138] 18 shows the 311th subject B 311. The 311th subject B 311 is an example of a subject. Nine sensors, a1 sensor a1 to a9 sensor a9, are attached to different predetermined locations on the body of the subject B311. The predetermined locations may be, for example, the positions of the joints. In this example, the machine learning teacher data b1 is generated from some or all of the information of the detection values ​​of the a1 sensor a1 to the a9 sensor a9, or from information obtained by calculation or the like based on the information. In the example of FIG. 1, the a1-th sensor a1 to the a9-th sensor a9 are included in the sensor unit 22.

[0139] In this example, it is assumed that a plurality of m videos are obtained by capturing the movements of the subject B 311. The m videos are captured in different environments. Moreover, m pieces of pattern information, that is, the p1-th pattern information p1-th pattern information pm, are obtained from each of a plurality of m moving images. When each of the p1-th pattern information p1 to pm-th pattern information is input to a predetermined analytical model pattern 611, the q1-th analysis result q1 to the qm-th analysis result qm, which are the respective analysis results, are output from the analytical model pattern 611.

[0140] For each of the q1th analysis result q1 to the qmth analysis result qm, the degree of agreement with the teaching data b1 is indicated as the c1th degree of agreement c1 to the cmth degree of agreement cm. For example, the learning control unit 156 performs learning on the model under learning 651 so that it can identify an image capture environment suitable for the analysis model pattern 611 based on the m number of the q1th analysis result q1 to the qmth analysis result qm. In other words, the learning control unit 156 performs learning so that the model under learning 651 can select an analysis model suitable for the image capture environment.

[0141] By learning the model under learning 651 in this manner, a trained model capable of selecting an analysis model suited to the shooting environment is obtained. In this example, the trained model may be used as a training model for the selection unit 152.

[0142] FIG. 19 is a diagram illustrating an example of the analysis content in the learning model 651 according to the embodiment. 19, the horizontal axis represents a predetermined index, and the vertical axis represents the degree of match, which indicates the degree of effect, as a value from 0% to 100%. On the vertical axis, 0% indicates no effect, the effect increases toward the top, and the effect is maximum at 100%. 19 shows a distribution characteristic 711 that indicates an analysis result when a large amount of pattern information is analyzed for one analytical model pattern 611. Note that the distribution characteristic 711 is an example for the purpose of explanation, and is not an actually obtained result. 19, the predetermined index is, for example, the predetermined parameter itself or a value obtained from the predetermined parameter. In this example, the predetermined index is the p1-th pattern information p1 to the pm-th pattern information pm.

[0143] In the example of Figure 19, the point corresponding to the data of the d1 pattern corresponds to the d1 point d1, the point corresponding to the data of the d2 pattern corresponds to the d2 point d2, and the point corresponding to the data of the d3 pattern corresponds to the d3 point d3. Here, the d1-th pattern may be a pattern of an arbitrary shooting environment, respectively. In this example, the d1-th pattern is a different pattern among the p1-th pattern information p1 to the pm-th pattern information pm, respectively. From the distribution characteristic 711 shown in FIG. 19, the imaging environment that matches the analytical model pattern 611 can be grasped.

[0144] Here, in this example, a learning model that selects an analysis model suitable for the shooting environment has been described, but as another example, a learning model that selects an analysis model suitable for the shooting environment and other information may be used. As another example, a learning model may be used in which there are cases where an analysis model is selected and cases where a suggestion for improving the shooting environment is selected without selecting an analysis model.

[0145] As described above, the information processing device 11 according to this embodiment includes the video acquisition unit 171, the imaging environment acquisition unit 172, the selection unit 152, the motion analysis unit 153, and the output unit 112. The video acquisition unit 171 acquires a video of the subject's exercise. The imaging environment acquisition unit 172 acquires imaging environment information relating to the imaging environment of the video acquired by the video acquisition unit 171. The selection unit 152 selects an analytical model based on the image capture environment information acquired by the image capture environment acquisition unit 172. In the example of Fig. 3, the analytical model is one of the first analytical model D1 to the n-th analytical model Dn. The motion analysis unit 153 uses the analysis model selected by the selection unit 152 to analyze the motion of the subject from the video. The output unit 112 outputs the analysis result of the subject's motion obtained by the motion analysis unit 153 .

[0146] Therefore, in the information processing device 11 according to this embodiment, an analysis result of the subject's movement can be obtained using an analysis model appropriate for the video shooting environment, thereby, for example, improving the accuracy of the analysis result. For example, in the information processing device 11 according to this embodiment, an appropriate analysis model for analyzing the subject's movements can be selected from among a plurality of analysis models according to information about the environment in which the video was shot, thereby appropriately analyzing the subject's movements according to the characteristics of the video, etc.

[0147] For example, an inference device that processes time-series data such as a video is assumed to receive input data similar to the data used for learning, and when a video captured in a different shooting environment is input, the expected accuracy may not be achieved. In contrast, the information processing device 11 according to this embodiment can achieve the expected accuracy by, for example, selecting an optimal analysis model each time and performing inference. In addition, in the information processing device 11 according to this embodiment, for example, it is possible to use various types of imaging equipment for capturing moving images while maintaining high analysis accuracy, thereby increasing the degree of freedom in the imaging equipment or the specifications of the imaging equipment.

[0148] In the information processing device 11 according to this embodiment, for example, the imaging environment information includes information indicating the frame rate of the video. Therefore, in the information processing device 11 according to this embodiment, it is possible to obtain an analysis result of the subject's motion using an analysis model appropriate for the frame rate of the video. Note that the frame rate and other information may be used as the shooting environment information. Furthermore, the frame rate does not necessarily have to be used as the shooting environment information.

[0149] In the information processing device 11 according to this embodiment, for example, the imaging environment information includes information indicating the resolution of a moving image. Therefore, in the information processing device 11 according to the present embodiment, it is possible to obtain an analysis result of the subject's motion by using an analysis model appropriate for the resolution of the video. As the imaging environment information, the resolution and other information may be used. Furthermore, the image capturing environment information does not necessarily have to be the resolution.

[0150] In the information processing device 11 according to the present embodiment, for example, the imaging environment information includes information on the location where the video was captured. Therefore, in the information processing device 11 according to this embodiment, it is possible to obtain an analysis result of the subject's motion using an analysis model suitable for the location where the video was captured. Note that information on the shooting location and other information may be used as the shooting environment information. Furthermore, information regarding the shooting location does not necessarily have to be used as the shooting environment information.

[0151] In the information processing device 11 according to the present embodiment, for example, the imaging environment information includes information indicating whether the location is indoors or outdoors. The imaging environment acquisition unit 172 acquires position information of the subject from an external device that captures the subject's movements, and determines whether the location is indoors or outdoors. Therefore, in the information processing device 11 according to this embodiment, it is possible to obtain an analysis result of the subject's exercise using an analysis model that is appropriate depending on whether the video was shot indoors or outdoors.

[0152] Here, the external device may be, for example, the photographing device 21 shown in Fig. 1. The photographing device 21 may be, for example, a smartphone carried by the subject, that is, the camera function of the smartphone may be used as the photographing function of the photographing device 21. When the photographing device 21 is integrated with the information processing device 11, the information processing device 11 may be, for example, a smartphone carried by the subject. In addition, the function of acquiring location information in the external device may be, for example, a function of a Global Navigation Satellite System (GNSS) such as a Global Positioning System (GPS). In this case, the location of the external device may be regarded as the location of a person carrying the external device. Whether the location where the video was taken is indoors or outdoors may be determined based on information such as a GPS, or may be determined based on the contents of a manual input by the user. Such a determination may be performed by, for example, the photography environment acquisition unit 172.

[0153] In the information processing device 11 according to this embodiment, for example, the imaging environment information includes information about the weather at the location where the video was captured. Therefore, in the information processing device 11 according to this embodiment, it is possible to obtain an analysis result of the subject's exercise using an analysis model suitable for the weather at the location where the video is shot. Note that information regarding the weather at the shooting location and other information may be used as the shooting environment information. Furthermore, information regarding the weather at the shooting location does not necessarily have to be used as the shooting environment information.

[0154] In the information processing device 11 according to this embodiment, for example, the photographing environment acquisition section 172 acquires weather information at the position of the subject and determines the weather. Therefore, in the information processing device 11 according to this embodiment, it is possible to obtain an analysis result of the movement of a subject by using an analysis model suitable for the weather at the position of the subject appearing in the video. Here, the weather information may be, for example, weather information provided by the Japan Meteorological Agency or a private company. The photographing environment acquisition unit 172 may receive the weather information via the communication unit 113, for example, in a wired or wireless manner.

[0155] The weather at the location where the video was shot may be determined based on, for example, information such as a GPS indicating the location of the location and weather information, or may be determined based on the contents of a manual input by the user. Such a determination may be performed, for example, by the shooting environment acquisition unit 172. For example, information such as a GPS indicating the location of the location may be information such as a GPS indicating the location of a subject present at the location. As another example, the photographing environment acquisition unit 172 may determine the weather by performing a predetermined image recognition process on the video.

[0156] In the information processing device 11 according to this embodiment, for example, the imaging environment information includes information on the time when the video was captured. Therefore, in the information processing device 11 according to this embodiment, it is possible to obtain an analysis result of the subject's motion using an analysis model appropriate for the shooting time of the video. Note that information related to the shooting time and other information may be used as the shooting environment information. Furthermore, information related to the shooting time does not necessarily have to be used as the shooting environment information.

[0157] In the information processing device 11 according to this embodiment, for example, the imaging environment acquisition section 172 acquires time information of the position of the subject and determines the time. Therefore, in the information processing device 11 according to this embodiment, it is possible to obtain an analysis result of the movement of a subject by using an analysis model suitable for the shooting time of the position of the subject appearing in a video.

[0158] Here, the time information may be, for example, information on a time signal or information on time based on a GPS signal. The photographing environment acquisition unit 172 may receive the time information by wire or wirelessly via the communication unit 113. In general, the time information has a time difference depending on the country or region. The time at the location where the video was taken may be determined based on, for example, information such as a GPS indicating the location of the location and time information, or may be determined based on the contents of a manual input by a user. Such a determination may be performed, for example, by the imaging environment acquisition unit 172. For example, information such as a GPS indicating the location of the location may be information such as a GPS indicating the location of a subject present at the location.

[0159] In the information processing device 11 according to the present embodiment, for example, the imaging environment information includes information on the background color of the location where the video was captured and the color of the subject's clothing. Therefore, in the information processing device 11 according to this embodiment, it is possible to obtain an analysis result of the subject's movement using an analysis model appropriate for the relationship between the color of the subject's clothing appearing in the video and the background color. As the photographing environment information, information regarding the relationship between the color of the subject's clothing and the background color and other information may be used. Furthermore, the image capturing environment information does not necessarily have to include information indicating the relationship between the color of the subject's clothing and the background color.

[0160] In the information processing device 11 according to the present embodiment, for example, the photographing environment acquisition unit 172 determines whether the background color and the clothing color are the same color based on whether the boundary between the subject and the surroundings can be detected. In other words, if the boundary can be detected, the background color and the clothing color are considered to be different colors, and if the boundary cannot be detected, the background color and the clothing color are considered to be the same color. Therefore, in the information processing device 11 according to this embodiment, an analysis result of the subject's movement can be obtained using an analysis model appropriate for whether or not the color of the subject's clothing in the video is the same as the background color. The boundary between a person and the background in a video may be called, for example, an edge.

[0161] Here, the subject's clothing may be, for example, all of the clothing worn by the subject, or a predetermined part of the clothing. In addition, the shooting environment acquisition unit 172 may determine whether the background color and the color of the clothing are the same color by performing a predetermined image recognition process based on, for example, a video in which the subject appears or an image of a frame of the video.

[0162] In the information processing device 11 according to this embodiment, for example, the imaging environment information includes information on the movement of the background of the location where the video was captured. Therefore, in the information processing device 11 according to this embodiment, it is possible to obtain an analysis result of the subject's movement by using an analysis model appropriate for the movement of the background in the video. Note that information regarding background movement in a video and other information may be used as the shooting environment information. Furthermore, information regarding the movement of the background in the video does not necessarily have to be used as the shooting environment information.

[0163] Here, the movement of the background may be, for example, the movement of an object other than the target person. The object may be, for example, a person other than the target person, an animal, a vehicle such as a car or an airplane, smoke, raindrops, snow, etc. The shooting environment acquisition unit 172 may perform a predetermined image recognition process based on, for example, a video in which the subject appears or an image of a frame of the video, thereby determining whether there is movement in the background or identifying objects moving in the background.

[0164] In the information processing device 11 according to this embodiment, for example, the imaging environment information includes information regarding blurring when a moving image is captured. Therefore, in the information processing device 11 according to this embodiment, it is possible to obtain an analysis result of the subject's movement by using an analysis model appropriate for the blur of the moving image. Note that information regarding blurring of a video and other information may be used as the shooting environment information. Furthermore, information regarding blurring of moving images does not necessarily have to be used as the shooting environment information.

[0165] In the information processing device 11 according to the present embodiment, for example, the photographing environment acquisition unit 172 determines blur based on whether the position of the subject's supporting foot has changed. In other words, if the position of the subject's supporting foot has changed, it is deemed that there is blur, and if the position of the subject's supporting foot has not changed, it is deemed that there is no blur. Therefore, in the information processing device 11 according to this embodiment, it is possible to obtain an analysis result of the subject's motion by using an analysis model appropriate for the presence or absence of blurring of the position of the subject's pivot foot captured in the video.

[0166] Here, the shooting environment acquisition unit 172 may determine whether the position of the subject's pivot foot has changed by performing a predetermined image recognition process based on, for example, a video in which the subject appears or an image of a frame of the video.

[0167] In the information processing device 11 according to this embodiment, for example, the motion analysis unit 153 analyzes the movement of the skeleton during the exercise of the subject. Therefore, the information processing device 11 according to this embodiment can obtain the results of analyzing the movement of the skeleton during the exercise of the subject. It should be noted that analysis of skeletal movement during exercise of the subject is not necessarily required.

[0168] Here, as the skeletal information, for example, information based on a general human skeleton may be used, or skeletal information of the subject's body may be measured and used. Furthermore, the skeleton information does not necessarily have to be strictly accurate, and for example, general skeleton information may be used.

[0169] In the information processing device 11 according to this embodiment, for example, the output unit 112 outputs the analysis result to an external device of the subject. Therefore, the information processing device 11 according to this embodiment can provide the analysis result to an external device of the subject. It should be noted that the analysis results do not necessarily have to be provided to an external device of the subject.

[0170] Here, the external device may be, for example, a smartphone carried by the subject. In this case, for example, a screen display function of the smartphone may be used as a display function of the display device 31.

[0171] In the information processing device 11 according to the present embodiment, for example, the selection unit 152 selects an analysis model based on information on the subject and information on the shooting environment. That is, the selection unit 152 selects an analysis model based on one or more pieces of information on the subject and one or more pieces of information included in the shooting environment information. Therefore, the information processing device 11 according to this embodiment can select an analysis model according to information on the subject. In addition, when selecting an analysis model, information about the subject does not necessarily have to be used.

[0172] Here, the subject information acquisition function of the acquisition unit 151 may acquire, for example, information input by a user as information about the subject. As another example, the subject information acquisition function of the acquisition unit 151 may receive and acquire information about the subject stored in a smartphone or the like carried by the subject from the smartphone or the like.

[0173] In the information processing device 11 according to the present embodiment, for example, the information on the subject includes information indicating the age of the subject and information indicating the exercise experience of the subject. Therefore, the information processing device 11 according to this embodiment can select an analysis model according to the age and exercise experience of the subject.

[0174] Here, the information indicating age may be, for example, a value indicating the age itself, or information indicating a classification such as young, middle-aged, or elderly, or information indicating a classification as either an adult or a child. In addition, various information may be used as information indicating exercise experience. For example, information indicating the number of years of exercise experience may be used without specifying the type of exercise, or information indicating whether or not a specified type of exercise has experience may be used, or information indicating the number of years of experience of a specified type of exercise may be used. Furthermore, as information indicating the exercise experience, information indicating the proficiency of the exercise technique, such as amateur or professional, may be used. In addition, various other information may be used as the information about the subject.

[0175] As a specific example, this embodiment can provide a body motion analysis system that estimates the positions of body joints in 2D or 3D. This body movement analysis system can maintain high analysis accuracy even when video images with different frame rates, etc. are input for each imaging device. For example, the performance of cameras mounted on smartphones or mobile phones often varies greatly depending on the manufacturer or price range. For this reason, when a video captured by a photographing device is input and an estimation is performed using a deep learning model, the performance may be significantly affected by differences in frame rate, etc., especially in network architectures that handle time-series data. Examples of such architectures include RNNs (Recurrent Neural Networks) and TCNs (Temporal Convolution Networks). Therefore, in the body motion analysis system, the inference device is trained in advance under each expected condition of frame rate, etc. Then, the body motion analysis system incorporates a function for selecting an analysis model for each condition of frame rate, etc., by referring to the properties of the input video, etc. With such a selection function, the body motion analysis system can absorb differences in frame rate, etc. due to the shooting device and achieve a certain level of inference accuracy.

[0176] The body motion analysis system includes an analysis model group having a plurality of analysis models, which are time-series reasoner models with different characteristics, and a selection function for selecting an analysis model to be used according to input information. The body motion analysis system selects an optimal analysis model from the group of analysis models according to the input characteristics and performs inference. As a specific example, the body motion analysis system selects an analysis model that has been trained in advance using a video with shooting conditions closest to the frame rate of the video to be analyzed. Note that various conditions of the shooting environment may be used as the conditions of the frame rate, etc. Furthermore, as the conditions of the frame rate, for example, conditions of the subject whose motion is to be estimated may be used in addition to the conditions of the shooting environment.

[0177] In this embodiment, the subject performs a golf swing motion as an example of a sport, but the sport may be any sport, and may be various sports such as baseball, soccer, tennis, etc. As a specific example, the information processing device 11 may analyze a baseball batting form or pitching form, a soccer kicking form, or a tennis swing form. Furthermore, the information processing device 11 according to the present embodiment and the like may be applied to things other than sports, for example, when a subject performs a motion other than in sports.

[0178] In the present embodiment, a case has been shown in which information on the joints of a subject appearing in a video is obtained based on video information, but as another example, information obtained by collecting a plurality of still image frames may be used. Even when such information obtained by collecting a plurality of still image frames is used, it is possible to perform substantially the same processing as when video information is used. Such information obtained by collecting a plurality of still image frames may be used as information on a video.

[0179] In addition, in this embodiment, a case has been shown in which information regarding the joints of a subject appearing in a video is obtained based on information about the video. However, as another example, a configuration may be used in which information regarding the joints of a subject is obtained based on information detected by a sensor. In other words, the movement of the subject's joints, etc. may be identified based on the detection results of a sensor attached to the subject's body, and the movement of the subject's joints, etc. may be identified based on the movement of a tool used by the subject, which is detected by a sensor attached to the tool.

[0180] For example, a configuration may be used in which one or both of the following are performed: information regarding the subject's joints, etc. is obtained based on information detected by a sensor attached to the subject's body, and information regarding the subject's joints, etc. is obtained based on information detected by a sensor attached to a specified location outside the subject. Here, the predetermined part on the outside of the subject is not particularly limited and may be, for example, a predetermined part of a tool used by the subject. The tool may be a tool for a sport played by the subject, such as a golf club, a baseball bat or glove, a soccer ball, or a tennis racket. Additionally, the number of sensors used may be any number greater than or equal to one. As the sensor, any sensor may be used, for example, one or more of an acceleration sensor, a position sensor, an angular velocity sensor, and the like may be used.

[0181] In this way, instead of a camera that captures video of the subject, a sensor that detects information about the subject may be used. Also, for example, a camera that captures a video of a subject may be used together with a sensor that detects information about the subject. In this case, various information about the subject may be acquired based on one or both of the information of the video captured by the camera and the information detected by the sensor. It should be noted that the sensor here refers to a device other than a camera, and various sensors may be used instead of or together with a camera.

[0182] A program for implementing the functions of any of the components of any of the above-described devices may be recorded in a computer-readable recording medium, and the program may be read into a computer system and executed. The term "computer system" as used herein includes hardware such as an operating system or peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CDs (Compact Discs)-ROMs, and storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also refers to storage devices that hold a program for a certain period of time, such as volatile memory in a computer system that serves as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. The volatile memory may be a RAM (Random Access Memory). The recording medium may be a non-transitory recording medium.

[0183] The above program may be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. The "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network such as the Internet or a communication line such as a telephone line. The above program may be for realizing a part of the above-mentioned functions. The above program may be a so-called differential file that can realize the above-mentioned functions in combination with a program already recorded in the computer system. The differential file may be called a differential program.

[0184] The function of any of the components in any of the above-described devices may be realized by a processor. Each process in the embodiment may be realized by a processor that operates based on information such as a program and a computer-readable recording medium that stores information such as a program. The functions of each part of the processor may be realized by individual hardware, or the functions of each part may be realized by integrated hardware. The processor includes hardware, and the hardware may include at least one of a circuit that processes digital signals and a circuit that processes analog signals. The processor may be configured using one or more circuit devices mounted on a circuit board, or one or both of one or more circuit elements. An IC (Integrated Circuit) or the like may be used as the circuit device, and a resistor or a capacitor or the like may be used as the circuit element.

[0185] The processor may be a CPU. However, the processor is not limited to a CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) may be used. The processor may be a hardware circuit using an ASIC (Application Specific Integrated Circuit). The processor may be configured with multiple CPUs, or may be configured with a hardware circuit using multiple ASICs. The processor may be configured with a combination of multiple CPUs and a hardware circuit using multiple ASICs. The processor may include one or more of an amplifier circuit or a filter circuit that processes an analog signal.

[0186] Although the embodiments have been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of this disclosure are also included.

[0187] [Note] Below, <Configuration Example 1> to <Configuration Example 20> are shown. In addition, the lower configuration example may or may not be applied to the higher configuration example. Furthermore, a lower configuration example that is applicable to any of the two or more higher configuration examples may be applied to any of the two or more higher configuration examples, i.e., two or more application examples are generated, and a configuration example that is even lower than the lower configuration example may be applied to any of these two or more application examples.

[0188] <Configuration example 1> A video acquisition unit that acquires a video of the subject's exercise; a photographing environment acquisition unit that acquires photographing environment information related to a photographing environment of the video acquired by the video acquisition unit; a selection unit that selects an analysis model based on the image capture environment information acquired by the image capture environment acquisition unit; a motion analysis unit that analyzes the motion of the subject from the video by using the analysis model selected by the selection unit; an output unit that outputs an analysis result of the motion of the subject obtained by the motion analysis unit; An information processing device comprising:

[0189] <Configuration example 2> The shooting environment information includes information indicating a frame rate of the video. The information processing device described in <Configuration Example 1>.

[0190] <Configuration example 3> The imaging environment information includes information indicating a resolution of the video. The information processing device according to <Configuration Example 1> or <Configuration Example 2>.

[0191] <Configuration Example 4> The shooting environment information includes information regarding a location where the video was shot. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 3>.

[0192] <Configuration Example 5> the photographing environment information includes information indicating whether the location is indoors or outdoors, the photographing environment acquisition unit acquires position information of the subject from an external device that photographs the movement of the subject, and determines whether the location is indoors or outdoors; The information processing device according to <Configuration Example 4>.

[0193] <Configuration Example 6> The shooting environment information includes information about the weather at the location where the video was shot. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 5>.

[0194] <Configuration Example 7> The photographing environment acquisition unit acquires weather information of the location of the subject and determines the weather. The information processing device according to <Configuration Example 6>.

[0195] <Configuration Example 8> The shooting environment information includes information regarding the time when the video was shot. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 7>.

[0196] <Configuration Example 9> The photographing environment acquisition unit acquires time information of the position of the subject and determines the time. The information processing device according to <Configuration Example 8>.

[0197] <Configuration Example 10> The shooting environment information includes information regarding a background color of a location where the video was shot and a clothing color of the subject. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 9>.

[0198] <Configuration Example 11> the photographing environment acquisition unit determines whether the background color and the clothing color are the same color based on whether a boundary between the subject and the surroundings can be detected; The information processing device according to <Configuration Example 10>.

[0199] <Configuration Example 12> The shooting environment information includes information regarding background movement of the location where the video was shot. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 11>.

[0200] <Configuration Example 13> The shooting environment information includes information regarding blurring when the video is shot. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 12>.

[0201] <Configuration Example 14> The photographing environment acquisition unit determines the blur based on whether or not a position of the pivot foot of the subject has changed. The information processing device described in <Configuration Example 13>.

[0202] <Configuration Example 15> The motion analysis unit analyzes skeletal movement during the motion of the subject. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 14>.

[0203] <Configuration Example 16> The output unit outputs the analysis result to an external device of the subject. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 15>.

[0204] <Configuration Example 17> The selection unit selects the analysis model based on information about the subject and the shooting environment information. The information processing device according to any one of <Configuration Example 1> to <Configuration Example 16>.

[0205] <Configuration Example 18> The information about the subject includes information indicating the subject's age and information indicating the subject's exercise experience; The information processing device described in <Configuration Example 17>.

[0206] It is also possible to provide a method of processing performed by the information processing device described above. <Configuration Example 19> The video acquisition unit acquires a video of the subject's exercise, a photographing environment acquisition unit acquires photographing environment information related to a photographing environment of the video acquired by the video acquisition unit; a selection unit that selects an analysis model based on the photography environment information acquired by the photography environment acquisition unit; a motion analysis unit that analyzes the motion of the subject from the video using the analysis model selected by the selection unit; an output unit outputs an analysis result of the motion of the subject obtained by the motion analysis unit; Information processing methods.

[0207] It is also possible to provide a program recording medium for recording a program executed by the processor in the information processing device as described above. <Configuration Example 20> A program recording medium for recording a program, The program includes: A video acquisition function for acquiring a video of the subject's exercise; a shooting environment acquisition function for acquiring shooting environment information related to a shooting environment of the video acquired by the video acquisition function; A selection function for selecting an analysis model based on the image capture environment information acquired by the image capture environment acquisition function; a motion analysis function that analyzes the motion of the subject from the video using the analysis model selected by the selection function; an output function that outputs an analysis result of the motion of the subject obtained by the motion analysis function; To achieve this, Program recording medium. [Explanation of symbols]

[0208] 1...information processing system, 11...information processing device, 21...imaging device, 22...sensor unit, 31...display device, 111...input unit, 112...output unit, 113...communication unit, 114...storage unit, 115...control unit, 151...acquisition unit, 152...selection unit, 153...motion analysis unit, 154...imaging environment proposal unit, 155...output control unit, 156...learning control unit, 171...video acquisition unit, 172...imaging environment acquisition unit, 211...analysis result, 611...analysis model pattern, 651...model under learning, 711...analysis characteristic, A1...analysis model group, B1...first subject, B11...eleventh subject, B21...21st subject subject, B31...the 31st subject, B41...the 41st subject, B51...the 51st subject, B61...the 61st subject, B71...the 71st subject, B81...the 81st subject, B91...the 91st subject, B111...the 111th subject, B121...the 121st subject, B131...the 131st subject, B141...the 141st subject, B151...the 151st subject, B161...the 161st subject, B211...the 211th subject, B311...the 311th subject, C1...video information, C2...filming environment information, C3...subject information, D...analysis model, D1...first analysis model, D2...second analysis model, D3...third analysis model , Dn...nth analysis model, E1...imaging environment change instruction, Fa1...a1th frame, Fa2...a2nd frame, Fa3...a3th frame, Fa4...a4th frame, Fa5...a5th frame, Fa6...a6th frame, Fb1...b1st frame, Fb2...b2nd frame, Fb3...b3rd frame, G1...first video information, G2...second video information, G11...eleventh video information, G12...twelfth video information, G21...twenty-first video information, G31...thirty-first video information, G32...thirty-second video information, G33...thirty-third video information, G41...forty-first video information, G42...forty-second video information , G51...51st video information, G61...61st video information, G62...62nd video information, H...joint, Ha1...a1th joint, Ha2...a2th joint, Ha3...a3th joint, Ha4...a4th joint, Ha5...a5th joint, Ha6...a6th joint, Hb1...b1th joint, Hb2...b2nd joint, Hb3...b3rd joint, Hc...cth joint, Hd...dth joint, He...eth joint, Hf...fth joint, Hg...gth joint, Hi...ith joint, Hj...jth joint, Hk...kth joint, Ho...oth joint, Hp...pth joint, Hq...qth joint, Hr...rth joint, Hs...sth joint, K...sun, K1...1st sun,K2...second sun, K3...third sun, L...bone part, Qa...a-th raindrop, T1...first pattern information, T2...second pattern information, T3...third pattern information, T4...fourth pattern information, T11...correspondence table, U1...first glove, U2...second glove, V1...first person, a1...a1-th sensor, a2...a2-th sensor, a3...a3-th sensor, a4...a4-th sensor, a5...a5-th sensor, a6...a6-th sensor, a7...a7-th sensor, 8...a8th sensor, a9...a9th sensor, b1...teaching data, c1...c1th match, c2...c2th match, c3...c3th match, cm...cm match, d1...d1th point, d2...d2th point, d3...d3th point, p1...p1th pattern information, p2...p2th pattern information, p3...p3th pattern information, pm...pmth pattern information, q1...q1th analysis result, q2...q2th analysis result, q3...q3th analysis result, qm...qmth analysis result,

Claims

1. A video acquisition unit that acquires a video of the subject's exercise; a photographing environment acquisition unit that acquires photographing environment information related to a photographing environment of the video acquired by the video acquisition unit; a selection unit that selects an analysis model based on the image capture environment information acquired by the image capture environment acquisition unit; a motion analysis unit that analyzes the motion of the subject from the video by using the analysis model selected by the selection unit; an output unit that outputs an analysis result of the motion of the subject obtained by the motion analysis unit; An information processing device comprising:

2. The shooting environment information includes information indicating a frame rate of the video. The information processing device according to claim 1 .

3. The imaging environment information includes information indicating a resolution of the video. The information processing device according to claim 1 .

4. The shooting environment information includes information regarding a location where the video was shot. The information processing device according to claim 1 .

5. the photographing environment information includes information indicating whether the location is indoors or outdoors, the photographing environment acquisition unit acquires position information of the subject from an external device that photographs the movement of the subject, and determines whether the location is indoors or outdoors; The information processing device according to claim 4.

6. The shooting environment information includes information about the weather at the location where the video was shot. The information processing device according to claim 1 .

7. The photographing environment acquisition unit acquires weather information of the location of the subject and determines the weather. The information processing device according to claim 6.

8. The shooting environment information includes information regarding the time when the video was shot. The information processing device according to claim 1 .

9. The photographing environment acquisition unit acquires time information of the position of the subject and determines the time. The information processing device according to claim 8.

10. The shooting environment information includes information regarding a background color of a location where the video was shot and a clothing color of the subject. The information processing device according to claim 1 .

11. the photographing environment acquisition unit determines whether the background color and the clothing color are the same color based on whether a boundary between the subject and the surroundings can be detected; The information processing device according to claim 10.

12. The shooting environment information includes information regarding background movement of the location where the video was shot. The information processing device according to claim 1 .

13. The shooting environment information includes information regarding blurring when the video is shot. The information processing device according to claim 1 .

14. The photographing environment acquisition unit determines the blur based on whether or not a position of the pivot foot of the subject has changed. The information processing device according to claim 13.

15. The motion analysis unit analyzes skeletal movement during the motion of the subject. The information processing device according to claim 1 .

16. The output unit outputs the analysis result to an external device of the subject. The information processing device according to claim 1 .

17. The selection unit selects the analysis model based on information about the subject and the shooting environment information. The information processing device according to claim 1 .

18. The information about the subject includes information indicating the subject's age and information indicating the subject's exercise experience; The information processing device according to claim 17.

19. The video acquisition unit acquires a video of the subject's exercise, a photographing environment acquisition unit acquires photographing environment information related to a photographing environment of the video acquired by the video acquisition unit; a selection unit that selects an analysis model based on the photography environment information acquired by the photography environment acquisition unit; a motion analysis unit that analyzes the motion of the subject from the video using the analysis model selected by the selection unit; an output unit outputs an analysis result of the motion of the subject obtained by the motion analysis unit; Information processing methods.

20. A program recording medium for recording a program, The program includes: A video acquisition function for acquiring a video of the subject's exercise; a shooting environment acquisition function for acquiring shooting environment information related to a shooting environment of the video acquired by the video acquisition function; A selection function for selecting an analysis model based on the image capture environment information acquired by the image capture environment acquisition function; a motion analysis function that analyzes the motion of the subject from the video using the analysis model selected by the selection function; an output function that outputs an analysis result of the motion of the subject obtained by the motion analysis function; To achieve this, Program recording medium.

Images