Information processing program, information processing method, and information processing device

Abstract

An information processing device (100) identifies a type of movement by a subject corresponding to skeletal information (101) at a first time point in time series data. The information processing device (100) determines a model of probability distribution that constrains the time variation of the position of any of a plurality of parts in the skeletal information (101) at the first time point according to a trend of motion by a part corresponding to the identified type of movement. The information processing device (100) generates a graph (110) which includes nodes (111) indicating the position of each part at each time point, first edges (112), and a second edge (113) and in which the determined model is mapped to the second edge (113). On the basis of the generated graph (110), the information processing device (100) corrects the skeletal information (101) at the first time point in the time series data.

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] In the fields of sports, healthcare, and entertainment, there is a need for technology that can recognize human movements. For example, there is a technology that uses deep learning to identify the 3D coordinates of each joint of a person based on multi-view images captured from different angles.

[0003] Prior art includes, for example, a technique that outputs one of the results of the first process, the second process, and the third process as the skeletal recognition result of the subject based on the likelihood of the result of the first process, the likelihood of the result of the second process, and the likelihood of the result of the third process. Another example is a technique that recognizes a heat map image in which the likelihood of the position of multiple joints of the subject is projected from multiple directions from a range image of the subject. Another example is a technique that performs optimization calculations based on inverse kinematics using candidate feature point positions and the multi-joint structure of the subject to obtain each joint angle of the subject, and then performs forward kinematics calculations using the joint angles to obtain the positions of feature points including the joints of the subject. Another example is a behavior detection technique that uses a recurrent neural network. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2021 / 064942 [Patent Document 2] International Publication No. 2021 / 002025 [Patent Document 3] Japanese Patent Publication No. 2020-42476 [Patent Document 4] US Patent Application Publication No. 2017 / 0344829 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technology, it may be difficult to accurately identify the three-dimensional coordinates of each joint of a person. For example, the three-dimensional coordinates of a joint in a person's right hand may be mistaken for the three-dimensional coordinates of a joint in the person's left hand. For example, the three-dimensional coordinates of a part of an object other than a person captured in a multi-viewpoint image may be mistaken for the three-dimensional coordinates of a person's joints.

[0006] In one aspect, the present invention aims to enable the position of a part of a subject to be identified with high accuracy. [Means for solving the problem]

[0007] According to one embodiment, an information processing program, an information processing method, and an information processing device are proposed that acquire time series data of skeletal information including the position of each of a plurality of body parts of a subject, identify a type of movement of the subject corresponding to the skeletal information at a first time point in the acquired time series data based on features of the skeletal information in the acquired time series data, determine a probability distribution model that constrains the change over time in the position of any of the plurality of body parts in the skeletal information at the first time point in the acquired time series data according to the tendency of movement of the any of the body parts corresponding to the identified type of movement, generate a graph that includes nodes indicating the position of each of the body parts at each time point, first edges connecting nodes indicating the positions of different body parts that are biologically connected at each time point, and second edges connecting nodes indicating the positions of any of the body parts at different time points, and associates the determined model with the second edges, and correct the skeletal information at the first time point in the time series data based on the generated graph. [Effects of the Invention]

[0008] According to one aspect, it is possible to identify the position of a body part of a subject with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of an information processing method according to an embodiment. [Figure 2] FIG. 2 is an explanatory diagram illustrating an example of an information processing system 200. As shown in FIG. [Figure 3] FIG. 3 is a block diagram showing an example of the hardware configuration of the information processing device 100. As shown in FIG. [Figure 4] FIG. 4 is a block diagram showing an example of the hardware configuration of the image capturing device 201. As shown in FIG. [Figure 5] FIG. 5 is a block diagram showing an example of the functional configuration of the information processing device 100. As shown in FIG. [Figure 6] FIG. 6 is an explanatory diagram showing the flow of operations of the information processing device 100. [Figure 7] FIG. 7 is an explanatory diagram (part 1) showing a specific example of identifying an abnormal joint. [Figure 8] FIG. 8 is an explanatory diagram (part 2) showing a specific example of identifying an abnormal joint. [Figure 9] FIG. 9 is an explanatory diagram showing a specific example of generating a factor graph. [Figure 10] FIG. 10 is an explanatory diagram showing a specific example of a template 911 of a factor graph corresponding to "jump." [Figure 11] FIG. 11 is an explanatory diagram showing a specific example of a template 911 of a factor graph corresponding to "lying down." [Figure 12] FIG. 12 is an explanatory diagram showing a specific example of adding a time series constraint. [Figure 13] FIG. 13 is an explanatory diagram showing a specific example of correcting the 3D skeleton inference result 602. [Figure 14] FIG. 14 is an explanatory diagram (part 1) showing a specific example of the flow of data processing in the operational example. [Figure 15] FIG. 15 is an explanatory diagram (part 2) showing a specific example of the flow of data processing in the operational example. [Figure 16] FIG. 16 is a flowchart illustrating an example of the overall processing procedure. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an information processing program, an information processing method, and an information processing device according to embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0011] (An example of an information processing method according to an embodiment) FIG. 1 is an explanatory diagram illustrating an example of an information processing method according to an embodiment. The information processing device 100 is a computer that enables accurate identification of the positions of body parts of a subject. The subject is, for example, a person. The body parts are, for example, the neck, head, right and left shoulders, right and left elbows, right and left hands, right and left knees, right and left feet, etc. The body parts are, for example, joints. The positions are, for example, three-dimensional coordinates.

[0012] Conventionally, there is a technology that uses deep learning to identify the 3D coordinates of each of a person's joints based on multi-view images taken from different angles at each point in time, thereby identifying changes in the 3D coordinates of each of the person's joints over time.

[0013] Specifically, it is conceivable to detect an area in which a person appears in a multi-view image, identify the two-dimensional coordinates of each of the person's joints based on the detected area, and then identify the three-dimensional coordinates of each of the person's joints based on the identified two-dimensional coordinates, taking into account the angle. Specifically, a model trained by deep learning is used to identify the three-dimensional coordinates of each of the person's joints. For an example of this technology, see Reference 1 and Reference 2 below.

[0014] Reference 1: Iskakov, Karim, et al. “Learnable triangulation of human pose.” Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019.

[0015] Reference 2: Moon, Gyeongsik, Ju Yong Chang, and Kyoung Mu Lee. “V2v-posenet: Voxel-to-voxel prediction network for accurate 3d hand and human pose estimation from a single depth map.” Proceedings of the IEEE conference on computer vision and pattern Recognition. 2018.

[0016] However, with conventional techniques, it may be difficult to accurately identify the three-dimensional coordinates of each of a person's joints. For example, although the distance between each of the joints of the same person is constant at different times, the fact that the distance between each of the person's joints is constant is not taken into account when learning a model using deep learning. As a result, it is not possible to accurately identify the three-dimensional coordinates of each of a person's joints, and it is not possible to accurately identify changes in the three-dimensional coordinates of each of a person's joints over time.

[0017] When an analyst analyzing human movement refers to the 3D coordinates of each identified joint of a person, the analyst is likely to get the impression that the 3D coordinates of each joint of the person are intuitively incorrect. Specifically, the analyst gets the impression that the length of the person's arm is expanding and contracting. Also, specifically, the analyst gets the impression that the person's arm is moving at a speed that is impossible for a human to achieve.

[0018] Therefore, in this embodiment, an information processing method that can accurately identify the positions of the joints of a subject will be described.

[0019] In FIG. 1, (1-1) an information processing device 100 acquires time-series data of skeletal information 101. The skeletal information 101 includes, for example, the position of each of a plurality of body parts of a subject. The body parts are, for example, the neck, head, right and left shoulders, right and left elbows, right and left hands, right and left knees, right and left feet, etc. The body parts are, for example, joints. In the example of FIG. 1, the body parts are specifically joint 1, joint 2, joint 3, etc. The positions are, for example, three-dimensional coordinates. The time-series data includes, for example, skeletal information 101 for each time point. In the example of FIG. 1, the time-series data specifically includes skeletal information 101 at time point T, skeletal information 101 at time point T-1, etc.

[0020] (1-2) The information processing device 100 identifies the type of motion of the subject corresponding to the skeletal information 101 at a first time point in the acquired time-series data based on the feature amounts of the skeletal information 101 in the acquired time-series data. The type of motion may be, for example, walking, running, jumping, sitting, lying down, horizontal rotations such as turns or spins, or vertical rotations such as somersaults or horizontal bar exercises. The feature amounts may be, for example, the position of each part of the subject indicated by the skeletal information 101. The feature amounts may be, for example, the deviation between the positions of each part of the subject indicated by the skeletal information 101 at different times. The feature amounts may be, for example, the distance between the positions of different parts of the subject indicated by the skeletal information 101.

[0021] The information processing device 100 has, for example, a first model for identifying the type of motion of the subject. The first model has a function of making it possible to determine the type of motion of the subject in accordance with, for example, input of feature amounts of skeletal information 101. The information processing device 100 uses the first model to identify the type of motion of the subject corresponding to the skeletal information 101 at a first time point in the acquired time-series data. In the example of FIG. 1, the information processing device 100 specifically identifies "lying down" as the type of motion of the subject corresponding to the skeletal information 101 at a first time point in the acquired time-series data.

[0022] (1-3) The information processing device 100 determines a second probability distribution model that constrains the change over time in the position of any of the multiple parts in the skeletal information 101 at the first time point in the acquired time-series data, depending on the tendency of the movement of any of the parts corresponding to the identified type of movement. The tendency of the movement is, for example, a tendency toward uniform position movement, uniform velocity movement, or uniform acceleration movement. In the example of FIG. 1, specifically, the information processing device 100 determines a second probability distribution model that constrains the change over time in the position of joint 1 in the skeletal information 101 at time point T, depending on the tendency of uniform position movement corresponding to lying down.

[0023] (1-4) The information processing device 100 generates a graph 110 including nodes 111 indicating the position of each body part at each time point, first edges 112 connecting the nodes 111 together, and second edges 113 connecting the nodes 111 together. The first edges 112 connect the nodes 111 indicating the positions of different body parts that are biologically connected at each time point. The second edges 113 connect the nodes 111 indicating the positions of any body part at different time points.

[0024] When generating the graph 110, the information processing device 100 associates the determined second model with the second edge 113. In the example of Fig. 1 , specifically, the information processing device 100 generates the graph 110 by associating the determined second model with the second edge 113 that connects the nodes 111 that indicate the positions of the joints 1 of the subject at time points T-1 and T.

[0025] (1-5) The information processing device 100 corrects the skeletal information 101 at the first time point in the time-series data based on the generated graph 110. For example, the information processing device 100 corrects the position of the joint 1 of the subject included in the skeletal information 101 at the time point T in the time-series data. This allows the information processing device 100 to accurately identify the position of each joint of the subject. The information processing device 100 can accurately identify the change over time in the position of each joint of the subject.

[0026] Here, the case where the information processing device 100 identifies the type of movement of the subject by using the first model has been described, but the present invention is not limited to this. For example, the information processing device 100 may identify the type of movement of the subject without using the first model.

[0027] Here, the case where the information processing device 100 operates independently has been described, but this is not limiting. For example, a plurality of computers may cooperate to realize the functions of the information processing device 100. Specifically, a computer that identifies the type of motion of the subject, a computer that generates the graph 110, and a computer that corrects the skeletal information 101 at the first time point in the time-series data based on the graph 110 may cooperate with each other.

[0028] (An example of the information processing system 200) Next, an example of an information processing system 200 to which the information processing device 100 shown in FIG. 1 is applied will be described with reference to FIG.

[0029] 2 is an explanatory diagram showing an example of an information processing system 200. In FIG. 2, the information processing system 200 includes the information processing device 100, one or more image capturing devices 201, and one or more client devices 202.

[0030] In the information processing system 200, the information processing device 100 and the image capturing device 201 are connected via a wired or wireless network 210. The network 210 is, for example, a local area network (LAN), a wide area network (WAN), the Internet, etc. In the information processing system 200, the information processing device 100 and the client device 202 are connected via the wired or wireless network 210.

[0031] The information processing device 100 acquires a plurality of images of a subject captured from different angles at each time point from one or more image capturing devices 201. Based on the acquired plurality of images for each time point, the information processing device 100 identifies the distribution of the existence probability of each part of the subject in three-dimensional space and identifies the three-dimensional coordinates of each part of the subject.

[0032] The information processing device 100 identifies the type of motion of the subject for each time point based on the three-dimensional coordinates of each identified body part of the subject. The information processing device 100 identifies any one of the subject's multiple body parts that corresponds to the type of motion of the subject for each time point based on the type of motion of the subject. The information processing device 100 determines a probability distribution model that constrains the change over time in the position of any one of the identified body parts according to the type of motion of the subject for each time point.

[0033] The information processing device 100 generates a graph including nodes indicating the three-dimensional coordinates of each part of the subject at each identified time point. When generating the graph, the information processing device 100 generates the graph so that, for each time point, the graph includes a first edge connecting nodes indicating the three-dimensional coordinates of different parts of the subject that are biologically connected.

[0034] When generating the graph, the information processing device 100 generates the graph so that, for each time point, a second edge connecting nodes indicating the three-dimensional coordinates of the time point and another time point other than the time point is included in the graph for any of the identified parts. The other time point is, for example, a time point immediately before the certain time point. The information processing device 100 associates the determined model with the second edge included in the graph.

[0035] The information processing device 100 refers to the graph and corrects the three-dimensional coordinates of each body part of the identified subject. The information processing device 100 outputs the corrected three-dimensional coordinates of each body part of the subject. The output format may be, for example, display on a display, printout on a printer, transmission to another computer, or storage in a memory area. The information processing device 100 transmits the corrected three-dimensional coordinates of each body part of the subject to the client device 202, for example. The information processing device 100 is, for example, a server or a PC (Personal Computer).

[0036] The image capturing device 201 is a computer that captures an image of a subject. The image capturing device 201 includes a camera having multiple image capturing elements, and captures an image of the subject using the camera. The image capturing device 201 generates an image of the subject and transmits the image to the information processing device 100. The image capturing device 201 is, for example, a smartphone. The image capturing device 201 may be, for example, a fixed camera. The image capturing device 201 may be, for example, a drone.

[0037] The client device 202 receives the three-dimensional coordinates of each body part of the subject from the information processing device 100. The client device 202 outputs the received three-dimensional coordinates of each body part of the subject so that the user can refer to them. For example, the client device 202 displays the received three-dimensional coordinates of each body part of the subject on a display. The client device 202 is, for example, a PC, a tablet terminal, or a smartphone.

[0038] Here, the case where the information processing device 100 is a device different from the image capturing device 201 has been described, but this is not limiting. For example, the information processing device 100 may have the function of the image capturing device 201 and operate as the image capturing device 201. Here, the case where the information processing device 100 is a device different from the client device 202 has been described, but this is not limiting. For example, the information processing device 100 may have the function of the client device 202 and operate as the client device 202.

[0039] (Example of hardware configuration of information processing device 100) Next, an example of the hardware configuration of the information processing device 100 will be described with reference to FIG.

[0040] Fig. 3 is a block diagram showing an example of the hardware configuration of the information processing device 100. In Fig. 3, the information processing device 100 has a CPU (Central Processing Unit) 301, a memory 302, a network I / F (Interface) 303, a recording medium I / F 304, and a recording medium 305. The information processing device 100 further has a display 306 and an input device 307. Furthermore, each component is connected to each other via a bus 300.

[0041] Here, CPU 301 is responsible for overall control of information processing device 100. Memory 302 includes, for example, a read-only memory (ROM), a random access memory (RAM), and a flash ROM. Specifically, for example, the flash ROM or ROM stores various programs, and RAM is used as a work area for CPU 301. The programs stored in memory 302 are loaded into CPU 301, causing CPU 301 to execute coded processes.

[0042] The network I / F 303 is connected to the network 210 via a communication line, and is connected to other computers via the network 210. The network I / F 303 manages the internal interface with the network 210 and controls the input and output of data from other computers. The network I / F 303 is, for example, a modem or a LAN adapter.

[0043] The recording medium I / F 304 controls reading and writing of data from and to the recording medium 305 under the control of the CPU 301. The recording medium I / F 304 is, for example, a disk drive, a solid state drive (SSD), or a universal serial bus (USB) port. The recording medium 305 is a non-volatile memory that stores data written under the control of the recording medium I / F 304. The recording medium 305 is, for example, a disk, a semiconductor memory, or a USB memory. The recording medium 305 may be detachable from the information processing device 100.

[0044] The display 306 displays data such as a cursor, an icon, a toolbox, a document, an image, or function information. The display 306 is, for example, a CRT (Cathode Ray Tube), a liquid crystal display, or an organic EL (Electroluminescence) display. The input device 307 has keys for inputting characters, numbers, various instructions, etc., and inputs data. The input device 307 is, for example, a keyboard or a mouse. The input device 307 may also be, for example, a touch panel input pad or a numeric keypad.

[0045] The information processing device 100 may have, in addition to the above-mentioned components, for example, a camera. Furthermore, the information processing device 100 may have, in addition to the above-mentioned components, for example, a printer, a scanner, a microphone, or a speaker. Furthermore, the information processing device 100 may have a plurality of recording medium I / Fs 304 and recording media 305. Furthermore, the information processing device 100 may not have, for example, a display 306 or an input device 307. Furthermore, the information processing device 100 may not have, for example, a recording medium I / F 304 or a recording medium 305.

[0046] (Example of hardware configuration of image capturing device 201) Next, an example of the hardware configuration of the image capturing device 201 will be described with reference to FIG.

[0047] Fig. 4 is a block diagram showing an example of the hardware configuration of the image capturing device 201. In Fig. 4, the image capturing device 201 has a CPU 401, a memory 402, a network I / F 403, a recording medium I / F 404, a recording medium 405, and a camera 406. Furthermore, each component is connected to each other via a bus 400.

[0048] Here, the CPU 401 is responsible for overall control of the image capturing device 201. The memory 402 includes, for example, a ROM, a RAM, and a flash ROM. Specifically, for example, the flash ROM and the ROM store various programs, and the RAM is used as a work area for the CPU 401. The programs stored in the memory 402 are loaded into the CPU 401, causing the CPU 401 to execute coded processes.

[0049] The network I / F 403 is connected to the network 210 via a communication line, and is connected to other computers via the network 210. The network I / F 403 manages an internal interface with the network 210 and controls input and output of data from other computers. The network I / F 403 is, for example, a modem or a LAN adapter.

[0050] The recording medium I / F 404 controls reading / writing of data from / to the recording medium 405 under the control of the CPU 401. The recording medium I / F 404 is, for example, a disk drive, an SSD, a USB port, etc. The recording medium 405 is a non-volatile memory that stores data written under the control of the recording medium I / F 404. The recording medium 405 is, for example, a disk, a semiconductor memory, a USB memory, etc. The recording medium 405 may be detachable from the image capturing device 201. The camera 406 has multiple image capturing elements and generates an image by capturing an object using the multiple image capturing elements. The camera 406 is, for example, a competition camera. The camera 406 is, for example, a surveillance camera.

[0051] In addition to the components described above, the image capturing device 201 may also include, for example, a keyboard, a mouse, a display, a printer, a scanner, a microphone, a speaker, etc. The image capturing device 201 may also include a plurality of recording medium I / Fs 404 and recording media 405. The image capturing device 201 may also not include the recording medium I / Fs 404 and recording media 405.

[0052] (Example of hardware configuration of client device 202) A specific example of the hardware configuration of the client device 202 is similar to the example of the hardware configuration of the information processing device 100 shown in FIG. 3, and therefore a description thereof will be omitted.

[0053] (Example of functional configuration of information processing device 100) Next, an example of the functional configuration of the information processing device 100 will be described with reference to FIG.

[0054] 5 is a block diagram showing an example of the functional configuration of the information processing device 100. The information processing device 100 includes a storage unit 500, an acquisition unit 501, an analysis unit 502, a learning unit 503, an identification unit 504, a determination unit 505, a generation unit 506, a correction unit 507, and an output unit 508.

[0055] The storage unit 500 is realized by, for example, a storage area such as the memory 302 or the recording medium 305 shown in Fig. 3. In the following, a case where the storage unit 500 is included in the information processing device 100 will be described, but this is not limiting. For example, the storage unit 500 may be included in a device different from the information processing device 100, and the stored contents of the storage unit 500 may be accessible from the information processing device 100.

[0056] The acquiring unit 501 to the output unit 508 function as an example of a control unit. Specifically, the acquiring unit 501 to the output unit 508 realize their functions by causing the CPU 301 to execute a program stored in a storage area such as the memory 302 or the recording medium 305 shown in Fig. 3, or by using the network I / F 303. The processing results of each functional unit are stored in a storage area such as the memory 302 or the recording medium 305 shown in Fig. 3, for example.

[0057] The storage unit 500 stores various information that is referenced or updated during processing by each functional unit. The storage unit 500 stores, for example, multiple images of a specific person captured from different angles at multiple consecutive points in time. The angles indicate the capture positions. The images are acquired by, for example, the acquisition unit 501.

[0058] The storage unit 500 stores, for example, time-series data of skeletal information. The time-series data includes skeletal information at each of a plurality of consecutive time points. The skeletal information includes the position of each of a plurality of body parts of a specific person. The body parts are, for example, joints. The body parts are, for example, the neck, head, right and left shoulders, right and left elbows, right and left hands, right and left knees, right and left feet, etc. The positions are, for example, three-dimensional coordinates. The time-series data is acquired, for example, by the acquisition unit 501. The time-series data may be generated, for example, by the analysis unit 502.

[0059] The acquisition unit 501 acquires various types of information used in the processing of each functional unit. The acquisition unit 501 stores the acquired various types of information in the storage unit 500 or outputs it to each functional unit. The acquisition unit 501 may also output the various types of information stored in the storage unit 500 to each functional unit. The acquisition unit 501 acquires various types of information based on, for example, a user's operation input. The acquisition unit 501 may also receive various types of information from, for example, a device different from the information processing device 100.

[0060] The acquiring unit 501 acquires, for example, time-series data of skeletal information of the subject. The skeletal information of the subject includes, for example, the positions of each of a plurality of body parts of the subject. Specifically, the acquiring unit 501 acquires the time-series data of the skeletal information of the subject by accepting input of the time-series data of the skeletal information of the subject based on an operation input by a user. Specifically, the acquiring unit 501 may acquire the time-series data of the skeletal information of the subject by receiving it from another computer.

[0061] The acquiring unit 501 may acquire, for example, time-series data of the subject's past skeletal information. The subject may be, for example, the same as the target person. The subject's skeletal information includes, for example, the positions of each of the subject's multiple body parts. Specifically, the acquiring unit 501 acquires the time-series data of the subject's skeletal information by accepting input of the time-series data of the subject's skeletal information based on an operation input by a user. Specifically, the acquiring unit 501 may acquire the time-series data of the subject's skeletal information by receiving it from another computer.

[0062] The acquiring unit 501 may acquire, for example, the type of subject's motion corresponding to each piece of skeletal information in the time-series data of the subject's past skeletal information. The types of motion include, for example, walking, running, jumping, sitting, lying down, horizontal rotations such as turns or spins, or vertical rotations such as somersaults or horizontal bar exercises. Specifically, the acquiring unit 501 acquires the type of subject's motion by accepting, based on a user's operation input, an input of the type of subject's motion corresponding to each piece of skeletal information in the time-series data of the subject's past skeletal information. Specifically, the acquiring unit 501 may acquire the type of subject's motion corresponding to each piece of skeletal information in the time-series data of the subject's past skeletal information by receiving it from another computer.

[0063] The acquisition unit 501 acquires, for example, a plurality of images of the subject captured from different angles at each of a plurality of consecutive time points. When time-series data of skeletal information of the subject is generated by the analysis unit 502 without being acquired, the acquisition unit 501 acquires the plurality of images. This enables the acquisition unit 501 to generate time-series data of skeletal information of the subject by the analysis unit 502.

[0064] For example, the acquisition unit 501 may acquire a plurality of images obtained by capturing the subject from different angles at each of a plurality of consecutive time points. When the time-series data of the subject's skeletal information is generated by the analysis unit 502 without being acquired, the acquisition unit 501 acquires a plurality of images. This enables the acquisition unit 501 to generate the time-series data of the subject's skeletal information by the analysis unit 502.

[0065] The acquisition unit 501 may receive a start trigger that starts processing of any of the functional units. The start trigger may be, for example, a predetermined operation input by a user. The start trigger may be, for example, reception of predetermined information from another computer. The start trigger may be, for example, output of predetermined information by any of the functional units.

[0066] The acquiring unit 501 may, for example, accept that a plurality of images have been acquired as a start trigger for starting the processing of the analyzing unit 502. The acquiring unit 501 may, for example, accept that time-series data of skeletal information of the subject has been acquired as a start trigger for starting the processing of the learning unit 503. The acquiring unit 501 may, for example, accept that time-series data of skeletal information of the subject has been acquired as a start trigger for starting the processing of the identifying unit 504, the determining unit 505, the generating unit 506, and the correcting unit 507.

[0067] The analysis unit 502 generates time-series data of skeletal information of a predetermined person. The analysis unit 502 generates, for example, time-series data of skeletal information of a subject. Specifically, the analysis unit 502 estimates the position of each part of the subject at each time point based on multiple images of the subject captured from different angles at each of multiple time points, and generates skeletal information of the subject including the estimated positions. Specifically, the analysis unit 502 generates time-series data of the skeletal information of the subject based on the generated skeletal information of the subject. This allows the analysis unit 502 to tentatively identify the position of each part of the subject at each time point and obtain a correction target.

[0068] The analysis unit 502 may, for example, generate time-series data of the subject's skeletal information. Specifically, the analysis unit 502 generates skeletal information of the subject at each of a plurality of time points based on a plurality of images of the subject captured from different angles at each of the multiple time points, thereby generating time-series data of the subject's skeletal information. The analysis unit 502 may add noise to the generated time-series data of the subject's skeletal information. The analysis unit 502 sets the subject's skeletal information as training information for generating a learning model. This allows the analysis unit 502 to obtain training information for generating a learning model.

[0069] The learning unit 503 learns a first learning model based on training information including the position of each of the multiple body parts of the subject. The first learning model has a function of making it possible to identify any of the multiple body parts of a predetermined person that is in an abnormal state with respect to its position, based on feature amounts related to skeletal information in the time-series data of the skeletal information of the predetermined person. The first learning model has a function of making it possible to determine, for example, whether each body part of the predetermined person is in an abnormal state with respect to its position.

[0070] Specifically, the first learning model has a function of calculating an index value indicating the magnitude of the probability that each body part of a predetermined person is in an abnormal state with respect to its position. More specifically, the first learning model outputs an index value indicating the magnitude of the probability that each body part of a predetermined person is in an abnormal state with respect to its position, in response to input of features related to skeletal information. Specifically, the first learning model is a neural network. This enables the learning unit 503 to identify any body part of the target person that is in an abnormal state with respect to its position.

[0071] The learning unit 503 learns a second learning model based on teacher information including the positions of each of a plurality of body parts of the subject. The second learning model has a function of making it possible to identify the type of movement of a predetermined person corresponding to each piece of skeletal information in the time-series data of the skeletal information of the predetermined person, according to feature amounts related to the skeletal information in the time-series data of the skeletal information of the predetermined person.

[0072] Specifically, the second learning model outputs an index value indicating the likelihood of each candidate type of a predetermined person's movement corresponding to any of the skeletal information in the time-series data in response to input of feature quantities related to the skeletal information in the time-series data. Specifically, the second learning model is a neural network. This allows the learning unit 503 to identify the type of movement of the target person.

[0073] The identification unit 504 identifies the type of movement of the subject corresponding to the skeletal information at the first time point in the acquired time-series data based on the feature amounts of the skeletal information in the acquired time-series data. The identification unit 504, for example, uses the trained second learning model to identify the type of movement of the subject corresponding to the skeletal information at the first time point based on the feature amounts of the skeletal information in the acquired time-series data of the skeletal information of the subject.

[0074] Specifically, the identification unit 504 inputs feature quantities related to skeletal information in the time-series data of the skeletal information of the subject into the second learning model, thereby calculating an index value indicating the likelihood of each candidate being the type of movement of the subject corresponding to the skeletal information at the first time point. Specifically, the identification unit 504 identifies the type of movement of the subject corresponding to the skeletal information at the first time point based on the calculated index value. More specifically, the identification unit 504 identifies the candidate with the largest calculated index value as the type of movement of the subject. This allows the identification unit 504 to obtain a guideline for correcting the position of each of the subject's multiple body parts. The identification unit 504 can determine which body part of the subject it is preferable to correct the position of.

[0075] The identification unit 504 identifies an abnormal part that is in an abnormal state with respect to position among multiple parts of the subject. The identification unit 504 identifies an abnormal part that is in an abnormal state with respect to position for the skeletal information at a first time point in the acquired time-series data of the skeletal information of the subject, based on feature amounts related to the skeletal information in the acquired time-series data of the skeletal information of the subject. The identification unit 504, for example, uses a trained first learning model to identify an abnormal part that is in an abnormal state with respect to position for the skeletal information at the first time point, based on feature amounts related to the skeletal information in the acquired time-series data of the skeletal information of the subject.

[0076] Specifically, the identification unit 504 inputs feature quantities related to skeletal information in the time-series data of the subject's skeletal information into the first learning model, thereby calculating an index value indicating the magnitude of the probability that each part of the subject's body is in an abnormal state for the skeletal information at the first time point. Specifically, the identification unit 504 identifies an abnormal part that is in an abnormal state with respect to its position for the skeletal information at the first time point based on the calculated index value. More specifically, the identification unit 504 identifies, among the multiple parts of the subject, a part whose calculated index value is equal to or greater than a threshold as an abnormal part that is in an abnormal state with respect to its position. This allows the identification unit 504 to obtain a guideline for correcting the position of each of the multiple parts of the subject. The identification unit 504 can determine which part of the subject's body is preferable to correct its position.

[0077] The determination unit 505 determines a distribution model of probability distribution that constrains the change over time in the position of any of the multiple parts corresponding to the identified type of movement from the skeletal information at the first time point in the acquired time-series data. The distribution model is, for example, a model for constraining the change over time in the position of any of the parts corresponding to the identified type of movement in accordance with the tendency of the movement of any of the parts corresponding to the identified type of movement. The tendency of the movement is, for example, a tendency of constant position movement, constant velocity movement, or constant acceleration movement. This allows the determination unit 505 to obtain a guideline for correcting the position of the part identified by the identification unit 504.

[0078] The generating unit 506 generates a graph including nodes indicating the position of each body part at each time point, a first edge, and a second edge. The first edge connects nodes indicating the positions of different body parts that are biologically connected at each time point. The second edge connects nodes indicating the positions of any body part corresponding to the identified type of movement at different time points. When generating the graph, the generating unit 506 associates the determined distribution model with the second edge. This allows the generating unit 506 to modify the skeletal information at the first time point in the time-series data of the subject's skeletal information.

[0079] The generating unit 506 may generate a graph such that a third edge connecting nodes indicating positions of other parts of the multiple parts other than any part corresponding to the identified type of movement is further included in the graph. For example, if there is one first edge connected to each of the nodes indicating the positions of other parts at different points in time, the generating unit 506 generates a graph such that a third edge connecting these nodes is included in the graph. This enables the generating unit 506 to accurately correct skeletal information at the first point in time in the time-series data of the subject's skeletal information. For example, the generating unit 506 enables the positions of other parts to be accurately corrected.

[0080] The generating unit 506 may generate a graph such that, in addition to any of the multiple body parts corresponding to the identified type of movement, the graph further includes a third edge connecting nodes indicating the positions of other body parts identified as abnormal. For example, if there is one first edge connected to each of the nodes indicating the positions of other body parts at different points in time, the generating unit 506 generates a graph such that a third edge connecting these nodes is included in the graph. This enables the generating unit 506 to accurately correct skeletal information at a first point in time in the time-series data of the subject's skeletal information. For example, the generating unit 506 enables the positions of other body parts determined to be abnormal to be accurately corrected.

[0081] The correction unit 507 corrects the skeletal information at the first time point in the time-series data of the skeletal information of the subject based on the generated graph. The correction unit 507 corrects the skeletal information at the first time point in the time-series data of the skeletal information of the subject, for example, by optimizing the generated graph. This allows the correction unit 507 to accurately identify the position of each body part of the subject, taking into account the type of movement of the subject. The correction unit 507 allows the position of each body part of the subject to be accurately identified, taking into account the probability that each body part of the subject is in an abnormal state.

[0082] The output unit 508 outputs the processing result of at least one of the functional units. The output format is, for example, display on a display, printout to a printer, transmission to an external device via the network I / F 303, or storage in a storage area such as the memory 302 or the recording medium 305. In this way, the output unit 508 can notify the user of the processing result of at least one of the functional units, thereby improving the convenience of the information processing device 100.

[0083] The output unit 508 outputs, for example, the skeletal information at the first time point corrected by the correction unit 507. Specifically, the output unit 508 transmits the skeletal information at the first time point corrected by the correction unit 507 to the client device 202. Specifically, the output unit 508 displays the skeletal information at the first time point corrected by the correction unit 507 on a display. In this way, the output unit 508 can make the position of each part of the subject available.

[0084] (Example of operation of information processing device 100) Next, an example of the operation of the information processing device 100 will be described with reference to Fig. 6 to Fig. 15. First, the flow of the operation of the information processing device 100 will be described with reference to Fig. 6, for example.

[0085] Fig. 6 is an explanatory diagram showing the flow of operation of information processing device 100. In Fig. 6, information processing device 100 acquires a plurality of multi-view images 600 in which a subject is captured at different angles at different times. Information processing device 100 performs a person detection process on each of the plurality of multi-view images 600, thereby detecting an area in each multi-view image 600 in which the subject is captured.

[0086] For each time point, the information processing device 100 performs a 2D (dimensional) pose estimation process on each multi-view image 600. By performing the 2D pose estimation process on each multi-view image 600 for each time point, the information processing device 100 generates a 2D heat map 601 indicating the distribution of the presence probability of each joint of the subject in each multi-view image 600. The 2D heat map 601 includes, for example, joint likelihoods indicating the presence probability of any joint of the subject at each point in 2D space corresponding to the multi-view image 600.

[0087] The information processing device 100 identifies the 2D coordinates of the subject's joints in the multi-view images 600 based on a 2D heat map 601 indicating the distribution of the presence probability of each joint of the subject in each multi-view image 600 for each time point. The variance of the joint likelihood indicating the presence probability of the subject's joint in the 2D heat map 601 can be used as an index value indicating the accuracy of the identified 2D coordinates.

[0088] The information processing device 100 acquires, for each time point, placement information indicating the angle of each multi-view image 600. For each time point, the information processing device 100 performs a 3D pose estimation process based on the placement information and the 2D coordinates of each joint of the subject in each multi-view image 600, thereby identifying the 3D coordinates of each joint of the subject in 3D space. For each time point, the information processing device 100 generates a 3D skeleton inference result 602 including the 3D coordinates of each identified joint of the subject, and generates time-series data of the 3D skeleton inference result 602.

[0089] The information processing device 100 corrects the 3D skeleton inference result 602 by performing a correction process on the time-series data of the 3D skeleton inference result 602. The information processing device 100 outputs the time-series data of the corrected 3D skeleton inference result 603 so that it can be used. For example, the information processing device 100 outputs the time-series data of the corrected 3D skeleton inference result 603 so that it can be referenced by a user.

[0090] The user performs a predetermined analysis process based on the time-series data of the corrected 3D skeleton inference result 603. Specifically, the subject may be a participant in an athletic competition. In this case, the analysis process is, for example, scoring the participants in the athletic competition. The user performs an analysis process to score the participants based on the time-series data of the corrected 3D skeleton inference result 603.

[0091] Specifically, the subject may be a patient undergoing rehabilitation at a medical institution, or a patient undergoing a diagnosis of motor skills such as walking ability. In this case, the analysis process may be, for example, assessing the effectiveness of rehabilitation or diagnosing motor skills or health status. Based on the time-series data of the corrected 3D skeleton inference result 603, the user assesses the effectiveness of rehabilitation for the patient at the medical institution, or diagnoses the patient's motor skills or health status.

[0092] The information processing device 100 may perform the above-described analysis process based on the time-series data of the corrected 3D skeleton inference result 603. The information processing device 100 outputs the result of the analysis process so that the user can refer to it. The information processing device 100 may output the time-series data of the corrected 3D skeleton inference result 603 to the analysis unit 502 that performs the above-described analysis process. The analysis unit 502 is included in, for example, a computer other than the information processing device 100. This enables the information processing device 100 to perform the analysis process with high accuracy.

[0093] Next, a specific example of the correction process will be described with reference to Fig. 7 to Fig. 15. Specifically, first, a specific example will be described with reference to Fig. 7 and Fig. 8 in which the information processing device 100 identifies an abnormal joint that is determined to be in an abnormal state in terms of 3D coordinates, among a plurality of joints of a subject.

[0094] 7 and 8 are explanatory diagrams showing a specific example of identifying an abnormal joint. In FIG. 7, the information processing device 100 acquires time-series data of a plurality of original data 700. The original data 700 indicates skeletal information of a subject. The original data 700 indicates the 3D coordinates of each of the subject's joints. The 3D coordinates of the joints are, for example, indicated by ● in the diagram.

[0095] The information processing device 100 generates processed data 701 by adding noise to original data 700. The information processing device 100 generates the processed data 701, for example, by changing the 3D coordinates of at least one of the multiple joints of the subject indicated by the original data 700 to 3D coordinates determined to be in an abnormal state. The abnormal state corresponds to, for example, a state in which the 3D coordinates of the joints are erroneously estimated. Specifically, the abnormal state is jitter, inversion, swap, or miss. In this way, the information processing device 100 can acquire time-series data of the processed data 701.

[0096] The information processing device 100 uses the time-series data of the processed data 701 to train the abnormality determination DNN 710. The abnormality determination DNN 710 has a function of outputting the abnormality probability of each joint of the subject in at least one of the 3D skeleton inference results 602 in response to input of the feature amount of the 3D skeleton inference result 602 in the time-series data of the 3D skeleton inference result 602. The abnormality probability indicates the magnitude of the probability that the 3D coordinates of the joints of the subject are in a positionally abnormal state.

[0097] The abnormality determination DNN 710 may have a function of outputting the abnormality probability of each joint of the subject in the entire time-series data in response to input of the feature amount of the 3D skeleton inference result 602 in the time-series data of the 3D skeleton inference result 602. Next, we move on to the description of FIG.

[0098] 8, the information processing device 100 inputs the feature quantities of the 3D skeleton inference result 602 in the time-series data of the 3D skeleton inference result 602 to the abnormality determination DNN 710. The information processing device 100 acquires the abnormality probability of each joint of the subject in each 3D skeleton inference result 602 output by the abnormality determination DNN 710 in response to the input. The information processing device 100 identifies an abnormal joint based on the acquired abnormality probability of each joint of the subject. For example, the information processing device 100 identifies, as an abnormal joint, any joint of the subject's multiple joints for which the acquired abnormality probability is equal to or greater than a threshold.

[0099] Here, the case where the information processing device 100 identifies an abnormal joint using the abnormality determination DNN 710 has been described, but this is not limiting. For example, the information processing device 100 may identify an abnormal joint based on rules. Specifically, the information processing device 100 may store rules for calculating the abnormality probability of each joint in the 3D skeleton inference result 602 according to the magnitude of the difference between the feature amount for that joint and a threshold. Specifically, the information processing device 100 may refer to the stored rules to calculate the abnormality probability of each joint, and identify any joint whose calculated abnormality probability is equal to or greater than the threshold as an abnormal joint. Next, a specific example in which the information processing device 100 generates a factor graph will be described with reference to FIG. 9 .

[0100] 9 is an explanatory diagram showing a specific example of generating a factor graph. In FIG. 9, the information processing device 100 has a state estimation DNN 900. The state estimation DNN 900 has a function of outputting the type of motion of the subject in at least one of the 3D skeleton inference results 602 in response to input of feature quantities of the 3D skeleton inference results 602 in the time-series data of the 3D skeleton inference results 602.

[0101] The state estimation DNN 900 may have a function of outputting the type of movement of the subject in the entire time series data in response to input of the feature of the 3D skeleton inference result 602 in the time series data of the 3D skeleton inference result 602, for example.

[0102] The information processing device 100 has a Factor Graph definition DB (DataBase) 910. The Factor Graph definition DB 910 stores a Factor Graph template 911 for each type of motion of the subject. The template 911 is formed, for example, by nodes indicating the positions of each joint of the subject, first edges connecting nodes indicating the positions of different biologically connected joints, and second edges connecting nodes indicating the positions of the same joint at different points in time. The first edges may be associated with a constraint on the distance between joints. The distance between joints is, for example, the length of a bone.

[0103] Specifically, the Factor Graph definition DB 910 stores a template 911 corresponding to the type of exercise "jumping" and a template 911 corresponding to the type of exercise "lying down." The second edge connects, for example, nodes indicating the position of any joint corresponding to each type of exercise of the subject. In other words, the second edge connects, for example, nodes indicating different joint positions for each type of exercise of the subject.

[0104] A distribution model is associated with the second edge. A distribution model showing a probability distribution that constrains the time change in the position of any of the joints is associated with the second edge connecting nodes that indicate the position of any of the joints according to the tendency of the movement of the any of the joints corresponding to the type of movement. For example, if the type of movement is "jumping," the tendency corresponds to uniform linear movement. For example, if the type of movement is "lying down," the tendency corresponds to uniform position movement.

[0105] The information processing device 100 uses the state estimation DNN 900 to identify the type of motion of the subject in each 3D skeleton inference result 602. The information processing device 100 refers to the Factor Graph definition DB 910 and selects a template 911 corresponding to the type of motion of the subject in each 3D skeleton inference result 602 as the Factor Graph to be used. Next, a specific example of the Factor Graph template 911 will be described with reference to FIGS. 10 and 11 .

[0106] 10 is an explanatory diagram showing a specific example of a template 911 of a Factor Graph corresponding to "jump." Template 911 includes, for example, nodes indicating the position of each joint of the subject. Specifically, template 911 includes nodes indicating the positions of the subject's head, upper cervical vertebrae, lower cervical vertebrae, thoracic vertebrae, lumbar vertebrae, left and right hip joints, left and right knee joints, left and right leg joints, left and right feet, left and right shoulder joints, left and right elbow joints, left and right wrists, and left and right hands.

[0107] 10, nodes indicating the positions of the subject's lower cervical vertebrae at different times are connected by a second edge 1001. Furthermore, nodes indicating the positions of the subject's thoracic vertebrae at different times are connected by a second edge 1001. Furthermore, nodes indicating the positions of the subject's lumbar vertebrae at different times are connected by a second edge 1001.

[0108] Furthermore, nodes indicating the positions of the subject's left hip joint at different times are connected by a second edge 1001. Furthermore, nodes indicating the positions of the subject's right hip joint at different times are connected by a second edge 1001. Each second edge is associated with a distribution model of Pairwise Term indicating a time series constraint corresponding to uniform linear motion.

[0109] Pairwise Term is, for example, g t (x j,t-1 ,x j,t )~N(||x j,t-1 ,x j,t ||┃v j ^Δt,Σ vj ^) x j,t-1is the estimated position of the joint at time t-1. j,t is the estimated position of the joint at time t. j ^ is the average velocity of the joint. Δt is the unit time width. Σ vj ^ is the velocity variance of the joint. Here, when the type of motion is "jump", it is considered that the time change in the position of the joints in the trunk part is likely to be regular. In contrast, when the type of motion is "jump", template 911 can constrain the time change in position of the joints in the trunk part, whose time change in position is considered to be easy to predict, by assuming uniform linear motion. Next, we move on to the explanation of Figure 11.

[0110] 11 is an explanatory diagram showing a specific example of a template 911 of a Factor Graph corresponding to "lying down." Template 911 includes, for example, nodes indicating the position of each joint of the subject. Specifically, template 911 includes nodes indicating the positions of the subject's head, upper cervical vertebrae, lower cervical vertebrae, thoracic vertebrae, lumbar vertebrae, left and right hip joints, left and right knee joints, left and right leg joints, left and right feet, left and right shoulder joints, left and right elbow joints, left and right wrists, and left and right hands.

[0111] In the example of FIG. 11 , nodes indicating the positions of the subject's head at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's upper cervical vertebrae at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's lower cervical vertebrae at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's thoracic vertebrae at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's lumbar vertebrae at different times are connected to each other by a second edge 1101.

[0112] Furthermore, nodes indicating the positions of the subject's left hip joints at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's right hip joints at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's left knee joints at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's right knee joints at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's left leg joints at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's right leg joints at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's left foot at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's right foot at different times are connected to each other by a second edge 1101.

[0113] Furthermore, nodes indicating the positions of the subject's left shoulder joints at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's right shoulder joints at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's left elbow joints at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's right elbow joints at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's left wrists at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's right wrists at different times are connected to each other by a second edge 1101.

[0114] Furthermore, nodes indicating the positions of the subject's left hand at different times are connected to each other by a second edge 1101. Furthermore, nodes indicating the positions of the subject's right hand at different times are connected to each other by a second edge 1101. In the example of FIG. 11, for convenience of illustration, some of the second edges 1101 are omitted from the illustration. Each second edge is associated with a distribution model of Pairwise Term indicating a time series constraint corresponding to equal position movement.

[0115] Pairwise Term is, for example, g t (x j,t-1 ,x j,t)~N(||x j,t-1 ,x j,t ||┃0,Σ xj ^) is. Σ xj ^ is the position variance of the joints. Here, when the type of exercise is "lying down", it is considered that the time change in the positions of the joints of the whole body is likely to be regular. In contrast, when the type of exercise is "lying down", the template 911 can constrain the time change in position for the joints of the whole body whose time change in position is considered to be easy to predict by assuming equal position movement. Next, a specific example in which the information processing device 100 adds a time series constraint to a selected Factor Graph will be described with reference to FIG. 12.

[0116] FIG. 12 is an explanatory diagram showing a specific example of adding a time series constraint. The information processing device 100 determines whether a leaf node in the selected factor graph that is not connected to a second edge and is connected to one first edge is a node indicating the position of the identified abnormal joint. If the leaf node is a node indicating the position of the identified abnormal joint, the information processing device 100 connects the leaf nodes at different time points with a third edge 1201. This enables the information processing device 100 to accurately correct the position of the abnormal joint. Next, a specific example will be described with reference to FIG. 13 in which the information processing device 100 corrects the 3D skeleton inference result 602 using the selected factor graph 1300.

[0117] Fig. 13 is an explanatory diagram showing a specific example of correcting the 3D skeleton inference result 602. In Fig. 13, the information processing device 100 corrects the 3D skeleton inference result 602 by using a selected factor graph 1300. In the example of Fig. 13, the factor graph 1300 includes a node group 1310 corresponding to time t-1, a node group 1320 corresponding to time t, and the like. The node group 1310 includes nodes 1311 to 1313, and the like. The node group 1320 includes nodes 1321 to 1323, and the like.

[0118] For example, node 1311 and node 1312 are connected by a first edge 1331. For example, node 1312 and node 1313 are connected by a first edge 1332. For example, node 1321 and node 1322 are connected by a first edge 1341. For example, node 1322 and node 1323 are connected by a first edge 1342. For example, first edge 1342 connecting node 1322 and node 1323 may be associated with a Pairwise Term indicating a bone length constraint.

[0119] For example, node 1312 and node 1322 are connected by a second edge 1351. A Pairwise Term indicating the above-mentioned time series constraint corresponding to the type of motion of the subject is associated with the second edge 1351. For example, node 1311 and node 1321 are connected by a third edge 1361. The third edge 1361 may be associated with a Pairwise Term indicating the above-mentioned time series constraint.

[0120] The information processing device 100 may associate a Unary Term with a node indicating the position of at least one of the joints in the Factor Graph 1300. The Unary Term may be, for example, f(x j )~N(x j |x j ^,Σ 3D j ^) x j ^ is the weighted sum of the joint likelihoods of the 3D heatmap, which is the joint likelihood of multiple 2D heatmaps combined. Σ 3D j ^ is the variance of the joint likelihood of the 3D heat map that integrates the joint likelihoods of multiple 2D heat maps.

[0121] The information processing device 100 may associate a unary term indicating a constraint on an abnormal joint, which acts to constrain the position of the joint in accordance with the abnormality probability of the joint, with a node indicating the position of at least one joint in the factor graph 1300. In the example of FIG. 13, for example, the information processing device 100 may associate a unary term including the abnormality probability of joint 1 with node 1321 indicating the position of joint 1 in the node group 1320. The unary term may be, for example, f(x j )~N(x j |x j ^,Σ 3D j ^)·p(x j ) is p(x j ) is the abnormal probability.

[0122] The information processing device 100 corrects the position of each joint at each time point based on the unary term and the pairwise term in the factor graph 1300. The information processing device 100 corrects the position of each joint at each time point, for example, by optimizing the factor graph 1300.

[0123] This allows the information processing device 100 to accurately correct the 3D skeleton inference result 602. The information processing device 100 can accurately identify the position of each joint at each time point. For example, even when the subject performs relatively high-speed or relatively complex movements such as gymnastics, the information processing device 100 can identify the position of each joint of the subject at each time point with relatively high accuracy.

[0124] Here, with reference to Reference 3 below, Comparative Example 1 can be considered, in which a Factor Graph that does not include a Pairwise Term indicating a time series constraint is used to correct the 3D coordinates of the joints of a subject. In Comparative Example 1, it is not possible to constrain the change in the position of the joints over time, so it is difficult to accurately correct the 3D coordinates of each joint of the subject, and it may be difficult to accurately identify the change in the 3D coordinates of each joint of the subject over time.

[0125] Reference 3: Bultmann, Simon, and Sven Behnke. “Real-time multi-view 3D human pose estimation using semantic feedback to smart edge sensors.” arXiv preprint arXiv:2106.14729 (2021).

[0126] In response to this, the information processing device 100 can use a Factor Graph 1300 that includes a Pairwise Term that indicates a time series constraint. This allows the information processing device 100 to appropriately correct the 3D coordinates of each joint of the subject. For example, the information processing device 100 can appropriately correct the 3D coordinates of the joints of the subject at each time point so that the time change from the 3D coordinates of the joints of the subject at one time point to the 3D coordinates of the joints of the subject at the next time point is a time change that is unlikely to be intuitively perceived as an error by a person.

[0127] Furthermore, consider Comparative Example 2, in which a Factor Graph including a Pairwise Term indicating a predetermined time series constraint is used to correct the 3D coordinates of the subject's joints. However, even in Comparative Example 2, it is difficult to accurately correct the 3D coordinates of each of the subject's joints, and it may be difficult to accurately identify the time change of the 3D coordinates of each of the subject's joints. In Comparative Example 2, it is not possible to dynamically change the Pairwise Term indicating the time series constraint in accordance with the subject's state, such as the type of movement, so it is difficult to accurately correct the 3D coordinates of each of the subject's joints.

[0128] In response to this, the information processing device 100 can set the Factor Graph 1300 by selectively using a plurality of Factor Graph templates 911 including Pairwise Term indicating different time series constraints depending on the type of motion of the subject. This allows the information processing device 100 to selectively use Pairwise Term indicating time series constraints corresponding to constant position motion, constant velocity linear motion, constant acceleration motion, or the like depending on the type of motion of the subject. Furthermore, the information processing device 100 can connect the second edge corresponding to the Pairwise Term indicating the time series constraint to a node indicating the 3D coordinates of a different joint depending on the type of motion of the subject.

[0129] Therefore, the information processing device 100 can appropriately correct the 3D coordinates of each joint of the subject. For example, the information processing device 100 can appropriately correct the 3D coordinates of the joints of the subject at each time point so that the time change from the 3D coordinates of the joints of the subject at one time point to the 3D coordinates of the joints of the subject at the next time point is a time change that is unlikely to be intuitively perceived as an error by a person. Next, a specific example of the flow of data processing in the operation example will be described with reference to FIGS. 14 and 15.

[0130] 14 and 15 are explanatory diagrams showing a specific example of the flow of data processing in the operation example. As shown in Fig. 14, the information processing device 100 acquires multiple camera images 1401 at each time point. The information processing device 100 stores a 2D skeletal inference model 1410. The information processing device 100 stores, for example, weight parameters that define a neural network that becomes the 2D skeletal inference model 1410.

[0131] The information processing device 100 refers to the 2D skeleton inference model 1410 for each time point and performs 2D skeleton inference processing on each of the multiple camera images 1401 to generate a 2D skeleton inference result 1402. The 2D skeleton inference result 1402 includes, for example, 2D coordinates (x [pixel], y [pixel]) indicating the position of the joint and likelihoods indicating the accuracy of the joint position.

[0132] The information processing device 100 stores the 3D skeletal inference model 1420. The information processing device 100 stores, for example, weight parameters that define the neural network that becomes the 3D skeletal inference model 1420.

[0133] The information processing device 100 generates a 3D skeleton inference result 1403 by performing a 3D skeleton inference process on a plurality of 2D skeleton inference results 1402 with reference to the 3D skeleton inference model 1420 for each time point. The 3D skeleton inference result 1403 includes, for example, 3D coordinates (x [mm], y [mm], z [mm]) indicating the positions of the joints. The information processing device 100 generates time-series data 1404 that compiles the 3D skeleton inference results 1403 for each time point. Next, we move on to the description of FIG. 15.

[0134] 15, the information processing device 100 stores an exercise state estimation model 1510. The information processing device 100 stores, for example, weight parameters that define a neural network that serves as the exercise state estimation model 1510.

[0135] The information processing device 100 estimates the type of exercise of the subject by performing an exercise state estimation process on the time series data 1404 with reference to the exercise state estimation model 1510, and generates an exercise state estimation result 1501 including the estimated type of exercise of the subject.

[0136] The information processing device 100 stores a Factor Graph definition DB 1520. The Factor Graph definition DB 1520 stores, for each type of exercise, a template of a Factor Graph corresponding to the type of exercise, including a Pairwise Term indicating a time-series constraint. The Pairwise Term indicates, for example, that the time change in the position of a joint corresponding to the type of exercise is constrained according to the tendency of the subject's movement corresponding to the type of exercise.

[0137] The Factor Graph definition DB 1520 indicates, for example, a type of motion, a type of joint of the subject, and a tendency of motion of the joint of the subject corresponding to the type of motion, in association with each other. The tendency of motion is, for example, constant position motion, constant velocity linear motion, constant acceleration motion, etc. The information processing device 100 refers to the Factor Graph definition DB 1520 and selects, as the Factor Graph to be used, a template of a Factor Graph that corresponds to the estimated type of motion of the subject included in the motion state estimation result 1501.

[0138] The information processing device 100 stores a bone length model 1530. The bone length model 1530 includes parameters that define a pairwise term that indicates a constraint on the bone length. The parameters are, for example, the mean and variance of the bone length. The information processing device 100 refers to the bone length model 1530 and assigns the pairwise term that indicates the constraint on the bone length to a selected factor graph.

[0139] The information processing device 100 performs an optimization process on the factor graph after the assignment, thereby correcting the position of each joint. The information processing device 100 generates a corrected 3D inferred skeleton model 1502 including the corrected positions of each joint. This allows the information processing device 100 to accurately identify the position of each joint of the subject at each time point.

[0140] (Overall processing procedure) Next, an example of an overall processing procedure executed by the information processing device 100 will be described with reference to Fig. 16. The overall processing is realized by, for example, the CPU 301, storage areas such as the memory 302 and the recording medium 305, and the network I / F 303 shown in Fig. 3.

[0141] Fig. 16 is a flowchart showing an example of an overall processing procedure. In Fig. 16, the information processing device 100 acquires time-series data of the subject's three-dimensional skeleton inference result (step S1601). Then, the information processing device 100 calculates the likelihood of each part of the subject based on the acquired time-series data of the subject's three-dimensional skeleton inference result (step S1602).

[0142] Next, the information processing device 100 estimates the subject's motion state for each time point based on the acquired time-series data of the subject's three-dimensional skeleton inference result (step S1603).Then, the information processing device 100 selects a factor graph corresponding to the estimated motion state of the subject for each time point (step S1604).

[0143] Next, the information processing device 100 sets Pairwise Term along the time axis for leaf nodes corresponding to parts with relatively small likelihood in the selected Factor Graph (step S1605).Then, the information processing device 100 sets Unary Term=Likelihood for a specific part of the subject in the selected Factor Graph, and sets Pairwise Term along the time axis for the specific part of the subject (step S1606).

[0144] Next, the information processing device 100 corrects the time series data of the subject's three-dimensional skeleton inference result by optimizing the factor graph (step S1607). Then, the information processing device 100 outputs the corrected time series data of the subject's three-dimensional skeleton inference result (step S1608). After that, the information processing device 100 ends the entire process.

[0145] This allows the information processing device 100 to accurately correct the subject's three-dimensional skeleton inference result. Therefore, the information processing device 100 can improve the usefulness of the subject's three-dimensional skeleton inference result. The information processing device 100 can, for example, improve the accuracy of analysis processing based on the subject's three-dimensional skeleton inference result.

[0146] Here, the information processing device 100 may change the order of the processes of some of the steps in Fig. 16. For example, the order of the processes of steps S1605 and S1606 can be changed. Furthermore, the information processing device 100 may omit some of the processes of the steps in Fig. 16. For example, the process of step S1605 can be omitted.

[0147] As described above, the information processing device 100 can acquire time-series data of skeletal information including the positions of each of a plurality of body parts of a subject. The information processing device 100 can identify the type of motion of the subject corresponding to the skeletal information at a first time point in the acquired time-series data based on the feature quantities of the skeletal information in the acquired time-series data. The information processing device 100 can determine a probability distribution model that constrains the temporal change in the position of any of the plurality of body parts in the skeletal information at the first time point in the acquired time-series data according to the tendency of the movement of any of the body parts corresponding to the identified type of motion. The information processing device 100 can generate a graph including nodes indicating the positions of each body part at each time point. The information processing device 100 can add a first edge to the graph, connecting nodes indicating the positions of different body parts that are biologically connected at each time point. The information processing device 100 can add a second edge to the graph, connecting nodes indicating the positions of any of the body parts at different time points. The information processing device 100 can associate the determined model with the second edge in the graph. According to the information processing device 100, it is possible to correct the skeletal information at the first time point in the time-series data based on the generated graph. This allows the information processing device 100 to accurately correct the skeletal information at the first time point.

[0148] The information processing device 100 can determine a probability distribution model that constrains the change over time in the position of any part of the skeletal information at the first time point according to the tendency of the part to be in uniform positional motion, uniform velocity motion, or uniform acceleration motion, corresponding to the identified type of movement. This allows the information processing device 100 to determine a model that allows the skeletal information at the first time point to be appropriately corrected in accordance with the type of movement.

[0149] The information processing device 100 can determine whether, for other parts other than one of the multiple parts, there is one first edge connected to each of the nodes indicating the positions of the other parts at different times. The information processing device 100 can generate a graph such that, if there is one first edge connected to each of the nodes indicating the positions of the other parts at different times, a third edge connecting the nodes is included in the graph. This allows the information processing device 100 to increase the number of edges connected to a node and enable accurate correction of the positions of the other parts indicated by the node.

[0150] According to the information processing device 100, it is possible to identify other parts among a plurality of parts that are not part of any of the parts and that are in an abnormal state with respect to their position. According to the information processing device 100, if there is one first edge connected to each of the nodes indicating the positions of the other parts at different times for the identified other parts, it is possible to generate a graph such that a third edge connecting the nodes is included in the graph. This allows the information processing device 100 to identify other parts that are preferable to correct, and makes it possible to accurately correct the positions of the identified other parts.

[0151] The information processing method described in this embodiment can be realized by executing a prepared program on a computer such as a PC or a workstation. The information processing program described in this embodiment is recorded on a computer-readable recording medium and executed by being read from the recording medium by the computer. The recording medium may be a hard disk, a flexible disk, a CD (Compact Disc)-ROM, an MO (Magneto Optical disc), a DVD (Digital Versatile Disc), or the like. The information processing program described in this embodiment may also be distributed via a network such as the Internet. [Explanation of symbols]

[0152] 100 Information processing device 101 Skeletal Information 110 graphs 111, 1311~1313, 1321~1323 nodes 112,1331,1332,1341,1342 First edge 113,1001,1101,1351 Second Edge 200 Information Processing Systems 201 Image capturing device 202 Client device 210 Network 300,400 buses 301,401 CPU 302,402 memory 303,403 Network I / F 304,404 Recording media I / F 305,405 Recording media 306 Display 307 Input Device 406 Camera 500 storage section 501 Acquisition Department 502 Analysis Department 503 Learning Department 504 Specific part 505 Decision Section 506 Generation part 507 Correction section 508 Output Section 600 multi-view images 601 2D Heatmaps 602,603,1403,1502 3D skeleton inference results 700 original data 701 Processing Data 710 Abnormality judgment DNN 900 State Estimation DNN 910,1520 Factor Graph definition DB 911 template 1201,1361 Third Edge 1300 Factor Graph 1310,1320 node group 1401 camera images 1402 2D skeleton inference results 1404 Time Series Data 1410 2D Skeletal Inference Model 1420 3D Skeletal Inference Model 1501 Motion state estimation results 1510 Motion State Estimation Model 1530 Bone Length Model

Claims

1. Acquire time-series data of skeletal information including the position of each of a plurality of body parts of the subject; identifying a first type of motion of the subject corresponding to skeletal information at a first time point in the acquired time-series data based on a feature amount of skeletal information in the acquired time-series data; determining a probability distribution model that constrains the temporal change in the position of each part of the body part, for which the edges are to be set, in accordance with the tendency of movement of the part corresponding to the identified first type of the body part, in the skeletal information at the first time point in the acquired time-series data, for which the edges are to be set, in accordance with the tendency of movement of the part corresponding to the identified first type of the body part; referencing the storage unit, generating a graph including nodes indicating the positions of the respective body parts at each time point, first edges connecting the nodes indicating the positions of different body parts that are biologically connected at each time point, and second edges connecting the nodes indicating the positions of the respective parts of the partial body parts at different time points, and associating the second edges corresponding to the respective parts of the partial body parts with the model determined for the respective parts; correcting skeletal information at the first time point in the time-series data based on the generated graph; An information processing program that causes a computer to execute a process.

2. The determining process includes:

2. The information processing program according to claim 1, further comprising: determining a probability distribution model that constrains the change over time in the position of each of the partial body parts of the skeletal information at the first time point in the acquired time-series data in accordance with a tendency of the body parts to uniform positional movement, uniform velocity movement, or uniform acceleration movement corresponding to the identified first type of movement.

3. The generating process includes: The information processing program according to claim 1 or 2, characterized in that, for parts other than the part among the plurality of parts, if there is one first edge connected to each node indicating the position of the other parts at different points in time, the graph is generated so that a third edge connecting the nodes is included in the graph.

4. Identifying other parts of the plurality of parts that are in an abnormal state with respect to their positions, other than the partial parts. causing the computer to execute a process; The generating process includes: The information processing program according to claim 3, characterized in that, for the identified other parts, if there is one first edge connected to each of the nodes indicating the positions of the other parts at different points in time, the graph is generated so that a third edge connecting the nodes is included in the graph.

5. Acquire time-series data of skeletal information including the position of each of a plurality of body parts of the subject; identifying a first type of motion of the subject corresponding to skeletal information at a first time point in the acquired time-series data based on a feature amount of skeletal information in the acquired time-series data; determining a probability distribution model that constrains the temporal change in the position of each part of the body part, for which the edges are to be set, in accordance with the tendency of movement of the part corresponding to the identified first type of the body part, in the skeletal information at the first time point in the acquired time-series data, for which the edges are to be set, in accordance with the tendency of movement of the part corresponding to the identified first type of the body part; referencing the storage unit, generating a graph including nodes indicating the positions of the respective body parts at each time point, first edges connecting the nodes indicating the positions of different body parts that are biologically connected at each time point, and second edges connecting the nodes indicating the positions of the respective parts of the partial body parts at different time points, and associating the second edges corresponding to the respective parts of the partial body parts with the model determined for the respective parts; correcting skeletal information at the first time point in the time-series data based on the generated graph; An information processing method characterized in that the processing is executed by a computer.

6. Acquire time-series data of skeletal information including the position of each of a plurality of body parts of the subject; identifying a first type of motion of the subject corresponding to skeletal information at a first time point in the acquired time-series data based on a feature amount of skeletal information in the acquired time-series data; determining a probability distribution model that constrains the temporal change in the position of each part of the body part, for which the edges are to be set, in accordance with the tendency of movement of the part corresponding to the identified first type of the body part, in the skeletal information at the first time point in the acquired time-series data, for which the edges are to be set, in accordance with the tendency of movement of the part corresponding to the identified first type of the body part; referencing the storage unit, generating a graph including nodes indicating the positions of the respective body parts at each time point, first edges connecting the nodes indicating the positions of different body parts that are biologically connected at each time point, and second edges connecting the nodes indicating the positions of the respective parts of the partial body parts at different time points, and associating the second edges corresponding to the respective parts of the partial body parts with the model determined for the respective parts; correcting skeletal information at the first time point in the time-series data based on the generated graph; An information processing device comprising a control unit.

Images