Information processing program, information processing method, and information processing device

Abstract

The information processing device (100) specifies, on the basis of the feature amount relating to the bone information (101) in the time-series data, any part in an abnormal state with respect to the position for the bone information (101) at the first time in the time-series data. The information processing device (100) determines a model that limits the probability distribution of the position of the portion in the bone information (101) at the first time in accordance with the magnitude of the probability of the determined portion being in the abnormal state. The information processing device (100) generates a graph (110) that includes a node (111) indicating the position of each part at each time point and a first edge (112), and associates the determined model with the node (111) indicating the position of any specified part. The information processing device (100) corrects the skeleton information (101) at the first time in the time-series data on the basis of the generated graph (110).

Description

Technical Field

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. Background Art

[0002] Conventionally, in the fields of sports, health care, or entertainment, there is a desire for technology to recognize human movements. For example, there is a technology that uses deep learning to determine the three-dimensional coordinates of each joint of a person based on multi-viewpoint images taken from different angles.

[0003] As a prior art, for example, there is a technology that outputs any one of the results of the first process, the results of the second process, and the results of the third process as a result of skeleton recognition of the subject based on the likelihood of the result of the first process, the likelihood of the result of the second process, and the likelihood of the result of the third process. In addition, for example, there is a technology that recognizes a heat map image that projects the likelihood of multiple joint positions of the subject from multiple directions based on a range image of the subject. In addition, for example, there is a technology that performs an optimization calculation based on the inverse kinematics of the multi-joint structure of the object using position candidates of feature points and obtains each joint angle of the object, and performs forward kinematics calculation using the joint angles to obtain the positions of feature points including the joints of the object. In addition, for example, there is a technology for action detection using a recurrent neural network.

[0004] Patent Document 1: International Publication No. 2021 / 064942

[0005] Patent Document 2: International Publication No. 2021 / 002025

[0006] Patent Document 3: Japanese Patent Application Publication No. 2020-42476

[0007] Patent Document 4: U.S. Patent Application Publication No. 2017 / 0344829

[0008] However, in the prior art, it is difficult to accurately determine the three-dimensional coordinates of each joint of a person. For example, the three-dimensional coordinates of the joints of a person's right hand may be mistaken for the three-dimensional coordinates of the joints of the person's left hand. For example, the three-dimensional coordinates of a part of an object other than a person reflected in a multi-viewpoint image may be mistaken for the three-dimensional coordinates of a person's joints. Summary of the invention

[0009] In one aspect, an object of the present invention is to accurately identify the position of a part of a subject.

[0010] According to one embodiment, an information processing program, an information processing method, and an information processing device are proposed, which obtain time series data of skeletal information of the positions of each of multiple parts of an object, determine any one part of the skeletal information at the first moment in the acquired time series data that is in an abnormal state based on the feature quantity related to the skeletal information in the acquired time series data, determine a model of the probability distribution of the position of any one of the parts determined in the skeletal information at the first moment according to the probability of the determined any one of the parts being in an abnormal state based on the feature quantity related to the skeletal information in the acquired time series data, generate a node representing the position of each of the parts at each moment, and a first edge connecting the nodes representing the positions of different parts connected on the biological body at each of the above moments, and establish a graph in which the determined model and the nodes representing the position of any one of the parts are corresponding to each other, and correct the skeletal information at the first moment in the time series data based on the generated graph.

[0011] According to one aspect, the position of a part of a subject person can be identified with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is an explanatory diagram showing an example of an information processing method according to an embodiment.

[0013] Figure 2 It is an explanatory diagram showing an example of the information processing system 200 .

[0014] Figure 3 It is a block diagram showing a hardware configuration example of the information processing device 100 .

[0015] Figure 4 It is a block diagram showing an example of the hardware configuration of the image pickup device 201 .

[0016] Figure 5 It is a block diagram showing an example of the functional configuration of the information processing device 100 .

[0017] Figure 6 It is an explanatory diagram showing the flow of operations of the information processing device 100 .

[0018] Figure 7 This is an explanatory diagram (part 1) showing a specific example of identifying an abnormal joint.

[0019] Figure 8 This is an explanatory diagram (part 2) showing a specific example of identifying an abnormal joint.

[0020] Fig. 9 900 is an explanatory diagram showing a specific example of the generated factor graph 900.

[0021] Fig.10 It is an explanatory diagram showing a specific example of the corrected 3D skeleton inference result 602.

[0022] Fig.11 This is an explanatory diagram (No. 1) showing a specific example of the flow of data processing in the operation example.

[0023] Fig.12 This is an explanatory diagram (part 2) showing a specific example of the flow of data processing in the operation example.

[0024] Fig.13 This is a flowchart showing an example of the overall processing sequence. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of an information processing program, an information processing method, and an information processing device according to the present invention will be described in detail with reference to the accompanying drawings.

[0026] (An Example of the Information Processing Method According to the Embodiment)

[0027] Figure 1 1 is an explanatory diagram showing an example of an information processing method according to an embodiment. The information processing device 100 is a computer for accurately determining the position of a joint of a subject. The position is, for example, a three-dimensional coordinate.

[0028] Conventionally, there is a technology for determining the time change of the three-dimensional coordinates of each joint of a person by using deep learning based on multi-viewpoint images taken from different angles at each time.

[0029] Specifically, consider detecting the area where a person is reflected in a multi-view image, determining the two-dimensional coordinates of each joint of the person based on the detected area, and considering the angle, determining the three-dimensional coordinates of each joint of the person based on the determined two-dimensional coordinates. Specifically, when determining the three-dimensional coordinates of each joint of the person, a model learned by deep learning is used. Specifically, an example of this technology can be referred to the following reference 1 and the following reference 2.

[0030] Reference 1: Iskakov, Karim, et al. “Learnable triangulation of humanpose.” Proceedings of the IEEE / CVF International Conference on ComputerVision.2019.

[0031] Reference 2: Moon, Gyeongsik, Ju Yong Chang, and Kyoung Mu Lee. "V2v-posenet: Voxel-to-voxel prediction network for accurate 3dhand and humanpose estimation from a singledepth map." Proceedings of the IEEE conference on computer vision and pattern Recognition. 2018.

[0032] However, in the prior art, it is difficult to accurately determine the three-dimensional coordinates of each joint of a person. For example, even if the distance between the joints of the same person is constant at different times, when learning the model through deep learning, the constant distance between the joints of a person is not taken into account. Therefore, it is not possible to accurately determine the three-dimensional coordinates of each joint of a person, and it is not possible to accurately determine the time change of the three-dimensional coordinates of each joint of a person.

[0033] When an analyzer analyzes a person's movements, referring to the determined three-dimensional coordinates of each joint of the person, the analyzer is likely to have the impression that the three-dimensional coordinates of each joint of the person are intuitively wrong. Specifically, the analyzer is left with the impression that the length of the person's arm is extended or contracted. In addition, specifically, the analyzer is left with the impression that the person's arm is moving at a speed that a person cannot produce.

[0034] Therefore, in this embodiment, an information processing method that can accurately identify the positions of the joints of a subject will be described.

[0035] exist Figure 1 In (1-1), the information processing device 100 obtains time series data of the skeleton information 101. The skeleton information 101 includes, for example, the positions of each of the multiple parts of the subject. The parts are, for example, the neck, the head, the right shoulder and the left shoulder, the right elbow and the left elbow, the right hand and the left hand, the right knee and the left knee, the right foot and the left foot, etc. The parts are specifically joints. Figure 1 In the example of , specifically, the parts are joint 1, joint 2, joint 3, etc. The position is, for example, a three-dimensional coordinate. The time series data, for example, includes the skeleton information 101 at each moment. Figure 1 In the example of , specifically, the time series data includes the skeleton information 101 at time T, the skeleton information 101 at time T-1, and the like.

[0036] (1-2) The information processing device 100 determines any part of the skeleton information 101 at the first moment in the acquired time series data that is in an abnormal state based on the feature quantity related to the skeleton information 101 in the acquired time series data. The feature quantity may be, for example, the position of each part of the subject shown in the skeleton information 101. The feature quantity may also be, for example, the deviation of the position of each part of the subject shown in the skeleton information 101 at different moments. The feature quantity may also be, for example, the distance between the positions of different parts of the subject shown in the skeleton information 101.

[0037] The information processing device 100, for example, has a first model for determining any one of the multiple parts of the subject whose position is in an abnormal state. The first model, for example, has a function of being able to calculate the probability that the position of each of the multiple parts of the subject is in an abnormal state based on the input of the feature quantity related to the skeleton information 101, and being able to determine whether the position of each part is in an abnormal state. The information processing device 100, for example, uses the first model to determine any one of the parts whose position is in an abnormal state for the skeleton information 101 at the first moment in the acquired time series data.

[0038] (1-3) The information processing device 100 determines a second model based on a feature quantity related to the skeleton information 101 in the acquired time series data. For example, the information processing device 100 determines a second model that limits the probability distribution of the position of any part in the skeleton information 101 at the first moment according to the magnitude of the probability that the part is in an abnormal state. For example, the probability is calculated using the first model based on the feature quantity related to the skeleton information 101 in the time series data. Figure 1 In the example, specifically, the information processing device 100 determines the second model that limits the probability distribution of the position of the joint 1 in the skeleton information 101 at the time T according to the magnitude of the probability that the joint 1 is in an abnormal state.

[0039] (1-4) The information processing device 100 generates a graph 110 including nodes 111 indicating the position of each part at each time, and first edges 112 connecting nodes indicating the positions of different parts connected on the biological body at each time. When generating the graph 110, the information processing device 100 establishes a correspondence between the determined second model and the node 111 indicating the position of any one of the parts determined. Figure 1 In the example of , specifically, the information processing device 100 associates the determined second model with the node 111 indicating the position of the joint 1 of the subject at time T, and generates the graph 110 .

[0040] (1-5) The information processing device 100 corrects the skeleton information 101 at the first moment in the time series data based on the generated graph 110. For example, the information processing device 100 corrects the position of the joint 1 of the subject included in the skeleton information 101 at the moment T in the time series data. Thus, the information processing device 100 can accurately determine the position of each joint of the subject. The information processing device 100 can accurately determine the time change of the position of each joint of the subject.

[0041] Here, the case where the information processing device 100 uses the first model to determine any part of the skeleton information 101 at the first moment in the time series data that is in an abnormal state is described, but the present invention is not limited to this. For example, there may be a case where the information processing device 100 does not use the first model, but determines any part of the skeleton information 101 at the first moment in the time series data that is in an abnormal state.

[0042] Here, the case where the information processing device 100 operates independently is described, but it is not limited to this. For example, there may be a case where a plurality of computers cooperate to realize the functions of the information processing device 100. Specifically, there may be a case where a computer that determines any part of the position that is in an abnormal state, a computer that generates the graph 110, and a computer that corrects the skeleton information 101 at the first moment in the time series data based on the graph 110 cooperate.

[0043] (An example of information processing system 200)

[0044] Next, use Figure 2 , for the application Figure 1 An example of an information processing system 200 of the information processing device 100 shown will be described.

[0045] Figure 2 2 is an explanatory diagram showing an example of the information processing system 200. Figure 2 In the embodiment, the information processing system 200 includes the information processing device 100 , one or more image capturing devices 201 , and one or more client devices 202 .

[0046] In the information processing system 200, the information processing device 100 and the image capturing device 201 are connected via a wired or wireless network 210. The network 210 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, etc. In addition, in the information processing system 200, the information processing device 100 and the client device 202 are connected via the wired or wireless network 210.

[0047] The information processing device 100 acquires a plurality of images of a subject person captured from different angles at each time from one or more image capturing devices 201. The information processing device 100 determines the distribution of the existence probability of each part of the subject person in the three-dimensional space based on the acquired plurality of images at each time, and determines the three-dimensional coordinates of each part of the subject person. The information processing device 100 calculates the degree of abnormality of each part of the subject person with respect to the three-dimensional coordinates based on the acquired plurality of images at each time, and determines any part of the subject person as an abnormal part in an abnormal state with respect to the three-dimensional coordinates.

[0048] The information processing device 100 determines a model for limiting the probability distribution of the position of the identified abnormal part based on the calculated abnormality degree. The information processing device 100 generates a graph including nodes representing the three-dimensional coordinates of each part of the subject at each time, and edges connecting the nodes, and establishes a correspondence relationship between the determined model and the nodes representing the three-dimensional coordinates of the abnormal part. The information processing device 100 corrects the three-dimensional coordinates of each part of the identified subject with reference to the graph.

[0049] The information processing device 100 outputs the corrected three-dimensional coordinates of each part of the subject. The output format is, for example, display on a display, printout on a printer, transmission to another computer, or storage in a storage area. The information processing device 100, for example, transmits the corrected three-dimensional coordinates of each part of the subject to the client device 202. The information processing device 100 is, for example, a server or a PC (Personal Computer).

[0050] The image capturing device 201 is a computer for capturing a subject. The image capturing device 201 includes a camera having a plurality of capturing elements, and captures the subject through the camera. The image capturing device 201 generates an image of the subject and transmits it to the information processing device 100. The image capturing device 201 is, for example, a smartphone. The image capturing device 201 may also be, for example, a fixed-point camera. The image capturing device 201 may also be, for example, a drone.

[0051] The client device 202 receives the three-dimensional coordinates of each part of the subject from the information processing device 100. The client device 202 outputs the received three-dimensional coordinates of each part of the subject in a manner that can be referenced by the user. The client device 202 displays the received three-dimensional coordinates of each part of the subject on a display, for example. The client device 202 is, for example, a PC, a tablet terminal, or a smartphone.

[0052] Here, the case where the information processing device 100 is a device different from the image capturing device 201 is described, but the present invention is not limited to this. For example, the information processing device 100 may have a function as the image capturing device 201 and may also operate as the image capturing device 201. Here, the case where the information processing device 100 is a device different from the client device 202 is described, but the present invention is not limited to this. For example, the information processing device 100 may have a function as the client device 202 and may also operate as the client device 202.

[0053] (Hardware Configuration Example of Information Processing Device 100)

[0054] Next, use Figure 3 , an example of the hardware configuration of the information processing device 100 is described.

[0055] Figure 3 is a block diagram showing an example of the hardware configuration of the information processing device 100. Figure 3 In the figure, the information processing device 100 has a CPU (Central Processing Unit) 301, a memory 302, a network I / F (Interface) 303, a recording medium I / F 304, and a recording medium 305. The information processing device 100 also has a display 306 and an input device 307. In addition, each component is connected via 300.

[0056] Here, the CPU 301 is responsible for the overall control of the information processing device 100. The memory 302 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and a flash ROM. Specifically, for example, the flash ROM or the ROM stores various programs, and the RAM is used as a work area of ​​the CPU 301. The program stored in the memory 302 is loaded into the CPU 301, so that the CPU 301 executes the coding process.

[0057] The network I / F 303 is connected to the network 210 through a communication line, and is connected to other computers via the network 210. The network I / F 303 manages the interface between the network 210 and the inside, and controls the input and output of data from other computers. The network I / F 303 is, for example, a modem or a LAN adapter.

[0058] The recording medium I / F 304 controls the reading / writing of data on the recording medium 305 according to the control of the CPU 301. The recording medium I / F 304 is, for example, a magnetic disk drive, an SSD (Solid State Drive), a USB (Universal Serial Bus) port, etc. The recording medium 305 is a non-volatile memory that stores data written by the control of the recording medium I / F 304. The recording medium 305 is, for example, a magnetic disk, a semiconductor memory, a USB memory, etc. The recording medium 305 may also be removable from the information processing device 100.

[0059] The display 306 displays data such as a cursor, an icon, a toolbox, a document, an image, or function information. The display 306 is, for example, a CRT (Cathode Ray Tube), a liquid crystal display, or an organic EL (Electroluminescence) display. The input device 307 has keys for inputting characters, numbers, or various instructions, and inputs data. The input device 307 is, for example, a keyboard or a mouse. The input device 307 may also be, for example, a touch panel input board or a numeric keypad.

[0060] The information processing device 100 may include a camera, etc., in addition to the above-mentioned components. In addition, the information processing device 100 may include a printer, a scanner, a microphone, or a speaker, etc., in addition to the above-mentioned components. In addition, the information processing device 100 may include a plurality of recording medium I / Fs 304 and recording media 305. In addition, the information processing device 100 may not include a display 306, or an input device 307, etc. In addition, the information processing device 100 may not include a recording medium I / F 304 and a recording medium 305.

[0061] (Hardware Configuration Example of Image Capturing Device 201)

[0062] Next, use Figure 4 , an example of the hardware configuration of the image capturing device 201 is described.

[0063] Figure 4 2 is a block diagram showing an example of the hardware configuration of the image pickup device 201. Figure 4 In the embodiment, the image pickup device 201 includes a CPU 401, a memory 402, a network I / F 403, a recording medium I / F 404, a recording medium 405, and a camera 406. In addition, each component is connected via a bus 400, respectively.

[0064] Here, the CPU 401 is responsible for overall control of the image capture device 201. The memory 402 includes, for example, a ROM, a RAM, and a flash ROM. Specifically, for example, the flash ROM or the ROM stores various programs, and the RAM is used as a work area of ​​the CPU 401. The program stored in the memory 402 is loaded into the CPU 401, so that the CPU 401 executes the encoding process.

[0065] The network I / F 403 is connected to the network 210 through a communication line, and is connected to other computers via the network 210. The network I / F 403 manages the interface between the network 210 and the inside, and controls the input and output of data from other computers. The network I / F 403 is, for example, a modem or a LAN adapter.

[0066] The recording medium I / F 404 controls the reading / writing of data on the recording medium 405 according to the control of the CPU 401. The recording medium I / F 404 is, for example, a disk drive, an SSD, a USB port, etc. The recording medium 405 is a non-volatile memory that stores data written by the control of the recording medium I / F 404. The recording medium 405 is, for example, a disk, a semiconductor memory, a USB memory, etc. The recording medium 405 may also be detachable from the image capturing device 201. The camera 406 has a plurality of capturing elements, and generates an image of an object captured by the plurality of capturing elements. The camera 406 is, for example, a camera for competition. The camera 406 is, for example, a surveillance camera.

[0067] In addition to the above-mentioned components, the image capture device 201 may also include a keyboard, a mouse, a display, a printer, a scanner, a microphone, a speaker, etc. In addition, the image capture device 201 may also include a plurality of recording medium I / Fs 404 and recording media 405. In addition, the image capture device 201 may not include the recording medium I / F 404 and the recording medium 405.

[0068] (Hardware Configuration Example of Client Device 202)

[0069] The hardware configuration example of the client device 202 is specifically similar to Figure 3 The hardware configuration example of the information processing device 100 shown is the same, so the description is omitted.

[0070] (Functional Configuration Example of Information Processing Device 100)

[0071] Next, use Figure 5 , an example of the functional configuration of the information processing device 100 is described.

[0072] Figure 51 is a block diagram showing an example of a functional configuration of the information processing device 100. The information processing device 100 includes a storage unit 500, an acquisition unit 501, an analysis unit 502, a learning unit 503, a determination unit 504, a decision unit 505, a generation unit 506, a correction unit 507, and an output unit 508.

[0073] For example, through Figure 3 The storage area such as the memory 302 and the recording medium 305 shown in the figure realizes the storage unit 500. In the following, the case where the storage unit 500 is included in the information processing device 100 is described, but the present invention is not limited to this. For example, there may be a case where the storage unit 500 is included in a device different from the information processing device 100, and the storage content of the storage unit 500 can be referenced from the information processing device 100.

[0074] The acquisition unit 501 to the output unit 508 function as an example of a control unit. Specifically, the acquisition unit 501 to the output unit 508, for example, cause the CPU 301 to execute the program stored in Figure 3 The functions are realized by a program in a storage area such as the memory 302 or the recording medium 305 shown in the figure, or through the network I / F 303. The processing results of each functional unit are stored in, for example, Figure 3 The storage area shown is the memory 302 or the recording medium 305.

[0075] The storage unit 500 stores various information referenced or updated in the processing of each functional unit. For example, the storage unit 500 stores multiple images of a specific person taken from different angles at each of multiple consecutive moments. The angle indicates the shooting position. For example, the image is acquired by the acquisition unit 501.

[0076] The storage unit 500 stores, for example, time series data of skeleton information. The time series data includes skeleton information at each of a plurality of consecutive moments. The skeleton information includes the position of each of a plurality of parts of a specific person. The parts are, for example, joints. The parts are, for example, the neck, the head, the right shoulder and the left shoulder, the right elbow and the left elbow, the right hand and the left hand, the right knee and the left knee, the right foot and the left foot, etc. The position is, for example, a three-dimensional coordinate. For example, the time series data is acquired by the acquisition unit 501. For example, the time series data can also be generated by the analysis unit 502.

[0077] The acquisition unit 501 acquires various information used for processing of each functional unit. The acquisition unit 501 stores the acquired various information in the storage unit 500, or outputs it to each functional unit. In addition, the acquisition unit 501 may also output the various information stored in the storage unit 500 to each functional unit. The acquisition unit 501 acquires various information based on, for example, the operation input of the user. The acquisition unit 501 may also receive various information from, for example, a device different from the information processing device 100.

[0078] The acquisition unit 501 acquires, for example, time series data of the skeleton information of the subject. The skeleton information of the subject includes, for example, the positions of each of the multiple parts of the subject. Specifically, the acquisition unit 501 acquires the time series data of the skeleton information of the subject by accepting input of the time series data of the skeleton information of the subject based on the operation input of the user. Specifically, the acquisition unit 501 may also acquire the time series data of the skeleton information of the subject by receiving it from another computer.

[0079] The acquisition unit 501 may also acquire, for example, time series data of the skeleton information of a past subject. The subject may be the same as the object, for example. The skeleton information of the subject may include, for example, the positions of each of a plurality of parts of the subject. Specifically, the acquisition unit 501 acquires the time series data of the skeleton information of the subject by accepting input of the time series data of the skeleton information of the subject based on the operation input of the user. Specifically, the acquisition unit 501 may also acquire the time series data of the skeleton information of the subject by receiving it from another computer.

[0080] The acquisition unit 501 acquires, for example, a plurality of images of the subject taken from different angles at each of a plurality of consecutive moments. The acquisition unit 501 acquires the plurality of images without acquiring the time series data of the subject's skeleton information, which is generated by the analysis unit 502. Thus, the acquisition unit 501 enables the analysis unit 502 to generate the time series data of the subject's skeleton information.

[0081] The acquisition unit 501 may, for example, acquire a plurality of images of the subject taken from different angles at each of a plurality of consecutive moments. The acquisition unit 501 acquires the plurality of images without acquiring the time series data of the subject's skeletal information generated by the analysis unit 502. Thus, the acquisition unit 501 enables the analysis unit 502 to generate the time series data of the subject's skeletal information.

[0082] The acquisition unit 501 may also receive a start trigger for starting the processing of any functional unit. The start trigger may be, for example, a predetermined operation input by a user. The start trigger may also be, for example, receiving predetermined information from another computer. The start trigger may also be, for example, outputting predetermined information from any functional unit.

[0083] The acquisition unit 501 may, for example, receive the acquisition of a plurality of images as a trigger for starting the processing of the analysis unit 502. The acquisition unit 501 may, for example, receive the acquisition of time series data of the skeleton information of the subject as a trigger for starting the processing of the learning unit 503. The acquisition unit 501 may, for example, receive the acquisition of time series data of the skeleton information of the subject as a trigger for starting the processing of the determination unit 504, the determination unit 505, the generation unit 506, and the correction unit 507.

[0084] The analysis unit 502 generates time series data of the skeleton information of a specified person. The analysis unit 502 generates time series data of the skeleton information of the subject, for example. Specifically, the analysis unit 502 estimates the position of each part of the subject at each time based on a plurality of images of the subject taken from different angles at each time, and generates the skeleton information of the subject including the estimated position. Specifically, the analysis unit 502 generates time series data of the skeleton information of the subject based on the generated skeleton information of the subject. Thus, the analysis unit 502 can temporarily determine the position of each part of the subject at each time, and can obtain the corrected object.

[0085] The analysis unit 502 may also generate time series data of the subject's skeletal information, for example. Specifically, the analysis unit 502 generates the subject's skeletal information at each time based on a plurality of images of the subject taken from different angles at each time, and generates time series data of the subject's skeletal information. The analysis unit 502 may also add noise to the generated time series data of the subject's skeletal information. The analysis unit 502 sets the subject's skeletal information as teacher information for generating a learning model. Thus, the analysis unit 502 can obtain teacher information for generating a learning model.

[0086] The learning unit 503 learns the learning model based on the teacher information including the positions of each of the multiple parts of the subject. The learning model has a function of determining any one of the multiple parts of the prescribed person that is in an abnormal state in terms of position based on the feature quantity related to the skeletal information in the time series data of the skeletal information of the prescribed person. For example, the learning model has a function of determining whether each part of the prescribed person is in an abnormal state in terms of position.

[0087] Specifically, the learning model has a function of calculating an index value representing the size of the probability that each part of the specified person is in an abnormal state with respect to the position. More specifically, the learning model outputs an index value representing the size of the probability that the part is in an abnormal state with respect to the position for each part of the specified person in response to the input of the feature quantity related to the skeletal information. The learning model is specifically a neural network. Thus, the learning unit 503 can determine any one of the multiple parts of the subject that is in an abnormal state with respect to the position.

[0088] The determination unit 504 determines any one part of the skeleton information at the first moment in the acquired time series data of the skeleton information of the subject, which is in an abnormal state, based on the feature amount related to the skeleton information in the acquired time series data of the skeleton information of the subject. The determination unit 504 determines any one part of the skeleton information at the first moment, which is in an abnormal state, based on the feature amount related to the skeleton information in the acquired time series data of the skeleton information of the subject, using, for example, a learned learning model.

[0089] Specifically, the determination unit 504 inputs the feature value related to the skeletal information in the time series data of the skeletal information of the subject into the learning model, and calculates the index value representing the probability of each part of the subject being in an abnormal state for the skeletal information at the first moment. Specifically, the determination unit 504 determines any part of the skeletal information at the first moment that is in an abnormal state in terms of position based on the calculated index value. More specifically, the determination unit 504 determines any part of the subject's multiple parts whose calculated index value is above the threshold as a part that is in an abnormal state in terms of position. Thus, the determination unit 504 can obtain a pointer to the position of each part of the multiple parts of the subject to be corrected. The determination unit 504 can identify which part of the subject's position is preferably corrected.

[0090] The determination unit 504 may also, for example, refer to the prescribed rules and determine any one part of the skeleton information at the first moment that is in an abnormal state in terms of position based on the feature quantity related to the skeleton information in the time series data of the skeleton information of the acquired subject. The prescribed rules may, for example, include rules that can determine any one part of the subject's multiple parts that is in an abnormal state in terms of position based on the feature quantity related to the skeleton information in the time series data of the skeleton information of the subject. Specifically, the prescribed rules may also include rules that can calculate an index value that represents the probability that each part of the prescribed subject is in an abnormal state in terms of position.

[0091] Specifically, the determination unit 504 refers to the prescribed rules and calculates an index value representing the probability of each part of the subject being in an abnormal state for the skeleton information at the first moment based on the feature value related to the skeleton information in the time series data of the skeleton information of the subject. Specifically, the determination unit 504 determines any part of the skeleton information at the first moment that is in an abnormal state in terms of position based on the calculated index value. More specifically, the determination unit 504 determines any part of the subject's multiple parts whose calculated index value is above a threshold as a part that is in an abnormal state in terms of position. Thus, the determination unit 504 can obtain a pointer to the position of each part of the multiple parts of the subject to be corrected. The determination unit 504 can identify which part of the subject's position is preferably corrected.

[0092] The decision unit 505 determines a distribution model for limiting the probability distribution of the position of the part determined in the skeletal information at the first moment based on the feature quantity related to the skeletal information in the time series data of the acquired skeletal information of the subject. The distribution model is, for example, a model for limiting the position of any part according to an index value indicating the magnitude of the probability of any part being in an abnormal state. Thus, the decision unit 505 can obtain a pointer for correcting the position of the part determined by the determination unit 504.

[0093] The generating unit 506 generates a graph including nodes indicating the positions of various parts at various moments and first edges connecting nodes indicating the positions of different parts connected on the biological body at various moments, and establishes a correspondence relationship between the determined distribution model and the nodes indicating the positions of the determined parts. Thus, the generating unit 506 can correct the skeleton information at the first moment in the time series data of the skeleton information of the subject.

[0094] The generator 506 may also generate a graph including a node, a first edge, and a second edge connecting nodes representing the position of any part at different times, and establishing a correspondence between the determined distribution model and the node representing the position of the determined part. Thus, the generator 506 can correct the skeleton information at the first moment in the time series data of the skeleton information of the subject.

[0095] The correction unit 507 corrects the skeleton information at the first moment in the time series data of the skeleton information of the subject based on the generated graph. The correction unit 507 corrects the skeleton information at the first moment in the time series data of the skeleton information of the subject, for example, by optimizing the generated graph. Thus, the correction unit 507 can determine the position of each part of the subject with good accuracy, taking into account the probability of each part of the subject being in an abnormal state.

[0096] The output unit 508 outputs the processing result of at least one of the functional units. The output format is, for example, display on a display, print output on a printer, transmission to an external device via the network I / F 303, or storage in a storage area such as the memory 302 or the recording medium 305. Thus, the output unit 508 can notify the user of the processing result of at least one of the functional units, and the convenience of the information processing device 100 can be improved.

[0097] The output unit 508, for example, outputs the skeleton information at the first moment corrected by the correction unit 507. Specifically, the output unit 508 sends the skeleton information at the first moment corrected by the correction unit 507 to the client device 202. Specifically, the output unit 508 displays the skeleton information at the first moment corrected by the correction unit 507 on the display. Thus, the output unit 508 can make the positions of various parts of the subject available.

[0098] (Operation Example of Information Processing Device 100)

[0099] Next, use Figure 6 to Figure 12 , an example of the operation of the information processing device 100 is described. First, for example, using Figure 6 , the operation flow of the information processing device 100 is described.

[0100] Figure 6 1 is an explanatory diagram showing the flow of operations of the information processing device 100. Figure 6 In the embodiment, the information processing device 100 acquires a plurality of multi-viewpoint images 600 of a subject person captured at different angles at different times. The information processing device 100 detects the area in which the subject person is reflected in each of the plurality of multi-viewpoint images 600 by performing a person detection process on each of the plurality of multi-viewpoint images 600.

[0101] The information processing device 100 performs 2D (Dimension) pose estimation processing on each multi-view image 600 at a time. The information processing device 100 generates a 2D heat map 601 representing the distribution of the existence probability of each joint of the subject in each multi-view image 600 by performing 2D pose estimation processing on each multi-view image 600 at a time. The 2D heat map 601 includes, for example, joint likelihood representing the existence probability of any joint of the subject at each point in the 2D space corresponding to the multi-view image 600.

[0102] The information processing device 100 determines the 2D coordinates of the joints of the subject in each multi-view image 600 at each time based on the 2D heat map 601 indicating the distribution of the existence probability of each joint of the subject in each multi-view image 600. The variance of the joint likelihood indicating the existence probability of the joint of the subject in the 2D heat map 601 can be processed as an indicator value indicating the accuracy of the determined 2D coordinates.

[0103] The information processing device 100 acquires configuration information indicating the angle of each multi-view image 600 at a time. The information processing device 100 determines the 3D coordinates of each joint of the subject in the 3D space by performing 3D pose estimation processing based on the configuration information and the 2D coordinates of each joint of the subject in each multi-view image 600 at a time. The information processing device 100 generates a 3D skeleton inference result 602 including the 3D coordinates of each joint of the subject determined at a time, and generates time series data of the 3D skeleton inference result 602.

[0104] The information processing device 100 corrects the 3D skeleton inference result 602 by performing correction processing on the time series data of the 3D skeleton inference result 602. The information processing device 100 outputs the time series data of the corrected 3D skeleton inference result 603 in a usable manner. The information processing device 100 outputs the time series data of the corrected 3D skeleton inference result 603 in a manner that can be referenced by a user, for example.

[0105] The user implements the specified analytical processing based on the time series data of the corrected 3D skeleton inference result 603. Specifically, consider the case where the subject is a participant of a sports meeting. In this case, the analytical processing is, for example, the scoring of the participants in the competition of the sports meeting. The user implements the analytical processing of scoring the participants based on the time series data of the corrected 3D skeleton inference result 603.

[0106] Specifically, consider a case where the subject is a patient at a medical institution undergoing rehabilitation training, or a patient at a medical institution undergoing diagnosis of motor abilities such as walking ability, etc. In this case, the analytical processing is, for example, determination of the effect of rehabilitation training, or diagnosis of motor abilities or health status, etc. The user determines the effect of rehabilitation training of the patient at the medical institution, or diagnoses the motor abilities or health status of the patient at the medical institution, based on the time series data of the corrected 3D skeleton inference result 603.

[0107] The information processing device 100 may also perform the above-mentioned analysis process based on the time series data of the corrected 3D skeleton inference result 603. The information processing device 100 outputs the result of the analysis process in a manner that can be referenced by the user. The information processing device 100 may also output the time series data of the corrected 3D skeleton inference result 603 to the analysis unit 502 that performs the above-mentioned analysis process. For example, the analysis unit 502 is provided by another computer other than the information processing device 100. Thus, the information processing device 100 can perform the analysis process with good accuracy.

[0108] Next, use Figure 7 to Figure 9, a specific example of the correction process is described. Specifically, first, use Figure 7 as well as Figure 8 , a specific example of an abnormal joint determined to be in an abnormal state with respect to 3D coordinates among a plurality of joints of a subject by the information processing apparatus 100 is described.

[0109] Figure 7 as well as Figure 8 FIG. 2 is an explanatory diagram showing a specific example of identifying an abnormal joint. Figure 7 In FIG. 1 , the information processing device 100 acquires time series data of a plurality of raw data 700. The raw data 700 represents the skeleton information of the subject. The raw data 700 represents the 3D coordinates of each of a plurality of joints of the subject. The 3D coordinates of the joints are, for example, ● in the figure.

[0110] The information processing device 100 generates processed data 701 by adding noise to the original data 700. The information processing device 100 generates the processed data 701, for example, by changing the 3D coordinates of at least one of the multiple joints of the subject shown in the original data 700 to 3D coordinates judged to be in an abnormal state. The abnormal state corresponds to a state in which the 3D coordinates of the joint are incorrectly estimated. Specifically, the abnormal state is jitter, inversion, swap, or miss. Thus, the information processing device 100 can obtain time series data of the processed data 701.

[0111] The information processing device 100 uses the time series data of the processed data 701 to learn the abnormality judgment DNN (Deep Neural Network) 710. The abnormality judgment DNN 710 has, for example, a function of outputting the abnormality probability of each joint of the subject in at least one 3D skeleton inference result 602 based on the input of the feature quantity of the 3D skeleton inference result 602 in the time series data of the 3D skeleton inference result 602. The abnormality probability represents the magnitude of the probability that the 3D coordinates of the joint of the subject are in an abnormal state in position. The abnormality judgment DNN 710 may also have, for example, a function of outputting the abnormality probability of each joint of the subject in the entire time series data based on the input of the feature quantity of the 3D skeleton inference result 602 in the time series data of the 3D skeleton inference result 602. Next, move to Figure 8 Description.

[0112] exist Figure 8In the example, the information processing device 100 inputs the feature value of the 3D skeleton inference result 602 in the time series data of the 3D skeleton inference result 602 to the abnormality determination DNN 710. The information processing device 100 obtains the abnormality probability of each joint of the subject in the entire time series data of the 3D skeleton inference result 602 output by the abnormality determination DNN 710 based on the input. The information processing device 100 determines the abnormal joint based on the acquired abnormality probability of each joint of the subject. For example, the information processing device 100 determines any joint of the subject whose abnormality probability is above a threshold value as an abnormal joint.

[0113] Here, the case where the information processing device 100 uses the abnormality judgment DNN 710 to determine the abnormal joint is described, but it is not limited to this. For example, there may be a case where the information processing device 100 determines the abnormal joint based on the rule base. Specifically, the information processing device 100 may also store a rule for calculating the abnormal probability of the joint based on the difference between the feature quantity related to each joint and the threshold in the 3D skeleton inference result 602. Specifically, consider that the information processing device 100 refers to the stored rules to calculate the abnormal probability of each joint, and determines any joint whose calculated abnormal probability is above the threshold as an abnormal joint. Next, use Fig. 9 , a specific example of the information processing device 100 generating the factor graph (Factor Graph) 900 is described.

[0114] Fig. 9 900 is an explanatory diagram showing a specific example of generating a factor graph 900. Fig. 9 In , the information processing device 100 generates a factor graph 900. The factor graph 900 includes, for example, nodes representing the positions of the joints of the subject at each time.

[0115] Specifically, the factor graph 900 includes nodes representing the positions of the subject's head, upper cervical spine, lower cervical spine, thoracic spine, lumbar spine, left and right hip joints, left and right knee joints, left and right foot joints, left and right feet, left and right shoulder joints, left and right elbow joints, left and right wrists, and left and right hands at each moment.

[0116] The information processing device 100 may also connect the nodes representing the positions of the specified joints at different times in the factor graph 900 through the second edge to generate the factor graph 901. The second edge may also establish a corresponding relationship with the Pairwise Term representing the time series restriction specified according to the type of joint. For example, the time series restriction is defined according to the probability distribution corresponding to the equal position motion, the equal speed linear motion, or the equal acceleration motion.

[0117] The pairwise term representing the time series constraint corresponding to the uniform linear motion is, for example, gt (x j,t-1 , x j,t )~N(||x j,t-1 , x j,t ||┃v j ^Δt,Σ vj In the following description, for convenience, a character with ^ added to the top may be referred to as a "character ^". j,t-1 is the estimated position of the joint at time t-1. j,t is the estimated position of the joint at time t. v j ^ is the average velocity of the joint. Δt is the unit time width. Σ vj ^ is the velocity variance of the joint. A corresponding relationship is established with the distribution model of the Pairwise Term representing the time series constraint corresponding to the iso-position motion. For example, the Pairwise Term is g t (x j,t-1 , x j,t )~N(||x j,t-1 , x j,t ||┃0,Σ xj ^). Σ xj ^ is the position variance of the joint.

[0118] The information processing device 100 may also connect leaf nodes in the factor graph 900 that represent the position of the same joint at different times and are connected to only one edge through a third edge to generate the factor graph 901. For example, a leaf node is a node that is connected to only one first edge and not to a second edge.

[0119] The information processing device 100, for example, determines whether a leaf node connected to only one first edge and not to a second edge in the factor graph 900 is a node representing the position of a specific abnormal joint. If the leaf node is a node representing the position of a determined abnormal joint, the information processing device 100 connects the leaf nodes at different times to each other through the third edge 910.

[0120] The third edge 910 may also be associated with a Pairwise Term representing a predetermined time series restriction. Thus, the information processing device 100 can accurately correct the position of the abnormal joint. Fig.10 , a specific example of how the information processing device 100 uses the generated factor graph 901 to correct the 3D skeleton inference result 602 is described.

[0121] Fig.10 602 is an explanatory diagram showing a specific example of correcting the 3D skeleton inference result. Fig.10 In , the information processing device 100 uses the generated factor graph 901 to correct the 3D skeleton inference result 602. Fig.10 In the example of , factor graph 901 includes node group 1010 corresponding to time t-1, node group 1020 corresponding to time t, and the like. Node group 1010 includes nodes 1011 to 1013, and the like. Node group 1020 includes nodes 1021 to 1023, and the like.

[0122] For example, the node 1011 is connected to the node 1012 via the first edge 1031. For example, the node 1012 is connected to the node 1013 via the first edge 1032. For example, the node 1021 is connected to the node 1022 via the first edge 1041. For example, the node 1022 is connected to the node 1023 via the first edge 1042. For example, the first edge 1042 connecting the node 1022 to the node 1023 may also establish a correspondence relationship with the Pairwise Term representing the limit of the length of the bone.

[0123] The information processing device 100 may further connect the node 1012 to the node 1022 through the second edge 1051. The second edge 1051 may also establish a correspondence relationship with a PairwiseTerm representing a time series restriction specified according to the type of joint. For example, the node 1011 and the node 1021 are connected through the third edge 1061. The third edge 1061 may also establish a correspondence relationship with a PairwiseTerm representing a specified time series restriction.

[0124] The information processing device 100 establishes a correspondence relationship between at least one node representing the position of a joint in the factor graph 901 and a Unary Term representing a restriction of an abnormal joint that acts to restrict the position of the joint according to the abnormal probability of the joint. Fig.10 In the example of , the information processing device 100 establishes a correspondence relationship between the node 1021 representing the position of the joint 1 in the node group 1020 and the Unary Term including the abnormality probability of the joint 1. The Unary Term is, for example, f(x j )~N(x j |x j ^,Σ 3D j ^)·p(x j ). p(x j ) is the abnormal probability.

[0125] The information processing device 100 corrects the position of each joint at each time based on the Unary Term and the Pairwise Term in the factor graph 901. The information processing device 100 corrects the position of each joint at each time by optimizing the factor graph 901, for example.

[0126] The information processing device 100 can accurately correct the 3D skeleton inference result 602. The information processing device 100 can accurately determine the position of each joint at each time. For example, even when the subject performs a relatively high-speed or relatively complex action such as gymnastics, the information processing device 100 can determine the position of each joint of the subject at each time with relatively high accuracy.

[0127] Here, referring to the following reference 3, consider using f(x j )~N(x j |x j ^,Σ 3D j ^) as a comparative example of Unary Term in the factor graph. j ^ is the weighted sum of the joint likelihood of the 3D heat map that integrates the joint likelihoods of multiple 2D heat maps. Σ 3D j ^ is the variance of the joint likelihood of the 3D heatmap that integrates the joint likelihoods of multiple 2D heatmaps.

[0128] Therefore, the comparative example uses the Unary Term to correct the 3D coordinates of any joint with a lower joint likelihood by using the 3D coordinates of other joints with a higher likelihood as a reference. However, in the comparative example, there are situations where it is difficult to determine the 3D coordinates of each joint of the subject with good accuracy, and it is difficult to correct the time change of the 3D coordinates of each joint of the person with good accuracy. For example, it is not necessarily preferred to correct the 3D coordinates of any joint with a lower joint likelihood, nor is it necessarily preferred to use the 3D coordinates of any joint with a higher joint likelihood as a reference. Therefore, in the comparative example, there are situations where the 3D coordinates of each joint of the subject cannot be properly corrected.

[0129] Reference 3: Bultmann, Simon, and Sven Behnke. "Real-time multi-view 3D human pose estimation using semantic feedback to smart edge sensors." arXivpreprint arXiv: 2106.14729 (2021).

[0130] In contrast, the information processing device 100 can use f(x j )~N(x j |x j ^,Σ 3D j ^)·p(x j), as the Unary Term in the factor graph 901. Therefore, the information processing device 100 can easily correct the 3D coordinates of the joint that is preferably corrected, and can easily fix the 3D coordinates of the joint that is preferably used as the reference. Therefore, the information processing device 100 can appropriately correct the 3D coordinates of each joint of the subject. Next, using Fig.11 as well as Fig.12 , a specific example of the data processing flow in the action example is explained.

[0131] Fig.11 as well as Fig.12 1 is an explanatory diagram showing a specific example of the flow of data processing in the action example. Fig.11 As shown, the information processing device 100 acquires a plurality of camera images 1101 at a time. The information processing device 100 stores a 2D skeleton inference model 1110. The information processing device 100 stores, for example, weight parameters of a neural network that defines the 2D skeleton inference model 1110.

[0132] The information processing device 100 generates a 2D skeleton inference result 1102 by performing 2D skeleton inference processing on each of the multiple camera images 1101 according to the time, with reference to the 2D skeleton inference model 1110. The 2D skeleton inference result 1102 includes, for example, a 2D coordinate (x[pixel], y[pixel]) representing the position of a joint, and a likelihood representing the accuracy of the position of the joint.

[0133] The information processing device 100 stores the 3D skeleton inference model 1120. The information processing device 100 stores, for example, weight parameters of a neural network defined as the 3D skeleton inference model 1120.

[0134] The information processing device 100 performs 3D skeleton inference processing on multiple 2D skeleton inference results 1102 by referring to the 3D skeleton inference model 1120 according to the time, and generates a 3D skeleton inference result 1103. The 3D skeleton inference result 1103 includes, for example, 3D coordinates (x [mm], y [mm], z [mm]) representing the position of the joint. The information processing device 100 generates time series data 1104 that summarizes the 3D skeleton inference results 1103 at each time. Next, move to Fig.12 Description.

[0135] like Fig.12 As shown, the information processing device 100 stores an abnormality detection model 1210. The information processing device 100 stores, for example, weight parameters of a neural network defined as the abnormality detection model 1210.

[0136] The information processing device 100 performs an abnormality detection process on the time series data 1104 by referring to the abnormality detection model 1210, calculates the abnormality probability of each joint, and generates a skeletal abnormality detection result 1201 including the calculated abnormality probability of each joint.

[0137] The information processing device 100 stores a bone length model 1220 and each time series motion model 1230. The bone length model 1220 includes parameters defining the pairwise terms that represent the constraints of the bone length. The parameters are, for example, the mean and variance of the bone length. The time series motion model 1230 includes the pairwise terms that represent the time series constraints of each joint.

[0138] The information processing device 100 generates a factor graph by performing factor graph generation processing with reference to the bone abnormality inspection result 1201, the bone length model 1220, and the time series motion model 1230. The information processing device 100 corrects the position of each joint by performing optimization processing on the generated factor graph. The information processing device 100 generates a corrected 3D bone inference result 1202 including the corrected position of each joint.

[0139] (Overall processing order)

[0140] Next, use Fig.13 , an example of the overall processing sequence performed by the information processing device 100 is described. For example, by Figure 3 The CPU 301 shown, a storage area such as a memory 302 or a recording medium 305 , and a network I / F 303 implement overall processing.

[0141] Fig.13 is a flowchart showing an example of the overall processing sequence. Fig.13 In step S1301, the information processing device 100 obtains time series data of the three-dimensional skeleton inference result of the subject.

[0142] Next, the information processing device 100 calculates the abnormality of each part of the subject based on the acquired time series data of the three-dimensional skeleton inference result of the subject, and determines the abnormal part among the multiple parts of the subject based on the calculated abnormality (step S1302). Then, the information processing device 100 calculates the likelihood of each part of the subject based on the abnormality of each part of the subject (step S1303).

[0143] Next, the information processing device 100 sets Unary Term = Likelihood for the specific part of the subject, and generates a factor graph in which Pairwise Term is set for the specific part of the subject along the time axis (step S1304). Then, the information processing device 100 corrects the time series data of the three-dimensional skeleton inference result of the subject by optimizing the factor graph (step S1305).

[0144] Next, the information processing device 100 outputs the time series data of the corrected three-dimensional skeleton inference result of the subject (step S1306). Then, the information processing device 100 ends the overall processing. Thus, the information processing device 100 can correct the three-dimensional skeleton inference result of the subject with good accuracy. Therefore, the information processing device 100 can improve the usefulness of the three-dimensional skeleton inference result of the subject. The information processing device 100, for example, can improve the accuracy of the analytical processing based on the three-dimensional skeleton inference result of the subject.

[0145] As described above, according to the information processing device 100, it is possible to acquire time series data including skeleton information of the positions of each of the plurality of parts of the subject. According to the information processing device 100, it is possible to determine, based on the feature quantity related to the skeleton information in the acquired time series data, any one part of the skeleton information at the first moment in the acquired time series data whose position is in an abnormal state. According to the information processing device 100, it is possible to determine a model of probability distribution of the position of any one part determined in the skeleton information at the first moment, which is limited according to the magnitude of the probability that the determined any one part is in an abnormal state. According to the information processing device 100, it is possible to generate a graph including nodes representing the positions of each part at each moment. According to the information processing device 100, it is possible to add a first edge connecting nodes representing the positions of different parts connected on the biological body at each moment to each other in the graph. According to the information processing device 100, it is possible to establish a correspondence relationship between the determined model and the node representing the position of any one part in the graph. According to the information processing device 100, it is possible to correct the skeleton information at the first moment in the time series data based on the generated graph. Thereby, the information processing device 100 can correct the skeleton information at the first time with high accuracy.

[0146] According to the information processing device 100, a model capable of determining any one of the multiple parts of the subject that is in an abnormal state of position can be learned based on teacher information including the positions of each of the multiple parts of the subject. According to the information processing device 100, it is possible to determine any one of the multiple parts of the subject that is in an abnormal state of position based on the feature quantity related to the skeleton information in the acquired time series data for the skeleton information at the first moment in the acquired time series data using the learned model. Thus, the information processing device 100 can determine any one of the multiple parts that is in an abnormal state of position with good accuracy.

[0147] The information processing device 100 can store a rule that can determine any one of the multiple parts of the subject that is in an abnormal state of position based on the feature quantity related to the skeleton information in the time series data. The information processing device 100 can refer to the rule and determine any one of the parts that is in an abnormal state of position based on the feature quantity related to the skeleton information in the acquired time series data at the first moment in the acquired time series data. Thus, the information processing device 100 can determine any one of the parts that is in an abnormal state of position with good accuracy.

[0148] According to information processing device 100, a graph including nodes, first edges, and second edges connecting nodes indicating positions of arbitrary parts at different times can be generated. Thus, information processing device 100 can easily and accurately correct the skeleton information at the first time.

[0149] In addition, the information processing method described in this embodiment can be realized by executing a pre-prepared program by a computer such as a PC or a workstation. The information processing program described in this embodiment is recorded on a recording medium that can be read by a computer, and is executed by reading it from the recording medium by a computer. The recording medium is a hard disk, a floppy disk, a CD (Compact Disc: optical disk)-ROM, an MO (Magneto Optical Disc: magnetic optical disk), a DVD (Digital Versatile Disc: digital versatile disk), etc. In addition, the information processing program described in this embodiment can also be distributed via a network such as the Internet.

[0150] Description of Reference Numerals

[0151] 100…information processing device, 101…skeleton information, 110…graph, 111, 1011 to 1013, 1021 to 1023…nodes, 112, 1031, 1032, 1041, 1042…first edge, 200…information processing system, 201…image capturing device, 202…client device, 210…network, 300, 400…bus, 301, 401…CPU, 302, 402…memory, 303, 403…network I / F, 304, 404…recording medium I / F, 305, 405…recording medium, 306…display, 307…input device, 406…camera, 500…storage unit, 501…acquisition unit, 502…analysis unit, 503…learning unit, 504…determination unit, 505… Determination unit, 506…Generation unit, 507…Correction unit, 508…Output unit, 600…Multi-view image, 601…2D heat map, 602, 603, 1103, 1202…3D skeleton inference result, 700…Original data, 701…Processed data, 710…Abnormality judgment DNN, 900, 901…Factor graph, 910, 1061…Third edge, 1010, 1020…Node group, 1051…Second edge, 1101…Camera image, 1102…2D skeleton inference result, 1104…Time series data, 1110…2D skeleton inference model, 1120…3D skeleton inference model, 1201…Bone abnormality inspection result, 1210…Abnormality inspection model, 1220…Bone length model, 1230…Time series motion model.

Claims

1. An information processing program, characterized in that: Causes the computer to perform the following processing: acquiring time series data including skeletal information of positions of each of a plurality of parts of a subject; Based on the feature quantity related to the skeleton information in the acquired time series data, determining any part of the skeleton information at the first moment in the acquired time series data that is in an abnormal state; Based on the feature quantity related to the skeleton information in the acquired time series data, determine a model for limiting the probability distribution of the position of the above-mentioned any one of the parts determined in the skeleton information at the above-mentioned first moment according to the magnitude of the probability that the above-mentioned any one of the parts is in an abnormal state; generating a graph including nodes representing the positions of the above-mentioned respective parts at each time and first edges connecting nodes representing the positions of different parts connected on the biological body at each of the above-mentioned times, and establishing a correspondence relationship between the determined model and the nodes representing the position of any one of the above-mentioned parts; and Based on the generated graph, the skeleton information at the first moment in the time series data is corrected.

2. The information processing program according to claim 1, characterized in that Based on teacher information including the positions of each of the multiple parts of the subject, a model is learned that can determine any one of the multiple parts of the subject that is in an abnormal state in terms of position based on feature quantities related to the skeletal information in the time series data. The determination process uses the learned model to determine any part of the skeleton information at the first moment in the acquired time series data in an abnormal state based on the feature quantity related to the skeleton information in the acquired time series data.

3. The information processing program according to claim 1, wherein: The processing reference determined above can determine a rule that any one of the multiple parts of the subject is in an abnormal state in position based on a feature value related to the skeletal information in the time series data, and determines any one of the multiple parts of the subject that is in an abnormal state in position based on the skeletal information at the first moment in the time series data acquired.

4. The information processing program according to any one of claims 1 to 3, characterized in that: The generating process generates a graph including the nodes, the first edges, and second edges connecting nodes indicating positions of any one of the parts at different times.

5. An information processing method, characterized in that: The computer performs the following processing: acquiring time series data including skeletal information of positions of each of a plurality of parts of a subject; Based on the feature quantity related to the skeleton information in the acquired time series data, determining any part of the skeleton information at the first moment in the acquired time series data that is in an abnormal state; Based on the feature quantity related to the skeleton information in the acquired time series data, determine a model for limiting the probability distribution of the position of the above-mentioned any one of the parts determined in the skeleton information at the above-mentioned first moment according to the magnitude of the probability that the above-mentioned any one of the parts is in an abnormal state; generating a graph including nodes representing the positions of the above-mentioned respective parts at each time, and first edges connecting nodes representing the positions of different parts connected on the biological body at each of the above-mentioned times, and establishing a correspondence relationship between the determined model and the nodes representing the position of any one of the above-mentioned parts; and Based on the generated graph, the skeleton information at the first moment in the time series data is corrected.

6. An information processing device, characterized in that: A control unit is provided, which performs the following processing: acquiring time series data including skeletal information of positions of each of a plurality of parts of a subject; Based on the feature quantity related to the skeleton information in the acquired time series data, determining any part of the skeleton information at the first moment in the acquired time series data that is in an abnormal state; Based on the feature quantity related to the skeleton information in the acquired time series data, determine a model for limiting the probability distribution of the position of the above-mentioned any one of the parts determined in the skeleton information at the above-mentioned first moment according to the magnitude of the probability that the above-mentioned any one of the parts is in an abnormal state; generating a graph including nodes representing the positions of the above-mentioned respective parts at each time, and first edges connecting nodes representing the positions of different parts connected on the biological body at each of the above-mentioned times, and establishing a correspondence relationship between the determined model and the nodes representing the position of any one of the above-mentioned parts; and Based on the generated graph, the skeleton information at the first moment in the time series data is corrected.

Images