Information processing method, information processing program, and information processing device

Abstract

This information processing device calculates an abnormal feature on the basis of a plurality of pieces of bone information in which a plurality of joints included in a human body and coordinates relating to the plurality of joints are set. On the basis of a first distribution of normal skeleton information relating to the abnormal feature and a second distribution of abnormal skeleton information relating to the abnormal feature, the information processing device calculates, as a first threshold value, a boundary value at which the degree of separation between the first distribution and the second distribution is maximum. The information processing apparatus sets a correction range of time-series bone information based on a first threshold value.

Description

Technical Field

[0001] The present invention relates to an information processing method and the like. Background Art

[0002] In relation to various sports, there is a prior art in which multiple cameras are used to capture images of a person and perform 3D skeleton recognition.

[0003] Figure 33 This is a diagram for explaining the prior art related to 3D skeleton recognition. In Figure 33 In the example shown, cameras 30a and 30b capture an image of person U1. The images captured by cameras 30a and 30b are respectively set as images 31a and 31b.

[0004] Images 31a and 31b are respectively input into learning models 32a and 32b, and learning models 32a and 32b respectively output 2D key points 33a and 33b. Learning models 32a and 32b are trained deep learning models and the like. 2D key points 33a and 33b are two-dimensional skeleton information and the like.

[0005] In the prior art, 3D key points 34 are generated by integrating 2D key points 33a and 33B. 3D key points 34 are three-dimensional skeleton information and the like. For example, three-dimensional coordinates of each joint of a human body model are set in 3D key points 34.

[0006] In the prior art, the above processing is repeatedly performed on the time-series images captured by cameras 30a and 30b to generate time-series 3D key points. The time-series 3D key points become the result of 3D skeleton recognition. In Figure 33 the above, cameras 30a and 30b are used for illustration, but other cameras can also be used to generate time-series 3D key points.

[0007] Here, when performing 3D skeleton recognition on a person performing gymnastics, sometimes an assistant other than the person being the object may intrude in front of the camera, resulting in confusion of the 3D key points.

[0008] Figure 34 This is a diagram showing an example of poor observation. Figure 34The images 35a, 35b, 35c, and 35d are images respectively captured by cameras with different shooting positions. In the images 35a to 35d, the person who is the object of 3D skeleton recognition is set as person U2. For example, in the image 35b, there is a person U3 other than person U2. In such a situation, if a 3D key point sequence is generated based on the images 35a to 35d and the subsequent respective images, a 3D key point sequence 10 is generated. In the 3D key point sequence 10, among the 3D key points 10-1, 10-2, 10-3, 10-4, and 10-5, the 3D key points 10-2, 10-3, and 10-4 are disordered.

[0009] In order to ensure the accuracy of 3D skeleton recognition, the interval where the 3D key points are disordered is determined as the correction interval, and the 3D key points in the correction interval are corrected based on the actions of the 3D key points before and after the correction interval.

[0010] Figure 35 It is a diagram for explaining the correction method of the prior art. In Figure 35 In the example shown, the 3D key point sequence 10 is used for explanation. The 3D key point sequence 10 includes 3D key points 10-1 to 10-5. In the prior art, by performing anomaly detection on the 3D key point sequence 10, the disordered 3D key points 10-2, 10-3, and 10-4 are detected. The interval of the disordered 3D key points 10-2, 10-3, and 10-4 is set as the correction interval 10a. In the prior art, the 3D key points 10-2 to 10-4 are corrected using the 3D key points 10-1 and 10-5 before and after the correction interval 10a.

[0011] Here, as the prior art for correcting the 3D key points in the correction interval based on the 3D key points before and after the correction interval, there are Spherical liner interpolation (Slerp) and Liner Interpolation (Lerp). Figure 36 It is a diagram for explaining Slerp and Lerp. Slerp is spherical linear interpolation, and Lerp is linear interpolation. Spherical linear interpolation is a method of correcting the joints of two points at separated locations by assuming that the space between each joint is a sphere (rotation). Linear interpolation is a method of correcting the joints of two points at separated locations by assuming that the space between each joint is a straight line. For example, in Figure 35 In the example described, based on the positions of the joints of the 3D key points 10-1 and 10-5 before and after the correction interval 10a, Slerp, and Lerp, the optimal positions of each joint are determined to correct the 3D key points 10-2 to 10-4.

[0012] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2021-135877

[0013] However, in the above prior art, there are problems such as being unable to appropriately set the correction range and being unable to improve the recognition accuracy of the human body's bones.

[0014] For example, if the correction range is not appropriately set, there may be a situation where the coordinates of each joint of the 3D key points before and after the correction range are inappropriate, and appropriate correction cannot be performed. Summary of the Invention

[0015] In one aspect of the technology, an object of the present invention is to provide an information processing method, an information processing program, and an information processing device that can appropriately set a correction range and improve the recognition accuracy of the human body's bones.

[0016] In a first aspect of the technology, the computer is made to execute the following processing. The computer calculates abnormal features based on a plurality of bone information respectively setting a plurality of joints included in the human body and coordinates regarding the plurality of joints. The computer calculates, as a first threshold value, a boundary value with the maximum separation degree between a first distribution of normal bone information related to the abnormal features and a second distribution of abnormal bone information related to the abnormal features among the plurality of bone information. The computer sets a correction range of the time-series bone information based on the first threshold value.

[0017] It is possible to appropriately set the correction range and improve the recognition accuracy of the human body's bones. Brief Description of the Drawings

[0018] Figure 1 It is a diagram showing an example of a human body model.

[0019] Figure 2 It is a diagram showing an example of joint names.

[0020] Figure 3 It is a diagram showing the system of this embodiment.

[0021] Figure 4 It is a diagram for explaining the stillness determination.

[0022] Figure 5 It is a diagram for explaining the circular motion determination.

[0023] Figure 6 It is a flowchart showing the processing steps of the correction processing corresponding to the motion state of this embodiment.

[0024] Figure 7 It is a diagram for explaining the implement contact determination.

[0025] Figure 8 It is a diagram for explaining the whole body position interpolation of the correction range based on the implement contact.

[0026] Figure 9 This is a diagram showing an example of the motion prediction result.

[0027] Figure 10 This is a diagram showing an example of the distribution of 3D key points related to abnormal features.

[0028] Figure 11 This is a diagram for explaining the process of calculating the first abnormal detection threshold.

[0029] Figure 12 This is a diagram for explaining the process of calculating the second abnormal detection threshold.

[0030] Figure 13 This is a diagram for explaining the first abnormal detection.

[0031] Figure 14 This is a diagram showing an example of the optimization of the abnormal interval based on the first convergence condition.

[0032] Figure 15 This is a diagram showing an example of the optimization of the abnormal interval based on the second convergence condition.

[0033] Figure 16 This is a functional block diagram showing the structure of the information processing device of this embodiment.

[0034] Figure 17 This is a diagram showing an example of the data structure of the 3D key point table.

[0035] Figure 18 This is a diagram showing an example of the reference posture.

[0036] Figure 19 This is a diagram for explaining an example of the three-dimensional rotation angle of ROOT.

[0037] Figure 20 This is a diagram for explaining an example of the three-dimensional rotation angle of JOINT.

[0038] Figure 21 This is a diagram for explaining the process of calculating the three-dimensional rotation angle of ROOT by rigid body alignment.

[0039] Figure 22 This is a diagram (1) for explaining the process of calculating the three-dimensional rotation angle of JOINT by the Rodrigues rotation formula.

[0040] Figure 23 This is a diagram (2) for explaining the process of calculating the three-dimensional rotation angle of JOINT by the Rodrigues rotation formula.

[0041] Figure 24It is a flowchart showing the processing steps of the information processing apparatus according to the present embodiment.

[0042] Figure 25 It is a flowchart showing the processing steps of the correction range determination process.

[0043] Figure 26 It is a flowchart showing the processing steps of the correction process.

[0044] Figure 27 It is a flowchart showing the processing steps of the correction process based on appliance contact.

[0045] Figure 28 It is a flowchart showing the processing steps of the process of setting a threshold value.

[0046] Figure 29 It is a diagram (1) for explaining the effects of the information processing apparatus.

[0047] Figure 30 It is a diagram (2) for explaining the effects of the information processing apparatus.

[0048] Figure 31 It is a diagram (3) for explaining the effects of the information processing apparatus.

[0049] Figure 32 It is a diagram showing an example of the hardware configuration of a computer that implements the same functions as the information processing apparatus of the embodiment.

[0050] Figure 33 It is a diagram for explaining the prior art related to 3D skeleton recognition.

[0051] Figure 34 It is a diagram showing an example of poor observation.

[0052] Figure 35 It is a diagram for explaining the prior art correction method.

[0053] Figure 36 It is a diagram for explaining Slerp and Lerp. Detailed implementation mode

[0054] Hereinafter, based on the drawings, embodiments of the information processing method, information processing program, and information processing apparatus disclosed in the present application will be described in detail. In addition, the present invention is not limited to this embodiment.

[0055] Embodiment

[0056] Before describing this embodiment, an example of a human model will be described. Figure 1 It is a diagram showing an example of a human model. As Figure 1As shown, a human model is defined by 21 joints ar0 to ar20.

[0057] Figure 1 The relationship between each of the joints ar0 to ar20 shown and the joint names is as Figure 2 shown. Figure 2 is a diagram showing an example of joint names. For example, the joint name of joint ar0 is "SPINE_BASE". The joint names of joints ar1 to a20 are as Figure 2 shown, and the description is omitted.

[0058] The 3D key points processed in this embodiment are data in which three-dimensional coordinates corresponding to the joints ar0 to ar20 of the human model are set. In the following description, the 3D key points arranged in time series are referred to as "3D key point sequence".

[0059] Next, an example of the system of this embodiment will be described. Figure 3 is a diagram showing the system of this embodiment. As Figure 3 shown, the system of this embodiment includes cameras 30a, 30b and an information processing device 100. The cameras 30a, 30b and the information processing device 100 are connected to each other wirelessly or by wire.

[0060] The cameras 30a, 30b capture images of the person U4. As an example, the person U4 uses an instrument to compete. In the following description, the cameras 30a, 30b are collectively referred to as the camera 30.

[0061] The camera 30 sends the data of the captured image to the information processing device 100. In the following description, the data of the image is referred to as "image data". The image data includes image frames in time series. Frame numbers are assigned to each image frame in ascending order. An image frame is data of a still image captured by the camera 30 at a certain timing.

[0062] The information processing device 100 generates a 3D key point sequence based on the image data received from the camera 30. The information processing device 100 sets a correction interval for the 3D key point sequence, and determines the motion state of the person U4 based on the 3D key points before and after the correction interval. The information processing device 100 determines whether the person U4 is in contact with the instrument based on the 3D key points before and after the correction interval. The information processing device 100 corrects the 3D key points in the correction interval based on the motion state of the person U4 and the determination result of whether the person U4 is in contact with the instrument.

[0063] In addition, the information processing device 100 acquires in advance the distribution of normal 3D key points and the distribution of abnormal 3D key points, and determines the first abnormal detection threshold Th 1 and the second abnormal detection threshold Th2 。The first anomaly detection threshold Th 1 is the boundary value at which the separation between the distribution of normal 3D key points and the distribution of abnormal 3D key points is maximized. The second anomaly detection threshold Th 2 is a threshold with a higher sensitivity than the first anomaly detection threshold Th 1 The information processing device 100 sets a correction interval for the 3D key point sequence based on the first anomaly detection threshold Th 1 and the second anomaly detection threshold Th 2

[0064] Hereinafter, an example of the process of the information processing device 100 determining the motion state will be described. The process of determining the motion state includes "long-term motion determination", "stationary determination", "circular motion determination", and "circular motion direction determination". In the following description, the 3D key points will be appropriately referred to as frames.

[0065] The "long-term motion determination" performed by the information processing device 100 will be described. The information processing device 100 determines the number of 3D key points (hereinafter, the number of frames) in the correction interval set for the 3D key point sequence. When the number of frames is the threshold Th 3 , the information processing device 100 determines that the motion state is long-term motion. On the other hand, when the number of frames is less than the threshold Th 3 , the information processing device 100 determines that the motion state is not long-term motion. The threshold Th 3 is set in advance.

[0066] The "stationary determination" performed by the information processing device 100 will be described. The information processing device 100 determines the 3D key points in front of the correction interval and the 3D key points behind the correction interval set for the 3D key point sequence. In the following description, the 3D key points in front of the correction interval will be referred to as "previous frames". The 3D key points behind the correction interval will be referred to as "subsequent frames". For example, if the correction interval is set to the correction interval 10a described in Figure 35 , the previous frame is the 3D key point 10-1, and the subsequent frame is the 3D key point 10-5. When the difference in the full-body rotation from the previous frame to the subsequent frame is less than the threshold Th 4 , the information processing device 100 determines that the motion state is stationary motion. On the other hand, when the difference in the full-body rotation from the previous frame to the subsequent frame is the threshold Th 4 or more, the information processing device 100 determines that the motion state is not stationary motion.

[0067] Figure 4 is a diagram for explaining the stationary determination. In Figure 4 ​In this case, the previous frame is set as "previous frame 14s", and the subsequent frame is set as "subsequent frame 14e". The reference line of the previous frame 14s is set as "reference line 14s-1". The reference line of the subsequent frame 14e is set as "reference line 14e-1". The information processing device 100 determines the angle 14-1 formed by the reference line 14s-1 and the reference line 14e-1 as the difference in the full-body rotation from the previous frame 14s to the subsequent frame 14e. When the formed angle 14-1 is less than the threshold Th 4 , the information processing device 100 determines that the motion state is a stationary motion. When the formed angle 14-1 is the threshold Th 4 or more, the information processing device 100 determines that the motion state is not a stationary motion.

[0068] An explanation is given for the "circular motion determination" performed by the information processing device 100. The information processing device 100 calculates the predicted rotation axis v start based on the quaternion q of the previous frame in the correction interval start-1 and the quaternion q of the frame immediately preceding the previous frame (3D key points). pred Here, the quaternion of a certain frame represents the rotation from the coordinates of the joints of a certain frame to the coordinates of the joints of the reference pose. The reference pose is data defined by a plurality of nodes corresponding to the joints of the human body, and the coordinates of each node are preset in advance. The predicted rotation axis v pred corresponds to the normal of the relative quaternion q start-1 representing the relative rotation from the quaternion q start to the quaternion q -1 start-1 *q start .

[0069] The information processing device 100 calculates the quaternion q start of the short path and the quaternion q end of the short path based on the quaternion q of the previous frame in the correction interval short_end and the quaternion q of the subsequent frame in the correction interval. The information processing device 100 calculates the quaternion q long_end based on Equation (1). The information processing device 100 calculates the quaternion q short_end based on Equation (2). long_end .

[0070] [Mathematical formula 1]

[0071]

[0072] [Mathematical formula 2]

[0073]

[0074] The information processing device 100 calculates the rotation angle θ based on the quaternion q start and the quaternion qshort_end to calculate the rotation axis v in the short path short The rotation axis v short corresponds to the normal of the relative quaternion q start from the quaternion q short-end to the quaternion q -1 start *q short-end corresponding to the relative rotation up to

[0075] The information processing device 100 calculates the rotation axis v in the long path based on the quaternion q start and the quaternion q long_end to calculate the rotation axis v in the long path long The rotation axis v long corresponds to the normal of the relative quaternion q start from the quaternion q long-end to the quaternion q -1 start *q long-end corresponding to the relative rotation up to

[0076] The information processing device 100 calculates the first cosine similarity between the predicted rotation axis v pred and v short In addition, the information processing device 100 calculates the second cosine similarity between the predicted rotation axis v pred and v long When at least one of the first cosine similarity or the second cosine similarity is greater than or equal to the threshold Th 5 , the information processing device 100 determines that the motion state is circular motion. On the other hand, when both the first cosine similarity and the second cosine similarity are less than the threshold Th 5 , the information processing device 100 determines that the motion state is not circular motion.

[0077] Figure 5 is a diagram for explaining the circular motion determination. In Figure 5 , the previous frame of the correction interval is set as frame 15s. The frame before frame 15s is set as frame 15s'. The subsequent frame is set as frame 15e. The information processing device 100 calculates the predicted rotation axis v start based on the quaternion q of frame 15s in the correction interval start-1 and the quaternion q of frame 15s' pred .

[0078] The information processing device 100 calculates the rotation axis v in the short path based on the quaternion q start and the quaternion q short_end to calculate the rotation axis v in the short path short The information processing device 100 calculates the rotation axis v in the short path based on the quaternion q start and the quaternion q long_endTo calculate the rotation axis v in the long path long . For example, the information processing device 100 calculates the predicted rotation axis v pred and q short_end 's first cosine similarity (0.2). The information processing device 100 calculates the predicted rotation axis v pred and q long_end 's second cosine similarity (0.8). If the information processing device 100 sets the threshold Th 5 to "0.5", the second cosine similarity is above the threshold Th 5 , so it is determined that the motion state is rotational motion.

[0079] An explanation is given for the "circular motion direction determination" performed by the information processing device 100. When the motion state of the information processing device 100 is circular motion, it performs the involved circular motion method determination. When the first cosine similarity among the first cosine similarity and the second cosine similarity is above the threshold Th 5 , the information processing device 100 determines the direction of the short path as the direction of circular motion. On the other hand, when the second cosine similarity among the first cosine similarity and the second cosine similarity is above the threshold Th 5 , the information processing device 100 determines the direction of the long path as the direction of circular motion.

[0080] In Figure 5 the example described, since the second cosine similarity is above the threshold Th 5 , the direction of the long path is determined as the direction of circular motion.

[0081] The information processing device 100 determines the motion state of the person U4 by performing the above processing. The information processing device 100 corrects the 3D key points in the correction interval based on the determination result of the motion state. The information processing device 100 selects any one of interpolation based on Slerp (short path), interpolation based on Slerp (long path), and interpolation based on a machine learning model for correction.

[0082] For example, the information processing device 100 is based on Figure 6 the processing steps, and selects any one of interpolation based on Slerp (short path), interpolation based on Slerp (long path), and interpolation based on a machine learning model for correction.

[0083] Figure 6 is a flowchart showing the processing steps of the correction process corresponding to the motion state of this embodiment. As Figure 6As shown, the information processing device 100 receives the input of 3D key points, full body position, full body rotation, and joint rotations (step S101). The information processing device 100 performs long-term motion determination (step S102).

[0084] When the motion state of the information processing device 100 is long-term motion (step S103, yes), it performs interpolation based on a machine learning model (step S104) and transfers to step S113.

[0085] On the other hand, when the motion state of the information processing device 100 is not long-term motion (step S103, no), it performs static motion determination (step S105). When the motion state is static motion (step S106, yes), it performs interpolation based on Slerp (short path) (step S107) and transfers to step S113.

[0086] On the other hand, when the motion state of the information processing device 100 is not static motion (step S106, no), it performs circular motion determination (step S108). When the motion state is not circular motion (step S109, no), it transfers to step S107.

[0087] On the other hand, when the motion state of the information processing device 100 is circular motion (step S109, yes), it performs circular motion direction determination (step S110). When the direction of circular motion is not the same as the long path direction (step S111, no), it transfers to step S107. In addition, the same direction as the long path means the same direction as the short path.

[0088] On the other hand, when the direction of circular motion is the same as the long path direction (step S111, yes), it performs interpolation based on Slerp (long path) (step S112) and outputs the interpolation result (step S113).

[0089] Next, the interpolation based on Slerp (short path), the interpolation based on Slerp (long path), and the interpolation based on the machine learning model performed by the information processing device 100 will be described respectively.

[0090] The "interpolation based on Slerp (short path)" performed by the information processing device 100 will be described. The information processing device 100 performs the interpolation based on Slerp (short path) based on Equation (3). In Equation (3), "t" represents the index of time, defined by "t = n / N". "n" represents the index. "N" represents the number of frames in the interpolation interval. "θ" represents the quaternion q start with the quaternion q short_endThe formed angle.

[0091] [Mathematical formula 3]

[0092]

[0093] The information processing device 100 calculates the quaternion q at time t based on Equation (3) t , and corrects the coordinates of the joints of the 3D key points corresponding to the time t in the correction interval in accordance with the quaternion q t .

[0094] An explanation is given of the "interpolation based on Slerp (long path)" performed by the information processing device 100. The information processing device 100 performs interpolation based on Slerp (long path) based on Equation (4). In Equation (4), "t" represents the index of time, defined by "t = n / N". "n" represents the index. "N" represents the number of frames in the interpolation interval. "θ" represents the angle formed by the quaternion q start and the quaternion q long_end .

[0095] [Mathematical formula 4]

[0096]

[0097] The information processing device 100 calculates the quaternion q at time t based on Equation (4) t , and corrects the coordinates of the joints of the 3D key points corresponding to the time t in the correction interval in accordance with the quaternion q t .

[0098] An explanation is given of the "interpolation based on a machine learning model" performed by the information processing device 100. For example, the information processing device 100 uses a trained machine learning model that takes the previous frame and the next frame as inputs and outputs the paths of the coordinates of each joint from the previous frame to the next frame. The machine learning model is an NN (Neural Network) or the like. The information processing device 100 corrects the 3D key points in the correction interval based on the output result of the machine learning model.

[0099] In addition, the information processing device 100 can also use the machine learning model described in the non-patent literature "Felix G. Harvey, et al. “Robust Motion In-betweening”, In ACM Transactions on Graphics (TOG), 2020" to interpolate the 3D key points in the correction interval.

[0100] Here, after the information processing device 100 performs the correction corresponding to the above-described motion state, it performs an "implement contact determination" to determine whether the person U4 is in contact with the implement. When the information processing device 100 determines that the person U4 is in contact with the implement through the implement contact determination, it performs a "full-body position interpolation based on the correction interval of the implement contact". On the other hand, when the information processing device 100 determines that the person U4 is not in contact with the implement through the implement contact determination, it performs a "full-body position correction based on motion prediction".

[0101] An "implement contact determination" performed by the information processing device 100 will be described. Figure 7 This is a diagram for explaining the implement contact determination. The coordinates of the implement eq1, the joints to be determined (determination joints), and the threshold Th are defined in advance. 6 The coordinates of the implement eq1 can be defined by a point based on an n-dimensional coordinate (= 1, 2, 3), a plane based on multiple n-dimensional coordinates, etc. For example, the information processing device 100 defines a point based on a two-dimensional coordinate by projecting the three-dimensional coordinates of the implement eq1 onto the yz plane. The determination joints are set to the joints of the left fingertip and the joints of the right fingertip. In Figure 1 the human model described, the joint of the left fingertip is the joint ar19. The joint of the right fingertip is the joint ar20.

[0102] The information processing device 100 selects one joint from among the multiple determination joints as the "attention joint" and calculates the distance d between the attention joint and the implement eq1. The information processing device 100 performs the above processing on all the determination joints in the previous frame and the subsequent frame of the correction interval. When there is one or more attention joints with d < Th 6 in both the previous frame and the subsequent frame, the information processing device 100 determines that the person U4 is in contact with the implement q1.

[0103] In Figure 7 the description, a preset point among the points on the implement eq1 in the yz plane is set as p e = (ye, ze). When the distance d between the joint ar19 and p e is d < Th 6 in both the previous frame and the subsequent frame, the information processing device 100 determines that the person U4 is in contact with eq1. In the following description, joints with d < Th 6 in the previous frame and the subsequent frame of the attention joint are recorded as "contact joints".

[0104] An "full-body position interpolation based on the correction interval of the implement contact" performed by the information processing device 100 will be described. Figure 8This is a diagram for explaining the interpolation of the whole body position in the correction interval based on the appliance contact. The information processing device 100 calculates the contact joint p of interest by performing the following processing start , p end , the rotation center p e_origin , the vector v start , v end .

[0105] The information processing device 100 calculates the average value of the coordinates of the contact joints included in the previous frame as the contact joint p of interest start . When there is only one contact joint included in the previous frame, the information processing device 100 directly sets the coordinates of the contact joint as the contact joint p of interest start .

[0106] The information processing device 100 calculates the average value of the coordinates of the contact joints included in the next frame as the contact joint p of interest end . When there is only one contact joint included in the next frame, the information processing device 100 directly sets the coordinates of the contact joint as the contact joint p of interest end .

[0107] The information processing device 100 calculates the rotation center p based on Equation (5) e_origin . The rotation center p e_origin is the rotation center for calculating the circular orbit centered on the appliance. The x coordinate of the rotation center p e_origin is the average of the x coordinate (x start ) of the contact joint p of interest and the x coordinate (x start ) of the contact joint p of interest. The y coordinate and z coordinate of the rotation center p end are p end in the yz plane as described in e_origin . Figure 7 e .

[0108] [Mathematical formula 5]

[0109]

[0110] The information processing device 100 calculates the vector v e_origin from the rotation center p start towards the contact joint p of interest. The information processing device 100 calculates the vector v start from the rotation center p e_origin towards the contact joint p of interest end . The information processing device 100 calculates the unit vector e start of the vector v end and the unit vector e of the vector v start end end ​​​。

[0111] Next, the information processing device 100 calculates, based on Equation (6), the interpolation result p that smoothly connects the focus contact joint p start with the focus contact joint p end through Slerp passing through the center of the appliance. t 。As defined in Equation (7), "s" included in Equation (6) is defined. In Equation (6), an example using unit vectors is illustrated, but quaternions can be used instead of unit vectors.

[0112] [Mathematical formula 6]

[0113]

[0114] [Mathematical formula 7]

[0115] s = (1 - t)[v start | + t[v end |…(7)

[0116] The information processing device 100 performs a translational movement of the whole body position for the 3D key points in the correction interval that have been corrected corresponding to the motion state, so that the position of the focus joint coincides with p t .

[0117] An explanation is given for "whole body position interpolation in the correction interval based on motion prediction" performed by the information processing device 100. The information processing device 100 performs spline interpolation based on multiple frames in front of and behind the correction interval to predict the motion in the correction interval.

[0118] The degree p of spline interpolation is defined in advance. If the start of the correction interval is set as s, the end is set as e, and the number of frames is set as N, then the information processing device 100 performs spline interpolation of degree p on the x-axis, y-axis, and z-axis based on the whole body position coordinates of the interval [s - N, s - 1], "e + 1, e + N", and outputs the interpolation result of the interval [s, e].

[0119] Figure 9 is a diagram showing an example of the motion prediction result. Figure 9 The horizontal axis of the chart of corresponds to the frame number, and the vertical axis corresponds to the coordinate value (whole body position coordinate). In Figure 9 , the plotted points in the area a1-1 correspond to the whole body position coordinates of the interval [s - N, s - 1]. The plotted points in the area a1-2 correspond to the whole body position coordinates of the interval "e + 1, e + N". The plotted points in the area a1-3 correspond to the whole body position coordinates of the interval [s, e] that becomes the interpolation result.

[0120] The information processing device 100 adjusts the 3D key points in the correction interval, which have been corrected corresponding to the motion state, based on the whole body position coordinates in the interval [s, e]. For example, the information processing device 100 translates the whole body position so that the center of the 3D key points in the correction interval coincides with the center of the whole body position coordinates.

[0121] Next, an example of the process of calculating the first anomaly detection threshold Th used by the information processing device 100 when setting the correction interval for the 3D key point sequence is described. 1 and the second anomaly detection threshold Th 2 is described. The information processing device 100 acquires in advance a plurality of normal 3D key points and a plurality of abnormal 3D key points. The information processing device 100 calculates the value of the "anomaly feature" for the plurality of normal 3D key points and the plurality of abnormal 3D key points respectively. In the following description, the value of the anomaly feature is simply referred to as the anomaly feature.

[0122] In order to calculate the anomaly feature, the information processing device 100 calculates the "3σ threshold" based on a plurality of normal 3D key points. The 3σ threshold involved is set for each bone length between the joints of the human body model. Here, as an example, the process of calculating the 3σ threshold set for the bone length l8-9 that defines the bone connecting the joint ar8 and the joint ar9 of the human body model shown is described. Figure 1 is described.

[0123] The information processing device 100 respectively obtains the bone length l8-9 from a plurality of normal 3D key points and calculates the variance σ 2 of the bone length l8-9. The information processing device 100 calculates the σ of the bone length l8-9 by taking the square root of the variance σ 2 of the bone length l8-9. The information processing device 100 calculates 3σ by multiplying σ by 3. Thus, the information processing device 100 calculates the 3σ threshold of the bone length l8-9.

[0124] The information processing device 100 also performs the above processing on the bone lengths between other joints of the human body model, thereby calculating the 3σ thresholds of the bone lengths between other joints. Thus, the 3σ thresholds corresponding to each bone length are obtained.

[0125] The information processing device 100 determines the bone lengths of the 3D key points. For each bone length, it calculates the difference (absolute value difference) between the bone length and the 3σ threshold corresponding to the bone length, and calculates the sum of the differences as the anomaly feature of the 3D key points.

[0126] The information processing device 100 calculates the anomaly feature for the plurality of normal 3D key points and the plurality of abnormal 3D key points acquired in advance respectively.

[0127] Figure 10 This is a diagram showing an example of the distribution of 3D key points related to abnormal features. Figure 10 In the graph of, the horizontal axis corresponds to the abnormal features, and the vertical axis corresponds to the number of data of the 3D key points. In Figure 10 , dis1 is the distribution of normal 3D key points. dis2 is the distribution of abnormal 3D key points.

[0128] Figure 11 This is a diagram for explaining the process of calculating the first anomaly detection threshold. The information processing device 100 calculates the boundary value with the largest separation between the distribution dis1 of normal 3D key points and the distribution dis2 of abnormal 3D key points as the first anomaly detection threshold Th 1 . For example, the information processing device 100 calculates the first anomaly detection threshold Th by performing the following process 1 .

[0129] The information processing device 100 calculates v based on Equation (8) normal . μ included in Equation (8) normal represents the average value of the abnormal features of normal 3D key points. σ normal is the value obtained by taking the square root of the variance of the abnormal features of normal 3D key points.

[0130] [Mathematical formula 8]

[0131] v normat = μ normal + 3σ normal …(8)

[0132] The information processing device 100 calculates v based on Equation (9) anormal . μ included in Equation (9) anormal represents the average value of the abnormal features of abnormal 3D key points. σ anormal is the value obtained by taking the square root of the variance of the abnormal features of abnormal 3D key points.

[0133] [Mathematical formula 9]

[0134] v anomaly = μ anomaly - 3σ anomaty …(9)

[0135] The information processing device 100 determines the search range [v low , v high based on Equation (10). The search range [v low , v high is the range with a high probability of the largest separation between the distribution dis1 of normal 3D key points and the distribution dis2 of abnormal 3D key points.

[0136] [Formula 10]

[0137]

[0138] The information processing device 100 calculates the search range [v low , v high ] is the value with the largest separation degree, as the first abnormality detection threshold Th 1 The information processing device 100 may also determine the search range [v based on the document “N. Otsu, “A threshold selection method from gray level histograms,” IEEE Trans. Syst. Man Cybern., vol. 9, no. 1, pp. 62-66, 1979.” low , v high ] has the largest separation value. In addition, the information processing device 100 searches the range [v low , v high The abnormal feature at the intersection of the distribution dis1 and the distribution dis2 of ] is set as the first abnormality detection threshold Th 1 .

[0139] Figure 12 1 is a diagram for explaining the process of calculating the second abnormality detection threshold. The information processing device 100 sets the sensitivity to be higher than the first abnormality detection threshold Th 1 A high threshold value is used, so the search range [v low , v high ] low Set as the second abnormality detection threshold Th 2 .

[0140] By executing the above-mentioned processing by the information processing device 100, the first abnormality detection threshold value Th is calculated. 1 and the second abnormality detection threshold Th 2 .

[0141] In addition, in this embodiment, as an example, the information processing device 100 sets the search range [v low , v high ] low Set as the second abnormality detection threshold Th 2 , but you can also end with high Set as the second abnormality detection threshold Th 2 In this way, if the end v is set high , then set the sensitivity to the first abnormality detection threshold Th 1 Low threshold.

[0142] Next, a description will be given of the process in which the information processing apparatus 100 determines a correction interval of the 3D key point sequence based on the first abnormality detection threshold Th 1 and the second abnormality detection threshold Th 2 The information processing apparatus 100 performs first abnormality detection and second abnormality detection when setting a correction interval.

[0143] A description will be given of the first abnormality detection performed by the information processing apparatus 100. Figure 13 is a diagram for explaining the first abnormality detection. In Figure 13 the graph, the vertical axis corresponds to the abnormality distance, and the horizontal axis corresponds to the frame number of the 3D key point sequence.

[0144] The information processing apparatus 100 selects the 3D key point of the frame number n included in the 3D key point sequence. The information processing apparatus 100 determines the length of each bone of the selected 3D key point. For each bone length, the difference (absolute value difference) between the bone length and the 3σ threshold corresponding to the bone length is calculated, and the sum of the differences is calculated as the abnormality feature of the 3D key point. The 3σ threshold is set in advance through the above processing.

[0145] The information processing apparatus 100 determines the relationship between the frame number of the 3D key point and the abnormality feature by repeatedly performing the above processing on the 3D key points of each frame number of the 3D key point sequence. In Figure 13 the example shown, the relationship between the frame number of each 3D key point included in the 3D key point sequence and the abnormality feature is represented by the line L1.

[0146] The information processing apparatus 100 compares the line L1 with the first abnormality detection threshold Th 1 , and determines the interval where the abnormality feature is greater than or equal to the first abnormality detection threshold Th 1 as the abnormal interval In1. In Figure 13 the example shown, the frame number of the previous frame of the abnormal interval In1 is "1054", and the frame number of the subsequent frame is "1060".

[0147] In addition, even in the interval where the abnormality feature is abnormal with respect to the first abnormality detection threshold Th 1 , the information processing apparatus 100 can remove the part where the number of frames of the interval is less than the specified number of frames from the abnormal interval.

[0148] A description will be given of the second abnormality detection performed by the information processing apparatus 100. Let the previous frame of the abnormal interval be "s", and the subsequent frame of the abnormal interval be "e". Let the abnormality feature in the previous frame s be "f s ". Let the abnormality feature in the previous frame e be "f e". Set the number of frames in the abnormal section to "N". The information processing device 100 sets the first convergence condition or the second convergence condition and optimizes the abnormal section. The initial value of the abnormal section becomes the abnormal section In1 detected by the above-mentioned first abnormality detection.

[0149] First, the process of setting the first convergence condition for the information processing device 100 and optimizing the abnormal section will be described. The first convergence condition is the following condition. Th 2 is the second abnormality detection threshold Th 2 .

[0150] f s <Th 2 and f e <Th 2 …(First convergence condition)

[0151] The information processing device 100 compares f s and f e , updates the frame with the larger abnormal feature, and advances the step. For example, when f s is larger than f e , the information processing device 100 sets the frame with the frame number obtained by subtracting 1 from the frame number of the previous frame s as the new previous frame. When f s is smaller than f e , the information processing device 100 sets the frame with the frame number obtained by adding 1 to the frame number of the subsequent frame e as the new subsequent frame e.

[0152] The information processing device 100 repeatedly executes the process of calculating f s or f e for the new previous frame s or the new subsequent frame e until the first convergence condition (or the second convergence condition) is met. The information processing device 100 uses the evaluation value E = f s + f e as the smallest previous frame s and subsequent frame e, and updates the abnormal section.

[0153] Figure 14 is a diagram showing an example of the optimization of the abnormal section based on the first convergence condition. The information processing device 100 executes the processes of steps 1 to 4 described below.

[0154] Step 1 will be described. The information processing device 100 sets the frame number of the previous frame and the frame number of the subsequent frame based on the abnormal section determined by the first abnormality detection. Here, it will be described by setting the frame number of the previous frame s to "1054" and the frame number of the subsequent frame e to "1060". Set the abnormal feature f s of the previous frame s to "104". Set the abnormal feature f eSet it to "115". The evaluation value E is "219". The number of frames N in the abnormal interval is "7". Since the first convergence condition is not satisfied, f in step 1 s and f e are "FALSE".

[0155] Explanation of step 2. Since f s is smaller than f e , the information processing device 100 updates the frame number of the subsequent frame e to "1061". Set the abnormal feature f e of the subsequent frame e to "15". The frame number and abnormal feature f s of the previous frame s are the same as those in step 1. The evaluation value E is "119". The number of frames N in the abnormal interval is "8". Since the first convergence condition is not satisfied, f in step 2 s and f e are "FALSE".

[0156] Explanation of step 3. Since f e is smaller than f s , the information processing device 100 updates the frame number of the previous frame s to "1053". Set the abnormal feature f s of the previous frame s to "108". The frame number and abnormal feature f e of the subsequent frame e are the same as those in step 2. The evaluation value E is "195". The number of frames N in the abnormal interval is "9". Since the first convergence condition is not satisfied, f in step 3 s and f e are "FALSE".

[0157] Explanation of step 4. Since f e is smaller than f s , the information processing device 100 updates the frame number of the previous frame s to "1052". Set the abnormal feature f s of the previous frame s to "23". The frame number and abnormal feature f e of the subsequent frame e are the same as those in step 3. The evaluation value E is "38". The number of frames N in the abnormal interval is "10". Since the first convergence condition is satisfied, f in step 4 s and f e are "TRUE".

[0158] In step 4, since the first convergence condition is satisfied, the information processing device 100 determines the step with the smallest evaluation value E among the evaluation values E of steps 1 to 4. In Figure 14In the example shown, the evaluation value "38" in step 4 is the smallest. The information processing device 100 sets the range from the frame number "1053" that is the frame number after the previous frame s in step 4 to the frame number "1060" that is the frame number before the subsequent frame e as the abnormal range In2. That is, the abnormal range In1 set by the first abnormal detection is optimized to the abnormal range In2.

[0159] Next, the process of setting the second convergence condition for the information processing device 100 and optimizing the abnormal range will be described. The second convergence condition is the following condition.

[0160] Satisfy the first convergence condition or N≥9 ··· (second convergence condition)

[0161] The information processing device 100 updates the frame number of the previous frame s and the frame number of the subsequent frame e until the second convergence condition is satisfied, and updates f s 、f e 、The processing of the evaluation value E is the same as the above-described processing related to the first convergence condition.

[0162] Figure 15 FIG. is an example showing the optimization of the abnormal range based on the second convergence condition. The information processing device 100 executes the processes of steps 1 to 3 described below.

[0163] Step 1 will be described. The information processing device 100 sets the frame number of the previous frame and the frame number of the subsequent frame based on the abnormal range determined by the first abnormal detection. Here, the description will be made by setting the frame number of the previous frame s as "1054" and the frame number of the subsequent frame e as "1060". The abnormal feature f s of the previous frame s is set as "104". The abnormal feature f e of the subsequent frame e is set as "115". The evaluation value E is "219". The number of frames N in the abnormal range is "7". Since the first convergence condition is not satisfied, f s and f e in step 1 are "FALSE".

[0164] Step 2 will be described. Since f s is smaller than f e , the information processing device 100 updates the frame number of the subsequent frame e to "1061". The abnormal feature f e of the subsequent frame e is set as "15". The frame number and the abnormal feature f s of the previous frame s are the same as those in step 1. The evaluation value E is "119". The number of frames N in the abnormal range is "8". Since the first convergence condition is not satisfied, f s and f e in step 2 are "FALSE".

[0165] Step 3 will be described. Since fe smaller than f s is smaller, so the information processing device 100 updates the frame number of the previous frame s to "1053". Sets the abnormal feature f of the previous frame s s to "108". The frame number and abnormal feature f of the subsequent frame e e are the same as in step 2. The evaluation value E is "195". The number of frames N in the abnormal interval is "9". Since the second convergence condition is satisfied, N in step 3 is "TURE".

[0166] In step 3, since the second convergence condition is satisfied, the information processing device 100 determines the step with the smallest evaluation value E among the evaluation values E in steps 1 to 3. In Figure 15 the example shown, the evaluation value "119" in step 2 is the smallest. The information processing device 100 sets the interval from the frame number "1055" after the previous frame s in step 2 to the frame number "1060" before the subsequent frame e as the abnormal interval In3. That is, the abnormal interval In1 set by the first abnormal detection is optimized to the abnormal interval In3.

[0167] The information processing device 100 determines the optimized abnormal interval as the correction interval for the 3D key point sequence.

[0168] Next, a structural example of the information processing device 100 that performs the processing described above Figures 4 to 15 will be described. Figure 16 is a functional block diagram showing the structure of the information processing device of this embodiment. As Figure 16 shown, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0169] The communication unit 110 performs data communication with the camera 30, external devices, etc. via a network. The communication unit 110 is a NIC (Network Interface Card), etc. The control unit 150 described later exchanges data with external devices via the communication unit 110.

[0170] The input unit 120 is an input device that inputs various information to the control unit 150 of the information processing device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc.

[0171] The display unit 130 is a display device that displays the information output from the control unit 150.

[0172] The storage unit 140 has an image buffer 141, a 3D key point table 142, a machine learning model 143, and a technology identification table 144. The storage unit 140 is a storage device such as a memory.

[0173] The image buffer 141 is a buffer that stores the image data acquired from the camera 30. The image data contains time-series image frames, and frame numbers are assigned to each image frame in ascending order.

[0174] The 3D key point table 142 is a table that stores information related to 3D key points. Figure 17 It is a diagram showing an example of the data structure of the 3D key point table. As Figure 17 shown, the 3D key point table 142 stores the frame numbers in correspondence with the three-dimensional coordinates corresponding to each joint (joint identification information).

[0175] The frame number is the frame number for identifying the image frame used in the case of generating 3D key points. The identification information of each joint is information that uniquely determines the joint. In Figure 17 the description, joints ar0 to ar20 are used as the identification information of the joints. For example, joint ar0 corresponds to "SPINE_BASE". Joint ar1 corresponds to "SPINE_MID". Joint ar20 corresponds to "HAND_TIP_RIGHT". In Figure 2 it shows the relationship between other joints and joint names.

[0176] The three-dimensional coordinates of joints ar0 to ar20 corresponding to a certain frame number (for example, 0001) in the 3D key point table 142 are the 3D key points corresponding to a certain frame number (for example, 0001).

[0177] In the case of performing the above-mentioned "interpolation based on a machine learning model", the machine learning model 143 is used. The machine learning model 143 is a model that takes the previous frame and the next frame as inputs and outputs the paths of the coordinates of each joint from the previous frame to the next frame.

[0178] The technique identification table 144 is a table that correlates the time-series changes in the positions of each joint included in the time-series 3D key points with the types of techniques. In addition, the technique identification table 144 correlates combinations of the types of techniques with scores. The score is calculated by the sum of the D (Difficulty) score and the E (Execution) score. For example, the D score is a score calculated based on the difficulty of the technique. The E score is a score calculated by the subtraction method according to the completion degree of the technique.

[0179] The control unit 150 includes an acquisition unit 151, a 3D key point generation unit 152, a correction interval determination unit 153, a conversion processing unit 154, a correction processing unit 155, and an evaluation unit 156. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.

[0180] The acquisition unit 151 acquires image data from the camera 30 via a network. The acquisition unit 151 stores the acquired image data in the image buffer 141.

[0181] The 3D key point generation unit 152 generates time-series 3D key points based on the image data (time-series image frames) stored in the image buffer 141, and stores them in the 3D key point table 142.

[0182] For example, the process of the 3D key point generation unit 152 generating 3D key points from the image frames is the same as the process Figure 33 described in. That is, the 3D key point generation unit 152 temporarily generates 2D key points by inputting the image frames into a trained learning model, and generates 3D key points by integrating the 2D key points.

[0183] The correction interval determination unit 153 performs a process of calculating the first anomaly detection threshold Th 1 and the second anomaly detection threshold Th 2 The correction interval determination unit 153 uses the first anomaly detection threshold Th 1 to perform the first anomaly detection. The correction interval determination unit 153 uses the second anomaly detection threshold Th 2 to perform the second anomaly detection.

[0184] The process of the correction interval determination unit 153 calculating the first anomaly detection threshold Th 1 is the same as the process described above using Figure 11 The process of the correction interval determination unit 153 calculating the second anomaly detection threshold Th 2 is the same as the process described above using Figure 12 The process of the correction interval determination unit 153 calculating the second anomaly detection threshold Th

[0185] The first anomaly detection performed by the correction interval determination unit 153 is the same as the process Figure 13 described in. The second anomaly detection performed by the correction interval determination unit 153 is the same as the process described in Figure 14 or the process described in Figure 15 The process of the correction interval determination unit 153 calculating the second anomaly detection threshold Th

[0186] By performing the above process, the correction interval determination unit 153 sets a correction interval for the 3D key point sequence. The correction interval determination unit 153 outputs information on the correction interval set for the 3D key point sequence to the correction processing unit 155.

[0187] The conversion processing unit 154 converts the coordinates of each joint of the 3D key points into data of the whole body position, the whole body rotation, and the rotation of each joint. The conversion processing unit 154 outputs the conversion result to the correction processing unit 155. For example, the whole body position is the position of the entire 3D key points based on the reference pose. The whole body rotation is the rotation angle of the entire 3D key points based on the reference pose. The rotation of each joint is the rotation angle of each joint of the 3D key points based on each joint of the reference pose.

[0188] Figure 18 is a diagram showing an example of the reference pose. In Figure 18 the example shown, the reference pose is represented by nodes rn, n0 to n20. The node rn is ROOT. "ROOT" represents the node of the reference joint of the human body. The nodes n0 to n2, n4 to n12, n14 to n16, n18 are JOINTS. "JOINT" represents the node of a joint. The nodes n3, n13, n17, n19, n20 are Ends. "End" represents the node of the front end such as hands and feet. For adjacent nodes, a "joint direction vector" is set from the upper node to the lower node. The joint direction vector between each node in the reference pose is defined as "OFFSET".

[0189] Figure 19 is a diagram for explaining an example of the three-dimensional rotation angle of ROOT. For example, the three-dimensional rotation angle of ROOT is the Euler angle (θx, θy, θz) that converts the node rn-1 of ROOT in the global coordinate system to the node rn-2 of ROOT in the ROOT coordinate system.

[0190] Figure 20 is a diagram for explaining an example of the three-dimensional rotation angle of a JOINT. For example, the three-dimensional rotation angle of a JOINT is the Euler angle (θx, θy, θz) that is converted from the upper JOINT coordinate system to the concerned JOINT coordinate system. For example, if the concerned JOINT is set as the node n4, the upper node of the node n4 is the node n2. In this case, the upper JOINT coordinate system is the coordinate system of the node n2. The concerned JOINT coordinate system is the coordinate system of the node n4.

[0191] Here, since the 3D key points are three-dimensional coordinate data and there is no information on ROOT and JOINT, it is impossible to simply obtain the three-dimensional rotation angle of ROOT and the three-dimensional rotation angle of each JOINT based on the 3D key points. For example, the conversion processing unit 154 obtains the three-dimensional rotation angle of ROOT and the three-dimensional rotation angle of each JOINT based on existing technologies such as inverse kinematics according to the 3D key points.

[0192] In addition, the conversion processing unit 154 can also calculate the three-dimensional rotation angle of the ROOT by rigid body alignment and calculate the three-dimensional rotation angle of the JOINT by the Rodrigues rotation formula as follows.

[0193] An example of the process in which the conversion processing unit 154 calculates the three-dimensional rotation angle of the ROOT by rigid body alignment will be described. Figure 21 FIG. is a diagram for explaining the process of calculating the three-dimensional rotation angle of the ROOT by rigid body alignment. The conversion processing unit 154 divides each node of the hierarchical structure data (data of the reference pose) into three or more rigid body joint nodes and other nodes according to a predefined rule. As Figure 18 explained in, each node of the reference pose is node rn, n0 to n20. In Figure 21 the example shown, the conversion processing unit 154 selects nodes n0, n10, and n14 among the nodes included in the reference pose 40 as the rigid body joint nodes.

[0194] The conversion processing unit 154 determines the joint group corresponding to the rigid body joint nodes (nodes n0, n10, n14) from among the joints included in the 3D key points 41. In Figure 21 the example shown, the joint group corresponding to the rigid body joint nodes is joints ar0, ar10, and ar14. In the following description, the joint group corresponding to the rigid body joint nodes among the joints included in the 3D key points is referred to as the "rigid body corresponding joint".

[0195] The conversion processing unit 154 calculates the relative rotation angle from the rigid body joint nodes (nodes n0, n10, n14) to the rigid body corresponding joints (joints ar0, ar10, ar14) by rigid body alignment and sets it as the three-dimensional rotation angle of the ROOT.

[0196] Rigid body alignment is a method of obtaining the conversion parameters that make the source and target consistent by the least squares method according to Equation (11) for combinations of three or more points of source and target. The conversion parameters include the rotation matrix R, the translation t, and the scale c. In the present embodiment, the rotation matrix R is converted into Euler angles and used as the three-dimensional rotation angle of the ROOT.

[0197] [Mathematical formula 11]

[0198]

[0199] In Equation (11), "x" is the coordinates of the source of three or more points. The coordinates of the source are the three-dimensional coordinates of the rigid body joint. "y" is the coordinates of the target of three or more points. The coordinates of the target are the three-dimensional coordinates of the rigid body corresponding joint. The conversion processing unit 154 obtains e of Equation (11)2 The smallest rotation matrix R, translation t, and scale c.

[0200] Next, an example of the process in which the conversion processing unit 154 calculates the three-dimensional rotation angle of the JOINT using the Rodrigues rotation formula will be described. Figure 22 And Figure 23 is a diagram for explaining the process of calculating the three-dimensional rotation angle of the JOINT using the Rodrigues rotation formula. First, Figure 22 will be described. The conversion processing unit 154 converts the joint direction vector of the target joint of the 3D key point 41 into a joint direction vector in the local coordinate system. The joint direction vector of the 3D key point 41 is a vector from the lower joint to the upper joint among adjacent joints.

[0201] In Figure 22 the example shown, the target joint is set as joint ar0 for explanation. The joint direction vector v_{tar} of joint ar0 is a vector from the lower joint ar0 to the upper joint ar1.

[0202] The conversion processing unit 154 converts it into a joint direction vector in the local coordinate system by multiplying the joint direction vector of the target joint by the inverse matrix (R -1 ) of the rotation matrix R obtained using the above formula (11). For example, by multiplying the joint direction vector v_{tar} of the target joint by R -1 , the joint direction vector v_{tar_local} in the local coordinate system is obtained. The conversion processing unit 154 repeatedly executes the above process for each joint of the 3D key point 41, thereby obtaining the 3D key point 42 in the local coordinate system.

[0203] Next, the conversion processing unit 154 determines the angle θ formed by the joint direction vector of the 3D key point 42 and the joint direction vector of the reference pose 40, with the normal of the joint direction vectors as the rotation axis. Here, as an example, the joint direction vector v_{tar_local} of the 3D key point 42 and the joint direction vector v_{src} of the reference pose 40 will be used for explanation. The joint direction vector v_{src} is a vector from node n0 of the reference pose to node n1.

[0204] Proceed to Figure 23 for the explanation. In Figure 23In this case, n is the normal of the joint direction vector vtar_local and the joint direction vector vsrc of the reference pose 40. The conversion processing unit 154 determines the normal n through the cross product of the joint direction vector vtar_local and the joint direction vector vsrc. The conversion processing unit 154 sets the normal n as the rotation axis and determines the angle θ formed by the joint direction vector vtar_local and the joint direction vector vsrc.

[0205] With the normal n as the rotation axis, the conversion processing unit 154 sets the formed angle θ as the rotation angle, calculates the relative rotation angle through the Rodrigues rotation formula, and uses it as the three-dimensional rotation angle of the JOINT.

[0206] The Rodrigues rotation formula refers to the formula for calculating the rotation matrix R through Equation (12) based on the rotation axis (normal n) determined by the source and target of the joint direction vector and the rotation angle (the formed angle θ). The source of the joint direction vector is the joint direction vector (vsrc) of the reference pose 40. The target of the joint direction vector is the joint direction vector (vtar_local) of the 3D key point 42 in the local coordinate system.

[0207] [Mathematical formula 12]

[0208] R n (θ) = cosθI + (1 - cosθ)nn + sinθ[n]×...(12) t n + sinθ[n]× × …(12)

[0209] The conversion processing unit 154 uses the R(θ) obtained through Equation (12) n and converts it into Euler angles, thereby using it as the three-dimensional rotation angle of the JOINT.

[0210] As described above, the conversion processing unit 154 calculates the relative rotation angle from the reference pose to the 3D key point, thereby converting the coordinates of each joint of the 3D key point into data of whole body rotation and each joint rotation. In addition, the conversion processing unit 154 calculates the position of the entire 3D key point based on the reference pose as the whole body position. The conversion processing unit 154 outputs the conversion result to the correction processing unit 155.

[0211] Return Figure 16 to the description. The correction processing unit 155 determines the motion state of the person U4 based on the previous frame and the next frame of the correction interval set for the 3D key point sequence. The process of the correction processing unit 155 determining the motion state of the person U4 corresponds to the above "long-term motion determination", "static determination", "circular motion determination", and "circular motion direction determination".

[0212] Based on the determination result of the motion state, the correction processing unit 155 corrects the 3D key points in the correction interval. The information processing device 100 Figure 6 According to the processing steps described in, based on the determination result of the motion state, any one of interpolation based on Slerp (short path), interpolation based on Slerp (long path), and interpolation based on a machine learning model is selected for correction. The interpolation based on Slerp (short path), the interpolation based on Slerp (long path), and the interpolation based on a machine learning model executed by the correction processing unit 155 are the same as the descriptions related to the above respective interpolations.

[0213] After performing the correction corresponding to the above motion state, the correction processing unit 155 performs "apparatus contact determination" to determine whether the person U4 is in contact with the apparatus. The apparatus contact determination executed by the correction processing unit 155 is the same as Figure 7 the processing described in.

[0214] When it is determined by the apparatus contact determination that the person U4 is in contact with the apparatus, the correction processing unit 155 performs "full-body position interpolation for the correction interval based on apparatus contact". On the other hand, when it is determined by the apparatus contact determination that the person U4 is not in contact with the apparatus, the correction processing unit 155 performs "full-body position correction based on motion prediction".

[0215] The "full-body position interpolation for the correction interval based on apparatus contact" executed by the correction processing unit 155 is the same as Figure 8 the above processing described in.

[0216] The "full-body position correction based on motion prediction" executed by the correction processing unit 155 is the same as Figure 9 the above processing described in.

[0217] Based on the correction result, the correction processing unit 155 updates the 3D key points in the correction interval stored in the 3D key point table 142.

[0218] The evaluation unit 156 is a processing unit that evaluates the performance of the person U4 based on the time-series 3D key points stored in the 3D key point table 142 and the technique identification table 144. For example, the evaluation unit 156 compares the time-series changes of each joint of the 3D key points with the technique identification table 144 to determine the type of technique. In addition, the evaluation unit 157 compares the combination of the types of techniques with the technique identification table to calculate the score of the performance of the person U4. The evaluation unit 156 generates screen information of the type of technique and the score and displays it on the display unit 130.

[0219] Next, an example of the processing steps of the information processing device 100 in this embodiment will be described. Figure 24This is a flowchart showing the processing steps of the information processing apparatus 100 according to the present embodiment. As Figure 24 shown, the acquisition unit 151 of the information processing apparatus 100 acquires image data from the camera 30 and stores it in the image buffer 141 (step S201).

[0220] The 3D key point generation unit 152 of the information processing apparatus 100 generates a 3D key point sequence based on the image frames of the time series of the image data (step S202). The correction interval determination unit 153 of the information processing apparatus 100 performs correction interval determination processing (step S203).

[0221] The information processing apparatus 100 determines the previous frame and the subsequent frame of the correction interval (step S204). The conversion processing unit 154 of the information processing apparatus 100 calculates the whole body position, the whole body rotation, and the rotation of each joint based on the 3D key points (step S205).

[0222] The correction processing unit 155 of the information processing apparatus 100 performs correction processing (step S206). The correction processing unit 155 updates the 3D key points in the correction interval (step S207). The evaluation unit 156 of the information processing apparatus 100 evaluates based on the 3D key points and outputs the evaluation result to the display unit 130 (step S208).

[0223] Next, the processing steps of the correction interval determination processing described in step S203 of Figure 24 will be described. Figure 25 This is a flowchart showing the processing steps of the correction interval determination processing. As Figure 25 shown, the correction interval determination unit 153 of the information processing apparatus 100 acquires the 3D key point sequence (step S301).

[0224] The correction interval determination unit 153 calculates the abnormal features for each frame of the 3D key point sequence (step S302). The correction interval determination unit 153 performs the first abnormal detection to determine the abnormal interval (step S303).

[0225] The correction interval determination unit 153 performs the second abnormal detection to optimize the abnormal interval (step S304). The correction interval determination unit 153 sets the optimized abnormal interval as the correction interval (step S305).

[0226] Next, the processing steps of the correction processing described in step S206 of Figure 24 will be described. Figure 26 This is a flowchart showing the processing steps of the correction processing. As Figure 26 shown, the correction processing unit 155 of the information processing apparatus 100 determines the motion state based on the 3D key points, the whole body position, the whole body rotation, and the rotation of each joint (step S401).

[0227] The correction processing unit 155 performs interpolation processing corresponding to the motion state (step S402). The correction processing unit 155 performs correction processing based on the appliance contact (step S403). The correction processing unit 155 outputs a correction result (step S404).

[0228] Figure 26 The processing steps of the interpolation processing corresponding to the motion state shown in step S402 correspond to Figure 6 the processing steps described in

[0229] Next, the processing steps of the correction processing based on the Figure 26 appliance contact described in step S403 are described. Figure 27 is a flowchart showing the processing steps of the correction processing based on the appliance contact. As Figure 27 shown, the correction processing unit 155 of the information processing apparatus 100 determines the motion state based on the 3D key points, the whole body position, the whole body rotation, and the rotation of each joint (step S501).

[0230] The correction processing unit 155 performs appliance contact determination (step S502). When a person is in contact with the appliance (step S503, yes), the correction processing unit 155 interpolates the whole body position based on the appliance center prediction (step S504), and proceeds to step S506.

[0231] On the other hand, when a person is not in contact with the appliance (step S503, no), the correction processing unit 155 interpolates the whole body position based on the motion prediction (step S505). The correction processing unit 155 outputs an interpolation result (step S506).

[0232] Next, an example of the processing steps for setting the first abnormality detection threshold Th 1 and the second abnormality detection threshold Th 2 in the information processing apparatus 100 is described. Figure 28 is a flowchart showing the processing steps of setting the threshold. As Figure 28 shown, the correction range determination unit 153 of the information processing apparatus 100 acquires a plurality of normal 3D key points and a plurality of abnormal 3D key points (step S601).

[0233] The correction range determination unit 153 calculates the abnormality feature of each 3D key point (step S602).

[0234] The correction interval determination unit 153 sets the boundary value with the maximum separation between the distribution of normal 3D key points and the distribution of abnormal 3D key points as the first abnormal detection threshold (step S603). The correction interval determination unit 153 sets the start position of the search range of the threshold as the second abnormal detection threshold (step S604).

[0235] Next, the effects of the information processing apparatus 100 of the present embodiment will be described. The information processing apparatus 100 sets a correction interval for the 3D key point sequence, and determines the motion state of the person U4 based on the 3D key points before and after the correction interval. The information processing apparatus 100 determines whether the person U4 is in contact with the implement based on the 3D key points before and after the correction interval. The information processing apparatus 100 corrects the 3D key points in the correction interval based on the determination result of the motion state of the person U4 and whether the person U4 is in contact with the implement. Thereby, the recognition accuracy of the bones of the person can be improved.

[0236] Figure 29 、 Figure 30 is a diagram for explaining the effects of the information processing apparatus. Figure 29 will be described. The 3D key point sequence 61 includes frames 61-1, 61-2, 61-3, 61-4, 61-5, 61-6. The 3D key point sequence 62 includes frames 62-1, 62-2, 62-3, 62-4, 62-5, 62-6.

[0237] The correction interval 61a of the 3D key point sequence 61 is corrected by the prior art. In the prior art, based on the frames 61-1, 61-6, an incorrect path (short path) is selected, and correction based on Slerp is performed. Therefore, the frames 61-2 to 61-5 in the correction interval 61a are not properly corrected.

[0238] On the other hand, the correction interval 62a of the 3D key point sequence 62 is corrected by the information processing apparatus 100 of the present invention. The information processing apparatus 100 selects an appropriate path (long path) based on the frames 62-1, 62-6, performs correction based on Slerp, and properly corrects the frames 62-2 to 62-5 in the correction interval 62a.

[0239] Transfer to Figure 30 description. The 3D key point sequence 63 includes frames 63-1, 63-2, 63-3, 63-4, 63-5. The 3D key point sequence 64 includes frames 64-1, 64-2, 64-3, 64-4, 64-5.

[0240] The correction interval 63a of the 3D key point sequence 63 is corrected by the prior art. In the prior art, in the correction interval 63a, the joint positions of the person in the frames 63-2 to 63-4 are far from the implement 63b and are not properly corrected.

[0241] On the other hand, the correction interval 64a of the 3D key point sequence 64 is corrected by the information processing device 100 of the present invention. When a person comes into contact with the implement 64b, the information processing device 100 performs correction through implement center prediction, and the joint positions of the person in each frame 64-2 to 64-4 of the correction interval 64a that come into contact with the implement 64b are appropriately corrected.

[0242] In addition, the information processing device 100 acquires in advance the distribution of normal 3D key points and the distribution of abnormal 3D key points, and determines the first anomaly detection threshold Th 1 and the second anomaly detection threshold Th 2 . The first anomaly detection threshold Th 1 is the boundary value with the largest separation degree between the distribution of normal 3D key points and the distribution of abnormal 3D key points. The second anomaly detection threshold Th 2 is a threshold with a higher sensitivity than the first anomaly detection threshold Th 1 . The information processing device 100 sets a correction interval for the 3D key point sequence based on the first anomaly detection threshold Th 1 and the second anomaly detection threshold Th 2 . Thus, the interval with abnormal coordinates of the joints of the 3D key points can be appropriately set as the correction interval.

[0243] Figure 31 is a diagram for explaining the effect of the information processing device. The 3D key point sequence 65 includes frames 65-1, 65-2, 65-3, 65-4, 65-5, 65-6. The 3D key point sequence 66 includes frames 66-1, 66-2, 66-3, 66-4, 66-5, 66-6.

[0244] In the prior art, for the 3D key point sequence 65, the correction interval 65a is set by simple threshold comparison, and the correction of the correction interval 65a is performed based on the front and rear frames (frames 65-2, 65-5) of the correction interval 65a. Here, the frames 65-2 and 65-5 sometimes become frames in which abnormal features are not suppressed. If the correction is performed based on the frames 65-2 and 65-5 in which abnormal features are not suppressed, the accuracy of the correction result is reduced.

[0245] On the other hand, for the 3D key point sequence 66, the information processing device 100 performs two-stage anomaly detection based on the anomaly detection threshold Th 1 and the second anomaly detection threshold Th 2 , and sets the correction interval 66a. The front and rear frames (frames 66-1, 66-6) of the correction interval 66a are frames in which abnormal features are suppressed. If the information processing device 100 performs correction based on the frames 66-1 and 66-6 in which abnormal features are suppressed, the reduction in the accuracy of the correction result can be suppressed.

[0246] Next, an example of the hardware configuration of a computer that implements the same functions as the above-described information processing apparatus 100 will be described. Figure 32 FIG. is a diagram showing an example of the hardware configuration of a computer that implements the same functions as the information processing apparatus of the embodiment.

[0247] As Figure 32 shown, the computer 200 includes: a CPU 201 that performs various arithmetic processes, an input device 202 that accepts input of data from a user, and a display 203. In addition, the computer 200 includes a communication device 204 that exchanges data with the camera 15, an external device, etc. via a wired or wireless network, and an interface device 205. In addition, the computer 200 includes a RAM 206 that temporarily stores various information, and a hard disk device 207. Moreover, each device 201 to 207 is connected to a bus 208.

[0248] The hard disk device 207 includes an acquisition program 207a, a 3D key point generation program 207b, a correction interval determination program 207c, a conversion processing program 207d, a correction processing program 207e, and an evaluation program 207f. In addition, the CPU 201 reads out each program 207a to 207f and expands them in the RAM 206.

[0249] The acquisition program 207a functions as an acquisition step 206a. The 3D key point generation program 207b functions as a 3D key point generation step 206b. The correction interval determination program 207c functions as a correction interval determination step 206c. The conversion processing program 207d functions as a conversion processing step 206d. The correction processing program 207e functions as a correction processing step 206e. The evaluation program 207f functions as an evaluation step 206f.

[0250] The process of the acquisition step 206a corresponds to the process of the acquisition unit 151. The process of the 3D key point generation step 206b corresponds to the process of the 3D key point generation unit 152. The process of the correction interval determination step 206c corresponds to the process of the correction interval determination unit 153. The process of the conversion processing step 206d corresponds to the process of the conversion processing unit 154. The process of the correction processing step 206e corresponds to the process of the correction processing unit 155. The process of the evaluation step 206f corresponds to the process of the evaluation unit 156.

[0251] In addition, for each of the programs 207a to 207f, it is not necessarily stored in the hard disk device 207 from the beginning. For example, each program is stored in a "removable physical medium" such as a floppy disk (FD), CD-ROM, DVD, optical disk, or IC card inserted into the computer 200. Further, the computer 200 can also read and execute each of the programs 207a to 207f.

[0252] Description of Reference Numerals

[0253] 100... Information processing device; 110... Communication unit; 120... Input unit; 130... Display unit; 140... Storage unit; 141... Image buffer; 142... 3D key point table; 143... Machine learning model; 144... Technology identification table; 150... Control unit; 151... Acquisition unit; 152... 3D key point generation unit; 153... Correction range determination unit; 154... Conversion processing unit; 155... Correction processing unit; 156... Evaluation unit.

Claims

1. An information processing method, characterized in that, the computer performs the following processing: Based on a plurality of bone information respectively setting a plurality of joints included in the human body and coordinates of the plurality of joints, an abnormal feature is calculated, Based on a first distribution of normal bone information related to the abnormal feature and a second distribution of abnormal bone information related to the abnormal feature among the plurality of bone information, a boundary value with the largest separation degree between the first distribution and the second distribution is calculated as a first threshold, Based on the first threshold, a correction range of the time series bone information is set.

2. The information processing method according to claim 1, characterized in that, the computer further performs the following processing: Calculating the minimum abnormal feature or the maximum abnormal feature in the search range representing the range of the abnormal feature as a second threshold, and using the second threshold to correct the correction range, wherein the search range is a range with a high possibility of the largest separation degree between the first distribution and the second distribution.

3. The information processing method according to claim 1, characterized in that, In the process of calculating the abnormal feature, based on the difference between a reference value determined from the bone lengths of the plurality of bone information and the bone lengths of the bone information, the abnormal feature of the bone information is calculated.

4. The information processing method according to claim 2, characterized in that, The correction range set for the time series bone information is further corrected.

5. An information processing method, characterized in that, the computer performs the following processing: Based on a plurality of bone information respectively setting a plurality of joints included in the human body and coordinates of the plurality of joints, an abnormal feature is calculated, Based on the minimum abnormal feature in the search range representing the range of the abnormal feature, the correction range set for the time series bone information is corrected by a certain threshold, wherein the search range is a range with a high possibility of the largest separation degree between a first distribution of normal bone information related to the abnormal feature and a second distribution of abnormal bone information related to the abnormal feature among the plurality of bone information.

6. An information processing program, characterized in that, causing the computer to perform the following processing: Based on a plurality of bone information respectively setting a plurality of joints included in the human body and coordinates of the plurality of joints, an abnormal feature is calculated, Based on a first distribution of normal bone information related to the abnormal feature and a second distribution of abnormal bone information related to the abnormal feature among the plurality of bone information, a boundary value with the largest separation degree between the first distribution and the second distribution is calculated as a first threshold, Based on the first threshold, a correction range of the time series bone information is set.

7. The information processing program according to claim 6, characterized in that, the computer further performs the following processing: Calculating the minimum abnormal feature or the maximum abnormal feature in the search range representing the range of the abnormal feature as a second threshold, and using the second threshold to correct the correction range, wherein the search range is a range with a high possibility of the largest separation degree between the first distribution and the second distribution.

8. The information processing program according to claim 6, wherein, in the process of calculating the abnormal feature, the abnormal feature of the skeletal information is calculated based on the difference between the reference value determined from the bone lengths of the multiple skeletal information and the bone length of the skeletal information.

9. The information processing program according to claim 7, wherein, correction is also performed on the correction range set for the time-series skeletal information.

10. An information processing program, wherein, it causes a computer to execute the following processing: calculate an abnormal feature based on multiple skeletal information respectively setting a plurality of joints included in a human body and coordinates of the multiple joints, correct the correction range set for the time-series skeletal information by a certain threshold based on the minimum abnormal feature in a search range representing the range of the abnormal feature, wherein the search range is a range with a high possibility of the largest separation between a first distribution of normal skeletal information related to the abnormal feature and a second distribution of abnormal skeletal information related to the abnormal feature among the multiple skeletal information.

11. An information processing device, wherein, it has a control unit, and the control unit executes the following processing: calculate an abnormal feature based on multiple skeletal information respectively setting a plurality of joints included in a human body and coordinates of the multiple joints, calculate a boundary value with the largest separation between the first distribution and the second distribution as a first threshold based on the first distribution of normal skeletal information related to the abnormal feature and the second distribution of abnormal skeletal information related to the abnormal feature among the multiple skeletal information, set the correction range of the time-series skeletal information based on the first threshold.

12. The information processing device according to claim 11, wherein, the control unit also executes the following processing: calculate the minimum abnormal feature or the maximum abnormal feature in a search range representing the range of the abnormal feature as a second threshold, and correct the correction range using the second threshold, wherein the search range is a range with a high possibility of the largest separation between the first distribution and the second distribution.

13. The information processing device according to claim 11, wherein, in the process of calculating the abnormal feature, the abnormal feature of the skeletal information is calculated based on the difference between the reference value determined from the bone lengths of the multiple skeletal information and the bone length of the skeletal information.

14. The information processing device according to claim 12, wherein, correction is also performed on the correction range set for the time-series skeletal information.

15. An information processing device, wherein, it has a control unit, and the control unit executes the following processing: calculate an abnormal feature based on multiple skeletal information respectively setting a plurality of joints included in a human body and coordinates of the multiple joints, Based on the minimum abnormal feature in the search range representing the range of the above abnormal features, a correction range set for the skeletal information of the time series is corrected by a certain threshold, where the above search range is a range with a high possibility of the largest separation between the first distribution of the normal skeletal information related to the above abnormal features and the second distribution of the abnormal skeletal information related to the above abnormal features among the above multiple skeletal information.

Images