Information processing method, information processing program, and information processing device

Abstract

This information processing device calculates an abnormal feature on the basis of multiple pieces of skeleton information in which multiple joints of a human and coordinates corresponding to the multiple joints are respectively set. The information processing device refers to a first distribution pertaining to normal skeleton information on the abnormal feature and a second distribution pertaining to abnormal skeleton information on the abnormal feature among the multiple pieces of skeleton information to calculate, as a first threshold, a border value at which the degree of separation between the first distribution and the second distribution is maximized. On the basis of the first threshold, the information processing device sets a correction range for skeleton information in a time series.

Description

Information processing method, information processing program, and information processing device

[0001] The present invention relates to an information processing method and the like.

[0002] There is a conventional technique for capturing images of a person using multiple cameras and performing 3D skeletal recognition in relation to various sports.

[0003] Fig. 33 is a diagram for explaining a conventional technique for 3D skeleton recognition. In the example shown in Fig. 33, cameras 30a and 30b capture images of a person U1. The images captured by cameras 30a and 30b are designated as images 31a and 31b, respectively.

[0004] The images 31a and 31b are input to learning models 32a and 32b, respectively, which output 2D keypoints 33a and 33b. The learning models 32a and 32b are trained deep learning models or the like. The 2D keypoints 33a and 33b are two-dimensional skeletal information or the like.

[0005] In the prior art, 2D key points 33 a and 33 b are integrated to generate 3D key points 34. The 3D key points 34 are three-dimensional skeletal information, etc. For example, the three-dimensional coordinates of each joint of a human body model are set in the 3D key points 34.

[0006] In the prior art, the above process is repeatedly performed on the time-series images captured by the cameras 30a and 30b to generate time-series 3D keypoints. The time-series 3D keypoints are the result of 3D skeleton recognition. While the explanation in FIG. 33 uses the cameras 30a and 30b, other cameras may also be used to generate time-series 3D keypoints.

[0007] When performing 3D skeleton recognition on a person performing gymnastics, an assistant other than the target person may appear in front of the camera, which may result in a disturbance in the 3D key points.

[0008] FIG. 34 shows an example of poor observation. Images 35a, 35b, 35c, and 35d in FIG. 34 were captured by cameras at different positions. In images 35a to 35d, the person targeted for 3D skeleton recognition is person U2. For example, image 35b includes person U3 in addition to person U2. Under these circumstances, a 3D keypoint sequence 10 is generated by generating a 3D keypoint sequence based on images 35a to 35d and the subsequent images. In 3D keypoint sequence 10, disturbances occur in 3D keypoints 10-2, 10-3, and 10-4 out of 3D keypoints 10-1, 10-2, 10-3, 10-4, and 10-5.

[0009] To ensure the accuracy of 3D skeleton recognition, a section where the 3D key points are disturbed is identified as a correction section, and the 3D key points in the correction section are corrected based on the movement of the 3D key points before and after the correction section.

[0010] FIG. 35 is a diagram illustrating a correction method of the related art. The example shown in FIG. 35 will be described using a 3D keypoint sequence 10. The 3D keypoint sequence 10 includes 3D keypoints 10-1 to 10-5. In the related art, anomaly detection is performed on the 3D keypoint sequence 10 to detect disturbed 3D keypoints 10-2, 10-3, and 10-4. The section of the disturbed 3D keypoints 10-2, 10-3, and 10-4 is designated as a correction section 10a. In the related art, the 3D keypoints 10-2 to 10-4 are corrected using the 3D keypoints 10-1 and 10-5 before and after the correction section 10a.

[0011] Conventional techniques for correcting 3D key points in a correction section based on 3D key points before and after the correction section include spherical liner interpolation (Slerp) and linear interpolation (Lerp). FIG. 36 is a diagram illustrating Slerp and Lerp. Slerp is spherical linear interpolation, and Lerp is linear interpolation. Spherical linear interpolation is a correction method for two distant joints, assuming that the space between the joints is a sphere (rotation). Linear interpolation is a correction method for two distant joints, assuming that the space between the joints is a straight line. For example, in the example described with reference to FIG. 35, the optimal positions of each joint are identified based on the joint positions, Slerp, and Lerp of 3D key points 10-1 and 10-5 before and after the correction section 10a, and 3D key points 10-2 to 10-4 are corrected.

[0012] Japanese Patent Application Laid-Open No. 2021-135877

[0013] However, the above-described conventional technology has a problem in that it is not possible to appropriately set the correction interval, and it is not possible to improve the accuracy of recognizing a person's skeleton.

[0014] For example, if the correction interval is not set appropriately, the coordinates of each joint of the 3D key points before and after the correction interval may not be appropriate, making it impossible to perform appropriate correction.

[0015] In one aspect, the present invention aims to provide an information processing method, an information processing program, and an information processing device that can appropriately set a correction section and improve the accuracy of recognizing a person's skeleton.

[0016] In the first proposal, the computer executes the following process: the computer calculates abnormal features based on a plurality of skeletal information sets in which a plurality of joints included in the human body and coordinates for the plurality of joints are respectively set; the computer calculates, as a first threshold, a boundary value that maximizes the degree of separation between the first distribution and the second distribution, based on a first distribution of normal skeletal information related to the abnormal features and a second distribution of abnormal skeletal information related to the abnormal features, among the plurality of skeletal information sets; the computer sets a correction range for the time-series skeletal information based on the first threshold;

[0017] By appropriately setting the correction interval, it is possible to improve the accuracy of recognizing a person's skeleton.

[0018] FIG. 1 is a diagram illustrating an example of a human body model. FIG. 2 is a diagram illustrating an example of joint names. FIG. 3 is a diagram illustrating a system according to this embodiment. FIG. 4 is a diagram illustrating a still state determination. FIG. 5 is a diagram illustrating a circular motion determination. FIG. 6 is a flowchart illustrating a processing procedure for a correction process according to a motion state according to this embodiment. FIG. 7 is a diagram illustrating an instrument contact determination. FIG. 8 is a diagram illustrating whole body position interpolation of a correction section based on instrument contact. FIG. 9 is a diagram illustrating an example of a motion prediction result. FIG. 10 is a diagram illustrating an example of a distribution of 3D key points related to abnormal features. FIG. 11 is a diagram illustrating a process for calculating a first abnormality detection threshold. FIG. 12 is a diagram illustrating a process for calculating a second abnormality detection threshold. FIG. 13 is a diagram illustrating first abnormality detection. FIG. 14 is a diagram illustrating an example of optimization of an abnormal section based on a first convergence condition. FIG. 15 is a diagram illustrating an example of optimization of an abnormal section based on a second convergence condition. FIG. 16 is a functional block diagram illustrating a configuration of an information processing device according to this embodiment. FIG. 17 is a diagram illustrating an example of a data structure of a 3D key point table. FIG. 18 is a diagram illustrating an example of a reference posture. FIG. 19 is a diagram illustrating an example of a three-dimensional rotation angle of a ROOT. FIG. 20 is a diagram illustrating an example of a three-dimensional rotation angle of a JOINT. FIG. 21 is a diagram illustrating a process for calculating a three-dimensional rotation angle of a ROOT by rigid body alignment. FIG. 22 is a diagram (1) illustrating a process for calculating a three-dimensional rotation angle of a JOINT using Rodrigues' rotation formula. FIG. 23 is a diagram (2) illustrating a process for calculating a three-dimensional rotation angle of a JOINT using Rodrigues' rotation formula. FIG. 24 is a flowchart illustrating a processing procedure of an information processing device according to this embodiment. FIG. 25 is a flowchart illustrating a processing procedure for correction section identification processing. FIG. 26 is a flowchart illustrating a processing procedure for correction processing. FIG. 27 is a flowchart illustrating a processing procedure for correction processing based on tool contact. FIG. 28 is a flowchart illustrating a processing procedure for threshold setting processing. FIG. 29 is a diagram (1) illustrating an effect of the information processing device. FIG. 30 is a diagram (2) illustrating an effect of the information processing device. FIG. 31 is a diagram (3) illustrating an effect of the information processing device.Fig. 32 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment. Fig. 33 is a diagram for explaining a conventional technique related to 3D skeleton recognition. Fig. 34 is a diagram showing an example of poor observation. Fig. 35 is a diagram for explaining a correction method of the conventional technique. Fig. 36 is a diagram for explaining Slerp and Lerp.

[0019] Hereinafter, an information processing method, an information processing program, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments.

[0020] Before describing this embodiment, an example of a human body model will be described. Fig. 1 is a diagram showing an example of a human body model. As shown in Fig. 1, the human body model is defined by 21 joints ar0 to ar20.

[0021] The relationship between the joint names and the joints ar0 to ar20 shown in Fig. 1 is as shown in Fig. 2. Fig. 2 is a diagram showing examples of joint names. For example, the joint name of joint ar0 is "SPINE_BASE". The joint names of joints ar1 to ar20 are as shown in Fig. 2, and explanations thereof will be omitted.

[0022] The 3D keypoints handled in this embodiment are data in which three-dimensional coordinates corresponding to the joints ar0 to ar20 of the human body model are set. In the following explanation, the 3D keypoints arranged in chronological order will be referred to as a "3D keypoint sequence."

[0023] Next, an example of a system according to this embodiment will be described. Fig. 3 is a diagram showing the system according to this embodiment. As shown in Fig. 3, the system according to this embodiment includes cameras 30a and 30b and an information processing device 100. The cameras 30a and 30b and the information processing device 100 are connected to each other wirelessly or via a wire.

[0024] The cameras 30a and 30b capture images of a person U4. As an example, the person U4 is playing a sport using equipment. In the following description, the cameras 30a and 30b are collectively referred to as the camera 30.

[0025] The camera 30 transmits data of the captured video to the information processing device 100. In the following description, the video data will be referred to as "video data." The video data includes image frames in time series. Each image frame is assigned a frame number in ascending order. An image frame is data of a still image captured by the camera 30 at a certain timing.

[0026] The information processing device 100 generates a 3D keypoint sequence based on the video data received from the camera 30. The information processing device 100 sets a correction interval in the 3D keypoint sequence and determines the motion state of the person U4 based on the 3D keypoints before and after the correction interval. The information processing device 100 determines whether the person U4 is in contact with the tool based on the 3D keypoints before and after the correction interval. The information processing device 100 corrects the 3D keypoints in the correction interval based on the motion state of the person U4 and the determination result of whether the person U4 is in contact with the tool.

[0027] The information processing device 100 acquires the distribution of normal 3D keypoints and the distribution of abnormal 3D keypoints in advance, and calculates the first abnormality detection threshold Th １ and the second abnormality detection threshold Th ２ The first abnormality detection threshold Th １ is a boundary value at which the degree of separation between the normal 3D keypoint distribution and the abnormal 3D keypoint distribution is maximized. ２ is the first abnormality detection threshold Th １ The information processing device 100 uses the first anomaly detection threshold Th １ and the second abnormality detection threshold Th ２ Based on this, a correction interval is set for the 3D keypoint sequence.

[0028] An example of a process for determining a motion state by the information processing device 100 will be described below. The process for determining a motion state includes "long-term motion determination," "stillness determination," "circular motion determination," and "circular motion direction determination." In the following description, 3D key points will be referred to as "frames" where appropriate.

[0029] The "long-term motion determination" executed by the information processing device 100 will be described. The information processing device 100 specifies the number of 3D key points (hereinafter referred to as the number of frames) in the correction section set in the 3D key point sequence. The information processing device 100 determines whether the number of frames is greater than or equal to a threshold Th ３ On the other hand, when the number of frames is equal to or exceeds the threshold value Th ３ If the threshold value Th is less than the threshold value Th, the exercise state is determined not to be long-term exercise. ３ is set in advance.

[0030] The "stillness determination" executed by the information processing device 100 will be described. The information processing device 100 identifies the 3D key points before and after the correction interval set in the 3D key point sequence. In the following description, the 3D key points before the correction interval will be referred to as the "previous frame." The 3D key points after the correction interval will be referred to as the "next frame." For example, if the correction interval is the correction interval 10a described in FIG. 35, the previous frame will be the 3D key point 10-1, and the subsequent frame will be the 3D key point 10-5. The information processing device 100 determines whether the difference in whole body rotation from the previous frame to the subsequent frame is greater than or equal to a threshold Th ４ On the other hand, the information processing device 100 determines that the motion state is static motion when the difference in the whole body rotation from the previous frame to the next frame is less than the threshold Th ４ In the above cases, it is determined that the motion state is not stationary motion.

[0031] FIG. 4 is a diagram for explaining stillness determination. In FIG. 4, the previous frame is referred to as a "previous frame 14s" and the subsequent frame is referred to as a "subsequent frame 14e." The reference line of the previous frame 14s is referred to as a "reference line 14s-1." The reference line of the subsequent frame 14e is referred to as a "reference line 14e-1." The information processing device 100 determines an angle 14-1 formed between the reference line 14s-1 and the reference line 14e-1 as the difference in whole-body rotation from the previous frame 14s to the subsequent frame 14e. The information processing device 100 determines whether the formed angle 14-1 is greater than or equal to a threshold Th ４ If the angle 14-1 is less than the threshold value Th ４In the above cases, it is determined that the motion state is not stationary motion.

[0032] The "circular motion determination" executed by the information processing device 100 will be described. The information processing device 100 determines the quaternion q start and the quaternion q of the frame (3D key point) one frame before the previous frame start-1 Based on this, the predicted rotation axis v pred Here, the quaternion of a certain frame indicates the rotation from the coordinates of the joints in a certain frame to the coordinates of the joints in the reference posture. The reference posture is data defined by multiple nodes corresponding to the joints of the human body, and the coordinates of each node are set in advance. The predicted rotation axis v pred is the quaternion q start-1 to quaternion q start Relative quaternion q representing the relative rotation to -1 start-1 *q start corresponds to the normal of

[0033] The information processing device 100 calculates the quaternion q of the previous frame of the correction interval. start and the quaternion q of the frame after the correction interval end Based on and, the quaternion q of the shortest path short _ end and the quaternion q of the short path long _ end The information processing device 100 calculates the quaternion q based on the formula (1). short _ end The information processing device 100 calculates the quaternion q based on the formula (2). long _ end Calculate.

[0034]

[0035]

[0036] The information processing device 100 calculates the quaternion q start and the quaternion q short _ end Based on this, the rotation axis v in the short path short Calculate the rotation axis vshort is the quaternion q start to quaternion q short-end Relative quaternion q representing the relative rotation to -1 start *q short-end corresponds to the normal of

[0037] The information processing device 100 calculates the quaternion q start and the quaternion q long _ end Based on this, the rotation axis v in the long path long Calculate the rotation axis v long is the quaternion q start to quaternion q long-end Relative quaternion q representing the relative rotation to -1 start *q long-end corresponds to the normal of

[0038] The information processing device 100 calculates the predicted rotation axis v pred and v short Furthermore, the information processing device 100 calculates the first cosine similarity between the predicted rotation axis v pred and v long The information processing apparatus 100 calculates a second cosine similarity between the first cosine similarity and the second cosine similarity. ５ On the other hand, the information processing device 100 determines that the movement state is circular movement when the first cosine similarity and the second cosine similarity are equal to or greater than the threshold Th ５ If it is less than this, it is determined that the motion state is not circular motion.

[0039] FIG. 5 is a diagram for explaining circular motion determination. In FIG. 5, the previous frame of the correction interval is frame 15s. The frame immediately preceding frame 15s is frame 15s'. The subsequent frame is frame 15e. The information processing device 100 calculates the quaternion q of frame 15s in the correction interval. start and the quaternion q of frame 15s' start-1 Based on this, the predicted rotation axis v pred Calculate.

[0040] The information processing device 100 calculates a quaternion q start and the quaternion q short _ end Based on this, the rotation axis v in the short path short The information processing device 100 calculates the quaternion q start and the quaternion q long _ end Based on this, the rotation axis v in the long path long For example, the information processing device 100 calculates the predicted rotation axis v pred and q short _ end The information processing device 100 calculates the first cosine similarity (0.2) between the predicted rotation axis v pred and q long _ end The information processing device 100 calculates the second cos similarity (0.8) with the threshold value Th ５ If the second cos similarity is set to "0.5", the threshold Th ５ As a result, the state of motion is determined to be rotational motion.

[0041] The "circular motion direction determination" executed by the information processing device 100 will be described. When the motion state is circular motion, the information processing device 100 performs the circular motion method determination. The information processing device 100 determines whether the first cos similarity of the first cos similarity is greater than or equal to a threshold Th ５ On the other hand, the information processing device 100 determines that the second cos similarity is equal to or greater than the threshold Th ５ If so, the direction of the longer path is determined to be the direction of circular motion.

[0042] In the example described with reference to FIG. 5, the second cos similarity is ５ As a result, the direction of the longer path is determined as the direction of the circular motion.

[0043] The information processing device 100 executes the above process to determine the motion state of the person U4. Based on the determination result of the motion state, the information processing device 100 corrects the 3D key points in the correction section. The information processing device 100 selects one of Slerp (short path) interpolation, Slerp (long path) interpolation, and interpolation using a machine learning model to perform the correction.

[0044] For example, the information processing device 100 selects one of Slerp (shorter path) interpolation, Slerp (longer path) interpolation, and machine learning model interpolation based on the processing procedure of FIG. 6 to perform correction.

[0045] 6 is a flowchart showing the processing procedure of the correction process according to the motion state according to this embodiment. As shown in FIG. 6, the information processing device 100 receives input of 3D key points, whole body position, whole body rotation, and each joint rotation (step S101). The information processing device 100 then performs long-term motion determination (step S102).

[0046] If the exercise state is long-term exercise (step S103, Yes), the information processing device 100 executes interpolation using a machine learning model (step S104) and proceeds to step S113.

[0047] On the other hand, if the motion state is not long-term motion (step S103, No), the information processing device 100 performs stationary motion determination (step S105). If the motion state is stationary motion (step S106, Yes), the information processing device 100 performs Slerp (shorter path) interpolation (step S107) and proceeds to step S113.

[0048] On the other hand, if the motion state is not stationary motion (step S106, No), the information processing device 100 performs a circular motion determination (step S108).If the motion state is not circular motion (step S109, No), the information processing device 100 proceeds to step S107.

[0049] On the other hand, if the motion state is circular motion (step S109, Yes), the information processing device 100 determines the direction of the circular motion (step S110). If the direction of the circular motion is not the same as the long path (step S111, No), the information processing device 100 proceeds to step S107. Note that the same direction as the long path means the same direction as the short path.

[0050] On the other hand, if the direction of the circular motion is the same as the long path (step S111, Yes), the information processing device 100 performs interpolation using Slerp (long path) (step S112) and outputs the interpolation result (step S113).

[0051] Next, the interpolation by Slerp (shorter path), the interpolation by Slerp (longer path), and the interpolation by machine learning model, which are executed by the information processing device 100, will be described.

[0052] The "Slerp (shorter path) interpolation" performed by the information processing device 100 will be described. The information processing device 100 performs Slerp (shorter path) interpolation based on equation (3). In equation (3), "t" indicates a time index and is defined as "t = n / N". "n" indicates an index. "N" indicates the number of frames in the interpolation section. "θ" is a quaternion q start and the quaternion q short _ end Indicates the angle between

[0053]

[0054] The information processing device 100 calculates the quaternion q at time t. ｔ is calculated based on Equation (3), and the coordinates of the joints of the 3D key points corresponding to time t in the correction interval are calculated as quaternion q ｔ Correct it accordingly.

[0055] The "Slerp (long path) interpolation" performed by the information processing device 100 will be described. The information processing device 100 performs Slerp (long path) interpolation based on equation (4). In equation (4), "t" indicates a time index and is defined as "t = n / N". "n" indicates an index. "N" indicates the number of frames in the interpolation section. "θ" indicates the quaternion q start and the quaternion q long _ end Indicates the angle between

[0056]

[0057] The information processing device 100 calculates the quaternion q at time t. ｔ is calculated based on Equation (4), and the coordinates of the joints of the 3D key points corresponding to time t in the correction interval are calculated as quaternion q ｔ Correct it accordingly.

[0058] The "interpolation using a machine learning model" performed by the information processing device 100 will be described. For example, the information processing device 100 uses a trained machine learning model that receives the previous frame and the next frame as input and outputs the path of the coordinates of each joint from the previous frame to the next frame. The machine learning model is a neural network (NN) or the like. The information processing device 100 corrects the 3D key points in the correction section based on the output result of the machine learning model.

[0059] In addition, the information processing device 100 may interpolate 3D keypoints in the correction section using the machine learning model described in the non-patent document “Felix G. Harvey, et al. “Robust Motion In-betweening”, In ACM Transactions on Graphics (TOG), 2020.”

[0060] Here, after performing the correction according to the motion state, the information processing device 100 performs an "appliance contact determination" to determine whether or not the person U4 is in contact with the appliance. If the information processing device 100 determines through the appliance contact determination that the person U4 is in contact with the appliance, it performs "whole-body position interpolation of the correction section based on appliance contact." On the other hand, if the information processing device 100 determines through the appliance contact determination that the person U4 is not in contact with the appliance, it performs "whole-body position correction based on motion prediction."

[0061] The "tool contact determination" executed by the information processing device 100 will be described. FIG. 7 is a diagram for explaining the tool contact determination. The coordinates of the tool eq1, the joint to be determined (determination joint), the threshold value Th ６ are assumed to be defined in advance. The coordinates of the instrument eq1 can be defined as a point using n-dimensional coordinates (= 1, 2, 3), a plane using multiple n-dimensional coordinates, or the like. For example, the information processing device 100 defines a point using two-dimensional coordinates by projecting the three-dimensional coordinates of the instrument eq1 onto the yz plane. The joints to be determined are the joint at the tip of the left hand and the joint at the tip of the right hand. In the human body model described in FIG. 1, the joint at the tip of the left hand is joint ar19. The joint at the tip of the right hand is joint ar20.

[0062] The information processing device 100 selects one joint from the plurality of determination joints as a "joint of interest" and calculates the distance d between the joint of interest and the instrument eq1. The information processing device 100 executes the above process for all determination joints in the previous and subsequent frames of the correction section. The information processing device 100 performs the above process for all determination joints in the previous and subsequent frames of the correction section, and calculates the distance d between the joint of interest and the instrument eq1. ６ If there is one or more joints of interest that satisfy the condition, it is determined that the person U4 is in contact with the tool q1.

[0063] In the explanation of FIG. 7, among the points on the tool eq1 in the yz plane, a point that is set in advance is p ｅ = (y ｅ , z ｅ The information processing device 100 is ｅ The distance d between the previous frame and the next frame is d<Th ６In the following description, it is determined that the person U4 is in contact with eq1 when d<Th ６ The joints in this case are called "contact joints."

[0064] The "whole body position interpolation of the correction section based on tool contact" executed by the information processing device 100 will be described. FIG. 8 is a diagram for explaining the whole body position interpolation of the correction section based on tool contact. The information processing device 100 executes the following process to calculate the whole body position of the target contact joint p start , p end , rotation center p e_origin , vector v start , v end Calculate.

[0065] The information processing device 100 calculates the average value of the coordinates of the contact joints included in the previous frame as the target contact joint p start When the previous frame contains one contact joint, the information processing device 100 calculates the coordinates of the contact joint as the target contact joint p start Let's say.

[0066] The information processing device 100 calculates the average value of the coordinates of the contact joints included in the subsequent frame as the target contact joint p end When the subsequent frame contains one contact joint, the information processing device 100 calculates the coordinates of the contact joint as the target contact joint p end Let's say.

[0067] The information processing device 100 has a rotation center p e_origin is calculated based on equation (5). e_origin is the center of rotation for calculating the circular orbit around the instrument. e_origin The x-coordinate of the target contact joint p start x coordinate of (x start ) and the contact joint p end x coordinate of (x end ) and the average of the center of rotation p e_origin The y and z coordinates of p in the yz plane explained in FIG. e is.

[0068]

[0069] The information processing device 100 has a rotation center p e_origin From the above, the contact joint p start vector v pointing to start The information processing device 100 calculates the rotation center p e_origin From the above, the contact joint p end vector v pointing to end The information processing device 100 calculates the vector v start unit vector e start and the vector v end unit vector e end Calculate.

[0070] Next, the information processing device 100 performs tool-centered Slerp to determine the target contact joint p start and the target contact joint p end The interpolation result p that smoothly connects t is calculated based on equation (6). "s" included in equation (6) is defined as in equation (7). In equation (6), an example using a unit vector has been described, but a quaternion may be used instead of the unit vector.

[0071]

[0072]

[0073] The information processing device 100 determines whether the position of the target joint is p t The whole body position is translated so that it matches the

[0074] A description will be given of the "whole body position interpolation of the correction interval based on the motion prediction" executed by the information processing device 100. The information processing device 100 executes spline interpolation based on a plurality of frames before the correction interval and a plurality of frames after the correction interval, and predicts the motion of the correction interval.

[0075] The order p of the spline interpolation is defined in advance. If the start and end of the correction interval are s, e, and N, respectively, the information processing device 100 performs spline interpolation of order p for the x-axis, y-axis, and z-axis based on the whole-body position coordinates in the intervals [s-N, s-1] and [e+1, e+N], and outputs the interpolation result for the interval [s, e].

[0076] Fig. 9 is a diagram showing an example of a motion prediction result. The horizontal axis of the graph in Fig. 9 corresponds to the frame number, and the vertical axis corresponds to the coordinate value (whole body position coordinate). In Fig. 9, the plot in area a1-1 corresponds to the whole body position coordinate in the interval [s-N, s-1]. The plot in area a1-2 corresponds to the whole body position coordinate in the interval "e+1, e+N". The plot in area a1-3 corresponds to the whole body position coordinate in the interval [s, e], which is the interpolation result.

[0077] The information processing device 100 adjusts the 3D key points in the correction section, which have been corrected according to the motion state, based on the whole-body position coordinates in the section [s, e]. For example, the information processing device 100 translates the whole-body position so that the center of the 3D key points in the correction section coincides with the center of the whole-body position coordinates.

[0078] Next, the information processing device 100 calculates the first anomaly detection threshold Th １ and the second abnormality detection threshold Th ２ An example of a process for calculating the 3D key points will be described below. The information processing device 100 acquires a plurality of normal 3D key points and a plurality of abnormal 3D key points in advance. The information processing device 100 calculates an "abnormal feature" value for each of the plurality of normal 3D key points and the plurality of abnormal 3D key points. In the following description, the value of the abnormal feature will be simply referred to as the "abnormal feature."

[0079] In order to calculate abnormal features, the information processing device 100 calculates a "3σ threshold" based on multiple normal 3D keypoints. The 3σ threshold is set for the length of each bone between joints in the human body model. As an example, the length of the bone connecting joint ar8 and joint ar9 in the human body model shown in FIG. 1 is defined as "l8-9," and a process for calculating the 3σ threshold set for the bone length l8-9 will be described.

[0080] The information processing device 100 acquires the bone lengths l8-9 from a plurality of normal 3D keypoints, and calculates the variance σ of the bone lengths l8-9. ２ The information processing device 100 calculates the variance σ of the bone length l8-9. ２ The information processing device 100 calculates σ of the bone length l8-9 by taking the root of this. The information processing device 100 calculates 3σ by multiplying σ by 3. In this way, the information processing device 100 calculates the 3σ threshold value of the bone length l8-9.

[0081] The information processing device 100 also performs the above process on the lengths of the bones between the other joints of the human body model to calculate the 3σ threshold values ​​for the lengths of the bones between the other joints, thereby obtaining the 3σ threshold values ​​corresponding to the lengths of each bone.

[0082] The information processing device 100 identifies the length of each bone at the 3D keypoint, calculates the difference (absolute value difference) between the bone length and the 3σ threshold corresponding to this bone length, and calculates the sum of these differences as the abnormal feature of the 3D keypoint.

[0083] The information processing device 100 calculates abnormal features for a plurality of normal 3D keypoints and a plurality of abnormal 3D keypoints that have been acquired in advance.

[0084] Fig. 10 shows an example of the distribution of 3D keypoints related to abnormal features. The horizontal axis of the graph in Fig. 10 corresponds to the abnormal features, and the vertical axis corresponds to the number of 3D keypoint data. In Fig. 10, dis1 is the distribution of normal 3D keypoints, and dis2 is the distribution of abnormal 3D keypoints.

[0085] 11 is a diagram illustrating the process of calculating the first anomaly detection threshold. The information processing apparatus 100 calculates the boundary value at which the degree of separation between the normal 3D keypoint distribution dis1 and the abnormal 3D keypoint distribution dis2 is maximized as the first anomaly detection threshold Th １ For example, the information processing device 100 executes the following process to calculate the first abnormality detection threshold Th １ Calculate.

[0086] The information processing device 100 calculates v based on the formula (8). normal Calculate μ included in equation (8). normal denotes the average value of the abnormal features of the normal 3D keypoints. normal is the root of the variance of the abnormal feature of the normal 3D keypoints.

[0087]

[0088] The information processing device 100 calculates v based on the formula (9). anormal Calculate μ included in equation (9). anormal denotes the mean value of the abnormal feature of the abnormal 3D keypoints. anormal is the root of the variance of the abnormal features of the abnormal 3D keypoints.

[0089]

[0090] The information processing device 100 calculates the search range [v low , v high ] is identified. low , v high ] is the range where the degree of separation between the normal 3D keypoint distribution dis1 and the abnormal 3D keypoint distribution dis2 is likely to be maximum.

[0091]

[0092] The information processing device 100 sets the search range [v low , v high ], the value at which the degree of separation is maximum is defined as the first anomaly detection threshold Th １The information processing device 100 calculates the search range [v low , v high ], the information processing device 100 may specify a value that maximizes the degree of separation within the search range [v low , v high ] and the distribution dis2 intersect, the abnormality feature is determined by the first abnormality detection threshold Th １ Let's say.

[0093] 12 is a diagram illustrating the process of calculating the second abnormality detection threshold. １ In order to set a threshold that is more sensitive than low , v high ]Start v low The second abnormality detection threshold Th ２ Set as.

[0094] By the information processing device 100 performing the above process, the first abnormality detection threshold Th １ and the second abnormality detection threshold Th ２ is calculated.

[0095] In this embodiment, as an example, the information processing device 100 low , v high ]Start v low The second abnormality detection threshold Th ２ I set it as, but the end v high The second abnormality detection threshold Th ２ In this way, the end v high When the first abnormality detection threshold Th １ This results in setting a threshold that is less sensitive than the

[0096] Next, the information processing device 100 sets the first abnormality detection threshold Th １ and the second abnormality detection threshold Th ２The following describes a process of specifying a correction interval of a 3D keypoint sequence based on the above. When setting a correction interval, the information processing device 100 executes first anomaly detection and second anomaly detection.

[0097] The following describes the first anomaly detection performed by the information processing device 100. Fig. 13 is a diagram for explaining the first anomaly detection. In the graph of Fig. 13, the vertical axis corresponds to the anomaly distance, and the horizontal axis corresponds to the frame number of the 3D keypoint sequence.

[0098] The information processing device 100 selects a 3D keypoint with frame number n included in the 3D keypoint sequence. The information processing device 100 identifies the length of each bone at the selected 3D keypoint, calculates the difference (absolute value difference) between the bone length and a 3σ threshold corresponding to the bone length, and calculates the sum of the differences as the abnormal feature of the 3D keypoint. The 3σ threshold is set in advance by the above process.

[0099] The information processing device 100 repeatedly performs the above process for the 3D keypoints of each frame number in the 3D keypoint sequence to identify the relationship between the frame number of the 3D keypoint and the anomalous feature. In the example shown in Fig. 13, the relationship between the frame number of each 3D keypoint included in the 3D keypoint sequence and the anomalous feature is indicated by a line L1.

[0100] The information processing device 100 is connected to the line L1 and the first abnormality detection threshold Th １ and the abnormality feature is compared with the first abnormality detection threshold Th １ A section in which the above occurs is identified as the abnormal section In1. In the example shown in Fig. 13, the frame number of the frame before the abnormal section In1 is "1054" and the frame number of the frame after it is "1060".

[0101] The information processing device 100 detects whether the abnormality feature is greater than or equal to the first abnormality detection threshold Th １ Even in an abnormal section, if the number of frames in the section is less than a predetermined number of frames, the section may be excluded from the abnormal section.

[0102] The second anomaly detection executed by the information processing device 100 will be described. The frame before the anomaly section is defined as "s" and the frame after the anomaly section is defined as "e". The anomaly feature in the previous frame s is defined as "f ｓ The abnormal feature in the previous frame e is defined as "f ｅ The number of frames in the abnormal section is "N". The information processing device 100 sets the first convergence condition or the second convergence condition and optimizes the abnormal section. The initial value of the abnormal section is the abnormal section In1 detected in the first abnormality detection described above.

[0103] First, a process in which the information processing device 100 sets the first convergence condition and performs optimization of the abnormal section will be described. The first convergence condition is as follows: Th ２ is the second abnormality detection threshold Th ２ is.

[0104] f ｓ <Th ２ and f ｅ <Th ２ ...(First convergence condition)

[0105] The information processing device 100 is ｓ and f ｅ For example, the information processing apparatus 100 compares f ｓ ga f ｅ If the frame number of the previous frame s is greater than f, the frame with the frame number obtained by subtracting 1 from the frame number of the previous frame s is set as the new previous frame. ｓ ga f ｅ If the frame number is smaller than the frame number of the next frame e, the frame with the frame number obtained by adding 1 to the frame number of the next frame e is set as the new next frame e.

[0106] The information processing device 100 again calculates f for the new previous frame s or the new subsequent frame e. ｓ or f ｅ The information processing device 100 repeatedly executes the step of calculating the evaluation value E=f until the first convergence condition (or the second convergence condition) is satisfied. ｓ +f ｅ The previous frame s and the subsequent frame e for which are the smallest are adopted, and the abnormal section is updated.

[0107] 14 is a diagram showing an example of optimization of the abnormal section based on the first convergence condition. The information processing device 100 executes the processes of steps 1 to 4 described below.

[0108] Step 1 will be described. The information processing device 100 sets the frame number of the previous frame and the frame number of the subsequent frame based on the abnormal section identified by the first abnormality detection. In this example, the frame number of the previous frame s is "1054" and the frame number of the subsequent frame e is "1060". The abnormal feature f of the previous frame s is ｓ The abnormal feature f of the next frame e is set to "104". ｅ is set to "115". The evaluation value E is set to "219". The number of frames N in the abnormal section is set to "7". ｓ and f ｅ is "FALSE" because it does not satisfy the first convergence condition.

[0109] Step 2 will be described. The information processing device 100 ｓ ga f ｅ Since the frame number of the subsequent frame e is smaller than ｅ The frame number of the previous frame s, the abnormal feature f ｓ is the same as in step 1. The evaluation value E is "119". The number of frames N in the abnormal section is "8". ｓ and f ｅ is "FALSE" because it does not satisfy the first convergence condition.

[0110] Step 3 will be described. The information processing device 100 ｅ ga f ｓ Since the frame number of the previous frame s is smaller than ｓ The frame number of the next frame e, the abnormal feature f ｅ is the same as in step 2. The evaluation value E is "195". The number of frames N in the abnormal section is "9". ｓ and f ｅis "FALSE" because it does not satisfy the first convergence condition.

[0111] Step 4 will be described. The information processing device 100 ｅ ga f ｓ Since the frame number of the previous frame s is smaller than ｓ The frame number of the next frame e, the abnormal feature f ｅ is the same as in step 3. The evaluation value E is "38". The number of frames N in the abnormal section is "10". ｓ and f ｅ is "TRUE" because it satisfies the first convergence condition.

[0112] In step 4, the information processing device 100 identifies the step with the smallest evaluation value E among the evaluation values ​​E of steps 1 to 4 in order to satisfy the first convergence condition. In the example shown in FIG. 14 , the evaluation value "38" of step 4 is the smallest. The information processing device 100 sets the section from the frame number "1053" immediately after the previous frame s in step 4 to the frame number "1060" immediately before the subsequent frame e as the abnormal section In2. In other words, the abnormal section In1 set in the first abnormality detection is optimized to the abnormal section In2.

[0113] Next, a process in which the information processing device 100 sets the second convergence condition and performs optimization of the abnormal section will be described. The second convergence condition is the following condition.

[0114] The first convergence condition is satisfied or N≧9 (the second convergence condition is satisfied)

[0115] The information processing device 100 updates the frame number of the previous frame s and the frame number of the subsequent frame e until the second convergence condition is satisfied, and ｓ , f ｅ The process of updating the evaluation value E is similar to the process described above regarding the first convergence condition.

[0116] 15 is a diagram showing an example of optimization of the abnormal section based on the second convergence condition. The information processing device 100 executes the processes of steps 1 to 3 described below.

[0117] Step 1 will be described. The information processing device 100 sets the frame number of the previous frame and the frame number of the subsequent frame based on the abnormal section identified by the first abnormality detection. In this example, the frame number of the previous frame s is "1054" and the frame number of the subsequent frame e is "1060". The abnormal feature f of the previous frame s is ｓ The abnormal feature f of the next frame e is set to "104". ｅ is set to "115". The evaluation value E is set to "219". The number of frames N in the abnormal section is set to "7". ｓ and f ｅ is "FALSE" because it does not satisfy the first convergence condition.

[0118] Step 2 will be described. The information processing device 100 ｓ ga f ｅ Since the frame number of the subsequent frame e is smaller than ｅ The frame number of the previous frame s, the abnormal feature f ｓ is the same as in step 1. The evaluation value E is "119". The number of frames N in the abnormal section is "8". ｓ and f ｅ is "FALSE" because it does not satisfy the first convergence condition.

[0119] Step 3 will be described. The information processing device 100 ｅ ga f ｓ Since the frame number of the previous frame s is smaller than ｓ The frame number of the next frame e, the abnormal feature f ｅ is the same as in step 2. The evaluation value E is "195". The number of frames N in the abnormal section is "9". N in step 3 satisfies the second convergence condition, so it is "TRUE".

[0120] In step 3, the information processing device 100 identifies the step with the smallest evaluation value E among the evaluation values ​​E of steps 1 to 3 in order to satisfy the second convergence condition. In the example shown in FIG. 15 , the evaluation value "119" of step 2 is the smallest. The information processing device 100 sets the section from the frame number "1055" immediately after the previous frame s in step 2 to the frame number "1060" immediately before the subsequent frame e as the abnormal section In3. In other words, the abnormal section In1 set in the first abnormality detection is optimized to the abnormal section In3.

[0121] The information processing device 100 identifies the optimized abnormal section as a correction section for the 3D keypoint sequence.

[0122] Next, a configuration example of an information processing device 100 that executes the processes described above with reference to Figures 4 to 15 will be described. Figure 16 is a functional block diagram showing the configuration of an information processing device according to this embodiment. As shown in Figure 16, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0123] The communication unit 110 executes data communication with the camera 30, external devices, etc. via a network. The communication unit 110 is a network interface card (NIC) or the like. The control unit 150, which will be described later, exchanges data with external devices via the communication unit 110.

[0124] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the information processing device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0125] The display unit 130 is a display device that displays information output from the control unit 150 .

[0126] The storage unit 140 includes a video buffer 141, a 3D key point table 142, a machine learning model 143, and a technique recognition table 144. The storage unit 140 is a storage device such as a memory.

[0127] The video buffer 141 is a buffer that stores video data acquired from the camera 30. The video data includes image frames in time series, and each image frame is assigned a frame number in ascending order.

[0128] The 3D key point table 142 is a table that stores information about 3D key points. Fig. 17 is a diagram showing an example of the data structure of the 3D key point table. As shown in Fig. 17, the 3D key point table 142 stores frame numbers and three-dimensional coordinates corresponding to each joint (joint identification information) in association with each other.

[0129] The frame number is a frame number that identifies the image frame used when generating the 3D keypoints. The identification information of each joint is information that uniquely identifies the joint. In the explanation of Figure 17, joints ar0 to ar20 are used as the identification information of the joints. For example, joint ar0 corresponds to "SPINE_BASE". joint ar1 corresponds to "SPINE_MID". joint ar20 corresponds to "HAND_TIP_RIGHT". The relationship between the other joints and the joint names is shown in Figure 2.

[0130] The three-dimensional coordinates of the joints ar0 to ar20 corresponding to a certain frame number (for example, 0001) in the 3D key point table 142 become the 3D key points corresponding to the certain frame number (for example, 0001).

[0131] The machine learning model 143 is used when performing the above-mentioned "interpolation using a machine learning model." This machine learning model 143 is a model that receives the previous frame and the next frame as input and outputs the path of the coordinates of each joint from the previous frame to the next frame.

[0132] The technique recognition table 144 is a table that associates the time-series changes in the positions of each joint included in the time-series 3D key points with the type of technique. The technique recognition table 144 also associates a combination of technique types with a score. The score is calculated as the sum of a D (Difficulty) score and an E (Execution) score. For example, the D score is calculated based on the difficulty of the technique. The E score is calculated using a point deduction system according to the degree of execution of the technique.

[0133] The control unit 150 includes an acquisition unit 151, a 3D key point generation unit 152, a correction section identification unit 153, a conversion processing unit 154, a correction processing unit 155, and an evaluation unit 156. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0134] The acquisition unit 151 acquires video data from the camera 30 via the network. The acquisition unit 151 stores the acquired video data in the video buffer 141.

[0135] The 3D key point generation unit 152 generates time-series 3D key points based on the video data (time-series image frames) stored in the video buffer 141 , and stores them in the 3D key point table 142 .

[0136] For example, the process by which the 3D keypoint generator 152 generates 3D keypoints from an image frame is similar to the process described in Fig. 33. That is, the 3D keypoint generator 152 first generates 2D keypoints by inputting the image frame into a trained learning model, and then generates 3D keypoints by integrating the 2D keypoints.

[0137] The correction section identification unit 153 determines the first abnormality detection threshold Th １ and the second abnormality detection threshold Th ２ The correction section identification unit 153 executes a process of calculating the first abnormality detection threshold Th １ The correction section specification unit 153 performs the first abnormality detection using the second abnormality detection threshold Th ２ The second abnormality detection is performed using

[0138] The correction section identification unit 153 determines the first abnormality detection threshold Th １ The process of calculating the second abnormality detection threshold Th ２ The process of calculating is the same as the process described above with reference to FIG.

[0139] The first abnormality detection performed by the correction section identification unit 153 is similar to the processing described with reference to Fig. 13. The second abnormality detection performed by the correction section identification unit 153 is similar to the processing described with reference to Fig. 14 or the processing described with reference to Fig. 15.

[0140] The correction interval specifying unit 153 executes the above process to set a correction interval in the 3D keypoint sequence. The correction interval specifying unit 153 outputs information about the correction interval set in the 3D keypoint sequence to the correction processing unit 155.

[0141] The conversion processing unit 154 converts the coordinates of each joint of the 3D keypoint into data on the whole body position, whole body rotation, and each joint rotation. The conversion processing unit 154 outputs the conversion results to the correction processing unit 155. For example, the whole body position is the position of all 3D keypoints with respect to the reference posture. The whole body rotation is the rotation angle of all 3D keypoints with respect to the reference posture. The joint rotation is the rotation angle of each joint of the 3D keypoint with respect to each joint in the reference posture.

[0142] FIG. 18 is a diagram showing an example of a reference posture. In the example shown in FIG. 18, the reference posture is represented by nodes rn and n0 to n20. Node rn is the ROOT. "ROOT" indicates the node of the reference joint of the human body. Nodes n0 to n2, n4 to n12, n14 to n16, and n18 are JOINT. "JOINT" indicates the joint node. Nodes n3, n13, n17, n19, and n20 are End. "End" indicates the node at the tip of the limbs, etc. For adjacent nodes, a "joint direction vector" is set from the higher node to the lower node. The joint direction vector between each node in the reference posture is defined as "OFFSET."

[0143] 19 is a diagram illustrating an example of a three-dimensional rotation angle of the ROOT. For example, the three-dimensional rotation angle of the ROOT is the Euler angles (θx, θy, θz) that transform the ROOT node rn-1 in the global coordinate system to the ROOT node rn-2 in the ROOT coordinate system.

[0144] 20 is a diagram illustrating an example of the three-dimensional rotation angle of a JOINT. For example, the three-dimensional rotation angle of a JOINT is the Euler angles (θx, θy, θz) that convert from the higher-level JOINT coordinate system to the JOINT coordinate system of interest. For example, if the JOINT of interest is node n4, the node above node n4 is node n2. In this case, the higher-level JOINT coordinate system is the coordinate system of node n2. The JOINT coordinate system of interest is the coordinate system of node n4.

[0145] Here, the 3D keypoints are three-dimensional coordinate data, and since there is no information about the root or joint, it is not possible to simply obtain the three-dimensional rotation angle of the root and the three-dimensional rotation angle of each joint from the 3D keypoints. For example, the transformation processing unit 154 obtains the three-dimensional rotation angle of the root and the three-dimensional rotation angle of each joint from the 3D keypoints based on conventional technology such as inverse kinematics.

[0146] The transformation processing unit 154 may calculate the three-dimensional rotation angle of the ROOT by rigid body alignment, and may calculate the three-dimensional rotation angle of the JOINT by the Rodrigues rotation formula, as described below.

[0147] An example of the process in which the conversion processing unit 154 calculates the three-dimensional rotation angle of the ROOT by rigid body alignment will be described. FIG. 21 is a diagram for explaining the process of calculating the three-dimensional rotation angle of the ROOT by rigid body alignment. The conversion processing unit 154 divides each node of the hierarchical structure data (data of the reference posture) into three or more rigid joint nodes and other nodes according to a predefined definition. As described in FIG. 18 , the nodes of the reference posture are nodes rn, n0 to n20. In the example shown in FIG. 21 , the conversion processing unit 154 selects nodes n0, n10, and n14 from the nodes included in the reference posture 40 as rigid joint nodes.

[0148] The conversion processing unit 154 identifies a group of joints corresponding to rigid joint nodes (nodes n0, n10, and n14) from the joints included in the 3D key points 41. In the example shown in FIG. 21 , the group of joints corresponding to the rigid joint nodes are joints ar0, ar10, and ar14. In the following description, of the joints included in the 3D key points, the group of joints corresponding to the rigid joint nodes will be referred to as "rigid-body corresponding joints."

[0149] The transformation processing unit 154 calculates the relative rotation angles from the rigid joint nodes (nodes n0, n10, n14) to the rigid corresponding joints (joints ar0, ar10, ar14) by rigid alignment, and sets the calculated angles as the three-dimensional rotation angles of the ROOT.

[0150] Rigid body registration is a method for finding transformation parameters for aligning a source with a target using a combination of three or more points between the source and target using the least squares method according to equation (11). The transformation parameters include a rotation matrix R, a translation t, and a scale c. In this embodiment, the rotation matrix R is converted into Euler angles and used as the three-dimensional rotation angle of ROOT.

[0151]

[0152] In formula (11), "x" is the coordinate of three or more sources. The coordinate of the source is the three-dimensional coordinate of the rigid joint. "y" is the coordinate of three or more targets. The coordinate of the target is the three-dimensional coordinate of the rigid joint. The conversion processing unit 154 converts e in formula (11) ２ The rotation matrix R, translation t, and scale c that minimize the following are found.

[0153] Next, an example of the process in which the conversion processing unit 154 calculates the three-dimensional rotation angle of a joint using the Rodrigues rotation formula will be described. FIGS. 22 and 23 are diagrams for explaining the process of calculating the three-dimensional rotation angle of a joint using the Rodrigues rotation formula. First, FIG. 22 will be described. The conversion processing unit 154 converts the joint direction vector of the target joint of the 3D key point 41 into a joint direction vector in the local coordinate system. The joint direction vector of the 3D key point 41 is a vector directed from a lower joint to a higher joint among adjacent joints.

[0154] 22, the joint of interest is assumed to be joint ar0. The joint direction vector v_{tar} of joint ar0 is a vector directed from the lower joint ar0 to the higher joint ar1.

[0155] The transformation processing unit 154 converts the joint direction vector of the target joint into the inverse matrix (R －１ ) to convert it into a joint direction vector in the local coordinate system. For example, multiplying the joint direction vector v_{tar} of the target joint by R －１ The transformation processing unit 154 repeatedly executes the above process for each joint of the 3D key points 41 to obtain the 3D key points 42 in the local coordinate system.

[0156] Next, the conversion processing unit 154 identifies the angle θ between the joint direction vector of the 3D key point 42 and the joint direction vector of the reference posture 40, with the normal to the joint direction vector of the 3D key point 42 and the joint direction vector of the reference posture 40 serving as the rotation axis. Here, as an example, a description will be given using the joint direction vector v_{tar_local} of the 3D key point 42 and the joint direction vector v_{src} of the reference posture 40. The joint direction vector v_{src} is a vector directed from node n0 of the reference posture to node n1.

[0157] We now move on to the description of Fig. 23. In Fig. 23, n is the normal between the joint direction vector v_{tar_local} and the joint direction vector v_{src} of the reference posture 40. The transformation processing unit 154 identifies the normal n by calculating the cross product of the joint direction vector v_{tar_local} and the joint direction vector v_{src}. The transformation processing unit 154 identifies the angle θ between the joint direction vector v_{tar_local} and the joint direction vector v_{src} using the normal n as the rotation axis.

[0158] The conversion processing unit 154 uses the normal n as the rotation axis and the angle θ as the rotation angle, calculates the relative rotation angle using Rodrigues' rotation formula, and uses it as the three-dimensional rotation angle of the JOINT.

[0159] The Rodrigues rotation formula is a formula for calculating the rotation matrix R according to equation (12) from the rotation axis (normal n) and rotation angle (angle θ) specified by the source and target of the joint direction vector. The source of the joint direction vector is the joint direction vector (v_{src}) of the reference posture 40. The target of the joint direction vector is the joint direction vector (v_{tar_local}) of the 3D key point 42 in the local coordinate system.

[0160]

[0161] The conversion processing unit 154 converts R calculated by the formula (12) ｎ (θ) is converted into an Euler angle and used as the three-dimensional rotation angle of the JOINT.

[0162] As described above, the conversion processing unit 154 converts the coordinates of each joint at the 3D key point into whole-body rotation and joint rotation data by calculating the relative rotation angle from the reference posture to the 3D key point. The conversion processing unit 154 calculates the position of all 3D key points relative to the reference posture as the whole-body position. The conversion processing unit 154 outputs the conversion result to the correction processing unit 155.

[0163] Returning to the explanation of Fig. 16, the correction processing unit 155 determines the motion state of person U4 based on the previous frame and the next frame of the correction section set in the 3D key point sequence. The process by which the correction processing unit 155 determines the motion state of person U4 corresponds to the above-mentioned "long-term motion determination," "stillness determination," "circular motion determination," and "circular motion direction determination."

[0164] The correction processing unit 155 corrects the 3D key points in the correction section based on the determination result of the motion state. The information processing device 100 selects one of Slerp (shorter path) interpolation, Slerp (longer path) interpolation, and machine learning model interpolation from the determination result of the motion state using the processing procedure described in Fig. 6 and performs correction. The Slerp (shorter path) interpolation, Slerp (longer path) interpolation, and machine learning model interpolation performed by the correction processing unit 155 are the same as those described above for each interpolation.

[0165] After making the correction according to the above-described motion state, the correction processing unit 155 performs "instrument contact determination" to determine whether or not the person U4 is in contact with the instrument. The instrument contact determination performed by the correction processing unit 155 is the same as the process described in FIG. 7.

[0166] When the correction processing unit 155 determines that the person U4 is in contact with the appliance through the appliance contact determination, it performs "whole body position interpolation of the correction section based on appliance contact." On the other hand, when the correction processing unit 155 determines that the person U4 is not in contact with the appliance through the appliance contact determination, it performs "whole body position correction based on movement prediction."

[0167] The "whole body position interpolation of the correction section based on tool contact" executed by the correction processing unit 155 is the same as the above-described process explained with reference to FIG.

[0168] The "whole body position correction based on motion prediction" executed by the correction processing unit 155 is similar to the above-described process described with reference to FIG.

[0169] The correction processing unit 155 updates the 3D key points of the correction section stored in the 3D key point table 142 based on the correction result.

[0170] The evaluation unit 156 is a processing unit that evaluates the performance of person U4 based on the time-series 3D key points stored in the 3D key point table 142 and the technique recognition table 144. For example, the evaluation unit 156 compares the time-series changes of each joint of the 3D key points with the technique recognition table 144 to identify the technique type. Furthermore, the evaluation unit 157 compares the combination of technique types with the technique recognition table to calculate the score of the performance of person U4. The evaluation unit 156 generates screen information of the technique type and score and displays it on the display unit 130.

[0171] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 24 is a flowchart showing the processing procedure of the information processing device 100 according to this embodiment. As shown in Fig. 24, the acquisition unit 151 of the information processing device 100 acquires video data from the camera 30 and stores it in the video buffer 141 (step S201).

[0172] The 3D keypoint generation unit 152 of the information processing device 100 generates a 3D keypoint sequence based on the time-series image frames of the video data (step S202). The correction section identification unit 153 of the information processing device 100 executes a correction section identification process (step S203).

[0173] The information processing device 100 identifies the previous and subsequent frames of the correction section (step S204). The conversion processing unit 154 of the information processing device 100 calculates the whole body position, whole body rotation, and each joint rotation based on the 3D key points (step S205).

[0174] The correction processing unit 155 of the information processing device 100 executes the correction processing (step S206). The correction processing unit 155 updates the 3D key points of the correction section (step S207). The evaluation unit 156 of the information processing device 100 performs evaluation based on the 3D key points and outputs the evaluation result to the display unit 130 (step S208).

[0175] Next, the processing procedure of the correction section identification process described in step S203 of Fig. 24 will be described. Fig. 25 is a flowchart showing the processing procedure of the correction section identification process. As shown in Fig. 25, the correction section identification unit 153 of the information processing device 100 acquires a 3D key point sequence (step S301).

[0176] The correction section identification unit 153 calculates abnormal features for each frame of the 3D keypoint sequence (step S302), and executes first abnormality detection to identify an abnormal section (step S303).

[0177] The correction section identification unit 153 executes second abnormality detection and optimizes the abnormal section (step S304).The correction section identification unit 153 sets the optimized abnormal section as the correction section (step S305).

[0178] Next, the processing procedure of the correction processing shown in step S206 in Fig. 24 will be described. Fig. 26 is a flowchart showing the processing procedure of the correction processing. As shown in Fig. 26, the correction processing unit 155 of the information processing device 100 determines the motion state based on the 3D key points, the whole body position, the whole body rotation, and the rotation of each joint (step S401).

[0179] The correction processing unit 155 executes interpolation processing according to the motion state (step S402), executes correction processing based on tool contact (step S403), and outputs the correction result (step S404).

[0180] The processing procedure of the interpolation processing according to the motion state shown in step S402 in FIG. 26 corresponds to the processing procedure described with reference to FIG.

[0181] Next, the processing procedure for the correction processing based on tool contact described in step S403 of Fig. 26 will be described. Fig. 27 is a flowchart showing the processing procedure for the correction processing based on tool contact. As shown in Fig. 27, the correction processing unit 155 of the information processing device 100 determines the motion state based on 3D key points, whole body position, whole body rotation, and rotation of each joint (step S501).

[0182] The correction processing unit 155 performs an appliance contact determination (step S502). If the person is contacting an appliance (step S503, Yes), the correction processing unit 155 interpolates the whole-body position based on the appliance center prediction (step S504), and proceeds to step S506.

[0183] On the other hand, if the person is not in contact with the tool (step S503, No), the correction processing unit 155 interpolates the whole-body position based on the motion prediction (step S505). The correction processing unit 155 outputs the interpolation result (step S506).

[0184] Next, the information processing device 100 determines whether the first abnormality detection threshold Th １ and the second abnormality detection threshold Th ２An example of a processing procedure for setting a threshold value will be described below. Fig. 28 is a flowchart showing the processing procedure for setting a threshold value. As shown in Fig. 28 , the correction section identification unit 153 of the information processing device 100 acquires a plurality of normal 3D key points and a plurality of abnormal 3D key points (step S601).

[0185] The correction section identification unit 153 calculates the abnormal feature of each 3D key point (step S602).

[0186] The correction section identification unit 153 sets the boundary value at which the degree of separation between the normal 3D keypoint distribution and the abnormal 3D keypoint distribution is maximized as the first anomaly detection threshold (step S603).The correction section identification unit 153 sets the start position of the threshold search section as the second anomaly detection threshold (step S604).

[0187] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 sets a correction interval in the 3D key point sequence and determines the motion state of person U4 based on the 3D key points before and after the correction interval. The information processing device 100 determines whether person U4 is in contact with an appliance based on the 3D key points before and after the correction interval. The information processing device 100 corrects the 3D key points in the correction interval based on the motion state of person U4 and the determination result of whether person U4 is in contact with an appliance. This can improve the accuracy of recognizing a person's skeleton.

[0188] 29 and 30 are diagrams for explaining the effects of the information processing device. Referring to Fig. 29, a 3D keypoint sequence 61 includes frames 61-1, 61-2, 61-3, 61-4, 61-5, and 61-6. A 3D keypoint sequence 62 includes frames 62-1, 62-2, 62-3, 62-4, 62-5, and 62-6.

[0189] The correction section 61a of the 3D keypoint sequence 61 is corrected using a conventional technique. In the conventional technique, an incorrect path (a shorter path) is selected based on frames 61-1 and 61-6, and correction is performed using Slerp. As a result, frames 61-2 to 61-5 of the correction section 61a are not properly corrected.

[0190] On the other hand, the correction section 62a of the 3D keypoint sequence 62 is corrected by the information processing device 100 according to the present invention. The information processing device 100 selects an appropriate path (longer path) based on frames 62-1 and 62-6 and performs correction using Slerp, so that each of frames 62-2 to 62-5 in the correction section 62a is appropriately corrected.

[0191] 30. The 3D keypoint sequence 63 includes frames 63-1, 63-2, 63-3, 63-4, and 63-5. The 3D keypoint sequence 64 includes frames 64-1, 64-2, 64-3, 64-4, and 64-5.

[0192] The correction section 63a of the 3D keypoint sequence 63 has been corrected using the conventional technique. In the conventional technique, the joint positions of the person in each of the frames 63-2 to 63-4 in the correction section 63a are far from the tool 63b and are not properly corrected.

[0193] On the other hand, a correction section 64a of the 3D keypoint sequence 64 is corrected by the information processing device 100 according to the present invention. When the person is in contact with the tool 64b, the information processing device 100 performs correction using tool-centered prediction, and the joint positions of the person in each of frames 64-2 to 64-4 in the correction section 64a are in contact with the tool 64b and are appropriately corrected.

[0194] Furthermore, the information processing device 100 acquires the distribution of normal 3D keypoints and the distribution of abnormal 3D keypoints in advance, and calculates the first abnormality detection threshold Th １ and the second abnormality detection threshold Th ２ The first abnormality detection threshold Th １ is a boundary value at which the degree of separation between the normal 3D keypoint distribution and the abnormal 3D keypoint distribution is maximized. ２ is the first abnormality detection threshold Th １ The information processing device 100 uses the first anomaly detection threshold Th １ and the second abnormality detection threshold Th ２Based on this, a correction interval is set for the 3D keypoint sequence. This makes it possible to appropriately set an interval where the coordinates of the joints of the 3D keypoints are abnormal as the correction interval.

[0195] 31 is a diagram for explaining the effect of the information processing device. A 3D keypoint sequence 65 includes frames 65-1, 65-2, 65-3, 65-4, 65-5, and 65-6. A 3D keypoint sequence 66 includes frames 66-1, 66-2, 66-3, 66-4, 66-5, and 66-6.

[0196] In the prior art, a correction interval 65a is set for a 3D keypoint sequence 65 by simple threshold comparison, and correction of the correction interval 65a is performed based on the frames before and after the correction interval 65a (frames 65-2 and 65-5). However, frames 65-2 and 65-5 may be frames in which abnormal features are not suppressed, and performing correction based on frames 65-2 and 65-5 in which abnormal features are not suppressed reduces the accuracy of the correction results.

[0197] On the other hand, the information processing device 100 applies an anomaly detection threshold Th １ and the second abnormality detection threshold Th ２ Based on this, two-stage anomaly detection is performed and a correction section 66a is set. The frames before and after the correction section 66a (frames 66-1 and 66-6) are frames in which abnormal features are suppressed. By performing correction based on frames 66-1 and 66-6 in which abnormal features are suppressed, the information processing device 100 can suppress a decrease in the accuracy of the correction results.

[0198] Next, a description will be given of an example of the hardware configuration of a computer that realizes the same functions as the above-described information processing device 100. Fig. 32 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0199] 32, the computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that exchanges data with the camera 15, external devices, etc. via a wired or wireless network, and an interface device 205. The computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk drive 207. The devices 201 to 207 are connected to a bus 208.

[0200] The hard disk drive 207 stores an acquisition program 207 a, a 3D keypoint generation program 207 b, a correction section identification program 207 c, a conversion processing program 207 d, a correction processing program 207 e, and an evaluation program 207 f. The CPU 201 reads out each of the programs 207 a to 207 f and loads them into the RAM 206.

[0201] The acquisition program 207a functions as an acquisition process 206a. The 3D keypoint generation program 207b functions as a 3D keypoint generation process 206b. The correction section identification program 207c functions as a correction section identification process 206c. The conversion processing program 207d functions as a conversion processing process 206d. The correction processing program 207e functions as a correction processing process 206e. The evaluation program 207f functions as an evaluation process 206f.

[0202] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the 3D keypoint generation process 206b corresponds to the processing of the 3D keypoint generation unit 152. The processing of the correction section identification process 206c corresponds to the processing of the correction section identification unit 153. The processing of the conversion processing process 206d corresponds to the processing of the conversion processing unit 154. The processing of the correction processing process 206e corresponds to the processing of the correction processing unit 155. The processing of the evaluation process 206f corresponds to the processing of the evaluation unit 156.

[0203] It should be noted that each of the programs 207a to 207f does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207f.

[0204] REFERENCE SIGNS LIST 100 Information processing device 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 141 Video buffer 142 3D keypoint table 143 Machine learning model 144 Technique recognition table 150 Control unit 151 Acquisition unit 152 3D keypoint generation unit 153 Correction section identification unit 154 Conversion processing unit 155 Correction processing unit 156 Evaluation unit

Claims

1. Based on a plurality of joints included in the human body and a plurality of skeletal information in which coordinates for the plurality of joints are respectively set, an abnormal feature is calculated, Among the plurality of skeletal information, based on a first distribution of normal skeletal information related to the abnormal feature and a second distribution of abnormal skeletal information related to the abnormal feature, a boundary value at which the separation degree between the first distribution and the second distribution is maximized is calculated as a first threshold value, Based on the first threshold value, a correction range for time-series skeletal information is set An information processing method, characterized in that a computer executes the process.

2. In a search range where there is a high possibility that the separation degree between the first distribution and the second distribution is maximized, among the search ranges indicating the range of the abnormal feature, the minimum abnormal feature or the maximum abnormal feature is calculated as a second threshold value, and the computer further executes a process of correcting the correction range using the second threshold value. The information processing method according to claim 1, characterized in that

3. The process of calculating the abnormal feature calculates the abnormal feature of the skeletal information based on the difference between a reference value specified from the bone length of the plurality of skeletal information and the bone length of the skeletal information. The information processing method according to claim 1, characterized in that

4. The information processing method according to claim 2, characterized in that correction is further executed for the correction range set for the time-series skeletal information.

5. Based on a plurality of joints included in the human body and a plurality of skeletal information in which coordinates for the plurality of joints are respectively set, an abnormal feature is calculated, Among the plurality of skeletal information, in a search range where there is a high possibility that the separation degree between a first distribution of normal skeletal information related to the abnormal feature and a second distribution of abnormal skeletal information related to the abnormal feature is maximized, and among the search ranges indicating the range of the abnormal feature, based on the minimum abnormal feature, the correction range set for the time-series skeletal information is corrected by a certain threshold value An information processing method, characterized in that a computer executes the process.

6. Based on a plurality of joints included in the human body and a plurality of skeletal information in which coordinates for the plurality of joints are respectively set, an abnormal feature is calculated, Among the plurality of skeletal information, based on a first distribution of normal skeletal information related to the abnormal feature and a second distribution of abnormal skeletal information related to the abnormal feature, a boundary value at which the separation degree between the first distribution and the second distribution is maximized is calculated as a first threshold value, Based on the first threshold value, a correction range for time-series skeletal information is set An information processing program characterized by causing a computer to execute a process.

7. Based on a plurality of joint coordinates included in the human body and a plurality of skeleton information in which coordinates for the plurality of joints are respectively set, an abnormal feature is calculated, Among the plurality of skeleton information, based on a first distribution of normal skeleton information related to the abnormal feature and a second distribution of abnormal skeleton information related to the abnormal feature, a boundary value at which the separation degree between the first distribution and the second distribution becomes maximum is calculated as a first threshold value, Based on the first threshold value, a correction range of time-series skeleton information is set An information processing apparatus having a control unit that executes a process.