Skeletal recognition device, learning method and learning program

Abstract

The skeletal recognition device of the present invention inputs teacher data into a learning model and calculates an output based on forward propagation. The skeletal recognition device calculates the value of a loss function based on the difference in bone length between the skeletal information (which is the forward solution value based on the teacher data) and the output of the recognition processing unit, and the difference in the angles between the bones based on the skeletal information and the output of the recognition processing unit.

Description

Technical Field

[0001] This invention relates to skeletal recognition devices, etc. Background Technology

[0002] Regarding skeletal recognition for detecting 3D human motion, a 3D sensing and detection technology was established to extract 3D skeletal coordinates from multiple 3D laser sensors with an accuracy of ±1cm. This technology is expected to be applied to various sports. The method of using 3D laser sensors will be referred to as the "laser method".

[0003] In the laser-based method described above, the laser is irradiated approximately 2 million times per second. Based on the laser's time of flight (ToF), depth data for each irradiated point, including the person being measured, is calculated. While the laser method can acquire high-precision depth data, the complex structure and processing involved in laser scanning and ToF measurement result in complex and expensive hardware.

[0004] For laser-based imaging, in image processing methods that acquire RGB data of each pixel using a CMOS (Complementary Metal Oxide Semiconductor) imager, inexpensive RGB cameras can be used for 3D skeleton recognition.

[0005] For example, as a prior art for skeletal recognition via image processing, there is prior art 1. Figure 11 This is a diagram used to illustrate prior art 1. In prior art 1, 2D features 2 representing joint features are obtained by applying 2D backbone processing 1a to each input image 1. Prior art 1 obtains aggregated volumes 3 by projecting each 2D feature 2 onto a 3D cube according to camera parameters.

[0006] Prior art 1 obtains processed volumes 5 representing the likelihood of each joint by inputting aggregated volumes 3 into a V2V (neural network, P3) 4. Processed volumes 5 correspond to a heatmap representing the likelihood of each joint in 3D. Prior art 1 obtains 3D skeleton information 7 by performing soft-argmax 6 on processed volumes 5.

[0007] An example of V2V4 learning used in prior art 1 is illustrated. V2V4 mainly consists of a 3D kernel, and the kernel weights and biases are learned through error backpropagation. The kernel weights and biases are appropriately referred to collectively as "parameters". In prior art 1, the average value L of the joint position error is calculated based on the 3D skeleton information output from soft-argmax6 and the forward solution value. p As the loss value, the parameters are learned (adjusted) to minimize that loss value.

[0008] In the prior art 1, if the error of some joints increases due to occlusion, it is difficult to properly adjust the parameters of V2V4. For example, even if the error of some joints is large, if the error of other joints is small, then through averaging, the error of some joints will not have a significant impact on the output of the loss function. That is, the parameters of V2V4 do not need to be adjusted to improve the error of some joints.

[0009] Figure 12 This diagram illustrates an example of a bone structure with significant errors in certain joints. Figure 12 In the diagram, 3D skeleton information 10a corresponds to the 3D skeleton information of the positive solution value. 3D skeleton information 10b corresponds to the skeleton recognition result output from soft-argmax6. Figure 12 In the example shown, the error in the left ankle joint is larger, but since the errors in other joints are smaller, averaging them will not have a significant impact on the loss value. Therefore, it is not possible to expect any improvement in the learning effect of the left ankle position.

[0010] To address the aforementioned problem, there is prior art 2. In prior art 2, in the first stage, joint positions are inferred based on distance images; in the second stage, the inferred joint positions are evaluated for matching based on a human model, and the inferred joint positions are corrected to achieve the best combination of joint positions. Prior art 2 improves the accuracy of joint position inference by processing in two stages as described above.

[0011] Patent Document 1: Japanese Patent Application Publication No. 2015-167008

[0012] Patent Document 2: Japanese Patent Application Publication No. 2004-13474

[0013] Non-patent literature 1: K. Iskakov, et al., “Learnable Triangulation of Human Pose”, ICCV 2019, oral.

[0014] Non-patent literature 2: G. Moon, et al., “V2V-PoseNet: Voxel-to-Voxel PredictionNetwork for Accurate 3D Hand and Human Pose Estimation from a Single DepthMap”, CVPR 2018

[0015] In the prior art 2 described above, although the accuracy of joint position inference is improved, two stages of processing are required before the corrected joint position is output, which results in the problem of processing time required for bone recognition. Summary of the Invention

[0016] In one aspect, the object of the present invention is to provide a skeletal recognition device, learning method, and learning program capable of performing skeletal recognition results without spending processing time.

[0017] In the first embodiment, the skeleton recognition device includes a recognition processing unit and a loss function calculation unit. The recognition processing unit inputs teacher data into the learning model and calculates the output based on forward propagation. The loss function calculation unit calculates the value of the loss function based on the difference in bone length between the skeleton information (which serves as the forward solution value from the teacher data) and the output of the recognition processing unit, as well as the difference in angles between the bones based on the skeleton information and the output of the recognition processing unit.

[0018] It can generate skeleton recognition results without spending processing time. Attached Figure Description

[0019] Figure 1 This is a diagram illustrating an example of the system involved in this embodiment.

[0020] Figure 2 This is a functional block diagram illustrating the structure of the skeletal recognition device involved in this embodiment.

[0021] Figure 3 This is a diagram representing an example of a data structure for teacher data.

[0022] Figure 4 This is a diagram representing an example of a data structure for a human body model.

[0023] Figure 5 This is a diagram used to illustrate the processing during the learning phase.

[0024] Figure 6 This is a flowchart illustrating the processing steps in the preprocessing stage of a skeletal recognition device.

[0025] Figure 7 This is a flowchart illustrating the processing steps of the learning phase of a skeletal recognition device.

[0026] Figure 8 This is a flowchart illustrating the processing steps of the inference phase of a skeletal recognition device.

[0027] Figure 9 This is a diagram used to illustrate the effects of this embodiment.

[0028] Figure 10 This is a diagram illustrating an example of the hardware structure of a computer that performs the same function as the skeletal recognition device in the embodiment.

[0029] Figure 11 This is a diagram used to illustrate prior art 1.

[0030] Figure 12 This is a diagram illustrating an example of a bone with significant errors in some joints. Detailed Implementation

[0031] The following detailed description of embodiments of the skeletal recognition device, learning method, and learning program disclosed in this application is based on the accompanying drawings. However, the present invention is not limited to these embodiments.

[0032] Figure 1 This is a diagram illustrating an example of the system involved in this embodiment. For example... Figure 1 As shown, the system includes cameras 20a, 20b, and 20c, and a skeletal recognition device 100. Cameras 20a-20c and the skeletal recognition device 100 are interconnected wirelessly or via a wired connection.

[0033] exist Figure 1 In this system, cameras 20a to 20c are shown as an example, but other cameras may also be included. In the following description, cameras 20a to 20c are collectively referred to as camera 20.

[0034] Camera 20 is a device for capturing images of person 15. Camera 20 sends the captured image data to skeleton recognition device 100. In the following description, the image data of person 15 will be referred to as image data. As will be described later, image data is used in the inference phase of skeleton recognition device 100.

[0035] The skeleton recognition device 100 performs a "preprocessing stage" and a "learning stage" using teacher data to learn a learning model for skeleton recognition. Furthermore, in the "inference stage," the skeleton recognition device 100 outputs the skeleton recognition result of the person 15 based on the image data input from the camera 20 and the learned model.

[0036] Next, an example of the structure of the skeletal recognition device involved in this embodiment will be described. Figure 2This is a functional block diagram illustrating the structure of the skeletal recognition device involved in this embodiment. For example... Figure 2 As shown, the skeleton recognition device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0037] The communication unit 110 connects to external devices via wired or wireless means to transmit and receive information with the camera 20 and the external devices. For example, the communication unit 110 may be implemented using a NIC (Network Interface Card). The control unit 150 acquires image data from the camera 20 via the communication unit 110. The communication unit 110 may also be connected to a network (not shown).

[0038] The input unit 120 is an input device that inputs various information into the skeletal recognition device 100. The input unit 120 corresponds to a keyboard, mouse, touch panel, etc.

[0039] Display unit 130 is a display device that displays information output from control unit 150. Display unit 130 corresponds to liquid crystal display, organic EL (Electroluminescence) display, touch panel, etc.

[0040] The storage unit 140 includes teacher data 141, human body model 142, learning model 143, and image data table 144. The storage unit 140 is implemented, for example, by semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or storage devices such as hard disks and optical disks.

[0041] Teacher data 141 is the data used for learning model 143. Figure 3 This is a diagram illustrating an example of a data structure representing teacher data. For example... Figure 3 As shown, teacher data 141 has multiple sets of input data and positive resolution values. The input data is generated based on multiple pre-prepared image data, such as data corresponding to "aggregated volumes". The positive resolution values ​​are the positive resolution values ​​of the skeleton recognition results corresponding to the input data, such as information including the three-dimensional coordinates of each joint, the length of the bones connecting the joints, and the angles between the bones.

[0042] Human Model 142 is a predefined human model with multiple joints defined, and the joints are connected by bones. Figure 4 This is a diagram illustrating an example of a data structure representing a human body model. For example... Figure 4As shown, the human body model 142 has joints A0, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17, A18, A19, A20, and A21. Additionally, 20 bones connecting the joints are defined in the human body model 142.

[0043] Learning model 143 corresponds to a neural network (NN) such as V2V. For example, learning model 143 consists of a 3D kernel in which parameters (weights, biases, etc.) are set. These parameters are adjusted through the learning phase described later.

[0044] Image data table 144 is a table that records the image data output from camera 20 during the "inference phase". For example, image data table 144 establishes and maintains a correspondence between the recognition information of camera 20 and the image data captured by camera 20.

[0045] Return to Figure 2 The control unit 150 includes a recognition processing unit 151, a loss function calculation unit 152, a learning unit 153, and an image data processing unit 154. The control unit 150 can be implemented, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Alternatively, the control unit 150 can also be executed by an integrated circuit such as an ASIC (Application-Specific Integrated Circuit) or a FPGA (Field-Programmable Gate Array).

[0046] The control unit 150 executes the following processes in sequence: (1) preprocessing stage, (2) learning stage, and (3) inference stage. For ease of explanation, the "learning stage" will be explained first, followed by the "preprocessing stage", and finally the "inference stage".

[0047] The "processing during the learning phase" performed by the control unit 150 is explained. Figure 5 This diagram illustrates the processing during the learning phase. When the control unit 150 performs the preprocessing phase, the identification processing unit 151, the loss function calculation unit 152, and the learning unit 153 perform the following processing.

[0048] The processing of the recognition processing unit 151 during the learning phase will be explained. The recognition processing unit 151 inputs the input data contained in the teacher data 141 into the learning model 143, and calculates the output based on forward propagation (skeleton recognition result p = (p1, p2, ..., p...). Np "p1, p2, ..., p Np "This represents each joint, and sets its three-dimensional coordinates." Figure 5 In the example shown, in the skeleton recognition result p, p is used n To represent a certain joint.

[0049] More specifically, the recognition processing unit 151 calculates the likelihood of each joint in 3D by inputting input data into the learning model 143, and calculates the skeleton recognition result p by performing soft-argmax to find the centroid of the heatmap.

[0050] The recognition processing unit 151 outputs the skeleton recognition result to the loss function calculation unit 152. The recognition processing unit 151 calculates the skeleton recognition result corresponding to each input data by repeatedly performing the above processing on each input data contained in the teacher data 141, and outputs it to the loss function calculation unit 152.

[0051] The processing of the loss function calculation unit 152 during the learning phase will be explained. The loss function calculation unit 152 performs the processing of the error in calculating the length of the bone as the loss value, the processing of the error in calculating the angle between the bones as the loss value, and the processing of the combined loss value.

[0052] The process of "processing the error in calculating bone length as a loss value" performed by the loss function calculation unit 152 will be explained. The loss function calculation unit 152 calculates the bone vector b = (b1, b2, ..., b...) by establishing a correspondence between each joint of the bone recognition result p and each joint of the human body model 142. Np ).exist Figure 5 In the example shown, in the bone vector b, b is used n To represent a certain bone vector.

[0053] The loss function calculation unit 152 calculates the bone length l = (l1, l2, ..., l) based on the bone vector b. Np ).exist Figure 5 In the example shown, in the length l of the bone, l is used n It is used to represent the length of a certain bone.

[0054] The loss function calculation unit 152 calculates the loss value of the bone length based on equation (1) according to the difference between the bone length l and the positive solution value (positive solution value of bone length) of the teacher data 141. The loss value of the bone length is denoted as "L". lIn equation (1), N l Indicates the number of bones (e.g., N) l =20). l pred i It is the length of the bone inferred from the bone vector b, and it is the length of the i-th bone. gt i is the length of the bone with the positive solution value, and is the length of the i-th bone.

[0055] [Formula 1]

[0056]

[0057] The process of "calculating the error in the angle between bones as a loss value" performed by the loss function calculation unit 152 will be explained. Np ).exist Figure 5 In the example shown, in the angle l between the bones, l is used n It indicates the angle between two bones.

[0058] The loss function calculation unit 152 calculates the loss value of the angle between the bones based on equation (2) according to the difference between the angle r between the bones and the positive solution value (positive solution value of the angle between the bones) of the teacher data 141. The loss value of the angle between the bones is denoted as "L". r In equation (2), N r The number representing the angle between bones. pred i It is the angle between bones inferred from the bone vector b, and it is the angle between the i-th bone. gt i is the angle between the bones in the positive solution, and is the angle between the i-th bone.

[0059] [Equation 2]

[0060]

[0061] The "processing of the overall loss value" performed by the loss function calculation unit 152 will be explained. The loss function calculation unit 152 calculates the joint coordinate loss value based on the skeleton recognition result p and the positive solution values ​​(three-dimensional coordinates of each joint) of the teacher data 141. The joint coordinate loss value is denoted as "L". p Although the formula is omitted, L p This is the value obtained by averaging the errors of the three-dimensional coordinates of each joint.

[0062] The loss function calculation unit 152 synthesizes the loss value based on equation (3). The synthesized loss value is denoted as "L". The loss function calculation unit 152 outputs the loss value L to the learning unit 153. The w included in equation (3) l w r It is for L l L r Instead of fitting, the weights are pre-calculated through a "preprocessing stage".

[0063] [Formula 3]

[0064] L = L p +w l L l +w r L r …(3)

[0065] The loss function calculation unit 152 repeatedly performs the above processing whenever it obtains the skeleton recognition result p from the recognition processing unit 151.

[0066] However, the loss function calculation unit 152 can also use techniques such as Kinematic Chain Space when calculating the length of bones and the angles between bones. For example, Kinematic Chain Space is described in "B. Wandt, et al., "A kinematic chain space for monocular motion capture", (ECCV Workshops, 2018). In the following description, Kinematic Chain Space will be referred to as "KCS". In KCS, a matrix B summing the bone vector b is calculated. Matrix B is shown in equation (4).

[0067] [Formula 4]

[0068] B = (b1, b2, ..., b) b (4)

[0069] The loss function calculation unit 152 calculates the transpose matrix B of the representation matrix B. T The feature Ψ is the product of B. The feature Ψ is represented by Equation (5). The diagonal component of the feature Ψ corresponds to the length of the bone, and the off-diagonal component corresponds to the angle between the bones.

[0070] [Formula 5]

[0071]

[0072] The loss function calculation unit 152 calculates the average value L of the difference between feature Ψ and the positive solution value (GT) of teacher data 141 using equation (6).KCS In equation (6), Ψ pred ij The element in row i and column j of expression (6). Ψ gt ij Is with Ψ pred ij The elements corresponding to the positive solution values. B corresponds to the number of elements in matrix B.

[0073] [Formula 6]

[0074]

[0075] The above description illustrates the use of techniques such as Kinematic ChainSpace in calculating bone length and the angles between bones, but it is not a limitation. The loss function calculation unit 152 can also use Euclidean distance to calculate bone length. The loss function calculation unit 152 can also use cosine similarity to calculate the angles between bones.

[0076] The processing of the learning unit 153 in the learning phase is explained. The learning unit 153 repeatedly performs the process of learning (adjusting) the parameters of the learning model 143 through error backpropagation to minimize the loss value L. When the loss value L has converged, the learning unit 153 ends the adjustment of the parameters of the learning model 143.

[0077] During the learning phase, as shown in equation (3), due to the use of "L" which is related to the error considering the length of human bones. l "L" is related to the error in the angle between the bones. r The loss value L is calculated using a method called "L = ", thus increasing the contribution of the loss value L to cases where some joints deviate significantly. Therefore, when the error is large in some joints, it is possible to perform learning to improve such errors.

[0078] Next, the "preprocessing stage" performed by the control unit 150 will be explained. In the preprocessing stage, the w used in the learning stage is calculated. l w r During the learning phase, when using equation (3) to synthesize the Loss value, in order to L l L r However, the fitting is flawed, based on the joint coordinate loss value "L p "Loss value of bone length" L l The relative value of "w" is used to calculate "w". l "Based on the joint coordinate loss value "L" p "Loss value of bone length" L r The relative value of "w" is used to calculate "w". r ".

[0079] When the control unit 150 performs the preprocessing stage, the identification processing unit 151, the loss function calculation unit 152, and the learning unit 153 perform the following processing.

[0080] The processing of the recognition processing unit 151 in the preprocessing stage will be described. The processing of the recognition processing unit 151 in the preprocessing stage is the same as that of the recognition processing unit 151 in the learning stage. That is, the recognition processing unit 151 inputs the input data contained in the teacher data 141 into the learning model 143 and calculates the output based on forward propagation (skeleton recognition result p = (p1, p2, ..., p...). Np The recognition processing unit 151 outputs the skeleton recognition result p to the loss function calculation unit 152.

[0081] The processing of the loss function calculation unit 152 in the preprocessing stage will be explained. The processing of the loss function calculation unit 152 is the same as that in the learning stage, calculating the loss value "L" of the bone length. l "L" represents the loss value of the angle between the bones. r "Joint coordinate loss value "L p ".

[0082] Here, the loss function calculation unit 152 only calculates the joint coordinate loss value "L". p The output is sent to the learning unit 153. The loss function calculation unit 152 repeatedly performs the above processing whenever it obtains the skeleton recognition result p from the recognition processing unit 151.

[0083] The loss function calculation unit 152 receives the joint coordinate loss value "L" from the learning unit 153. p Given the information indicating convergence, calculate the joint coordinate loss value "L" respectively. p "The moment of convergence" L l “L” r “L” p The order of magnitude of "L". l The order of magnitude of " is set to "o". Ll ". "L" r The order of magnitude of " is set to "o". Lr ". "L" p The order of magnitude of " is set to "o". Lp ".

[0084] For example, in L p =4.0×10 －3 In this case, the loss function calculation unit 152 is "o Lp =10 －3 Regarding "o" Ll “o”Lp The loss function calculation unit 152 also calculates it in the same way.

[0085] The loss function calculation unit 152 calculates the weight w based on equation (7). l The weight w is calculated based on equation (8). r Such a weight w l w r It is utilized during the learning phase. The weight w is determined by the loss function calculation unit 152 through the above processing. l w r Thus L l L r Never exceed L p The order of magnitude, therefore enabling L l L r However, it is not fitted, and L l L r It has an appropriate impact on learning.

[0086] [Formula 7]

[0087]

[0088] [Formula 8]

[0089]

[0090] The processing of the learning unit 153 in the preprocessing stage will be explained. The learning unit 153 repeatedly performs error backpropagation to learn (adjust) the parameters of the learning model 143 to improve the loss value L. p Minimization processing. At the loss value L p Upon convergence, the learning unit 153 terminates the adjustment of the parameters of the learning model 143.

[0091] In the loss value L p In the case of convergence, the learning department 153 will set the loss value L. p The message indicating convergence has been output to the loss function calculation unit 152, and the learning process ends.

[0092] Next, the "inference stage" performed by the control unit 150 will be explained. When the control unit 150 performs the preprocessing stage, the recognition processing unit 151 and the image data processing unit 154 perform the following processing.

[0093] The processing of the recognition processing unit 151 in the inference stage will be described. The recognition processing unit 151 acquires input data from the image data processing unit 154. The recognition processing unit 151 inputs the acquired input data into the learned learning model 143 and performs forward propagation-based output (the skeleton recognition result of the person 15). The recognition processing unit 151 may also output the skeleton recognition result to the display unit 130 for display, or it may output the skeleton recognition result to an external device that performs various processing using the skeleton recognition result.

[0094] The recognition processing unit 151 repeatedly performs the above processing whenever it receives input data from the image data processing unit 154.

[0095] The processing of the image data processing unit 154 in the inference stage will be described. The image data processing unit 154 acquires image data from the camera 20 and stores the acquired image data in the image data table 144. The image data processing unit 154 determines the 2D features representing each joint feature by applying 2D backbone processing to the image data. The image data processing unit 154 calculates aggregated volumes by projecting each 2D feature onto a 3D cube according to camera parameters, and outputs the calculated aggregated volumes as input data to the recognition processing unit 151.

[0096] Furthermore, the image data processing unit 154 described above calculates input data based on each image data as an example. Other techniques can also be used to combine multiple image data to calculate input data. The image data processing unit 154 acquires image data from the camera 20 at regular intervals and repeatedly performs the above processing.

[0097] Next, an example of the processing steps of the skeleton recognition device 100 involved in this embodiment will be described. Figure 6 This is a flowchart illustrating the preprocessing steps of a skeletal recognition device. For example... Figure 6 As shown, the recognition processing unit 151 of the skeleton recognition device 100 inputs the input data of the teacher data 141 into the learning model 143 (step S101). The recognition processing unit 151 calculates the skeleton recognition result based on the forward propagation of the learning model 143 (step S102).

[0098] The loss function calculation unit 152 of the skeleton recognition device 100 calculates the error of the joint coordinates as the loss value (step S103). The loss function calculation unit 152 calculates the error of the bone length and the error of the angle between the bones as the non-loss value (step S104).

[0099] The learning unit 153 of the skeleton recognition device 100 adjusts the parameters of the learning model 143 based on error backpropagation to minimize the loss value (step S105).

[0100] The loss function calculation unit 152 calculates the weights w based on the loss value and the non-loss value. l w r (Step S106). The loss function calculation unit 152 outputs the weights w. l w r (Step S107).

[0101] Figure 7 This is a flowchart illustrating the processing steps of the learning phase of a skeletal recognition device. For example... Figure 7 As shown, the recognition processing unit 151 of the skeleton recognition device 100 inputs the input data of the teacher data 141 into the learning model 143 (step S201). The recognition processing unit 151 calculates the skeleton recognition result based on the forward propagation of the learning model 143 (step S202).

[0102] The loss function calculation unit 152 of the skeleton recognition device 100 calculates the errors in joint coordinates, bone length, and the angles between bones as loss values ​​(step S203). The loss function calculation unit 152 uses weight w l w r To synthesize the Loss value (step S204).

[0103] The learning unit 153 of the skeleton recognition device 100 adjusts the parameters of the learning model 143 based on error backpropagation to minimize the loss value (step S205). The learning unit 153 outputs the learned model 143 (step S206).

[0104] Figure 8 This is a flowchart illustrating the processing steps of the inference phase of a skeletal recognition device. For example... Figure 8 As shown, the image data processing unit 154 of the skeleton recognition device 100 acquires image data from the camera 20 (step S301).

[0105] The image data processing unit 154 generates input data based on multiple image data (step S302). The recognition processing unit 151 of the skeleton recognition device 100 inputs the input data into the learned model 143 after it has been learned (step S303).

[0106] The recognition processing unit 151 calculates the skeleton recognition result based on the forward propagation of the learning model 143 (step S304). The recognition processing unit 151 outputs the skeleton recognition result (step S305).

[0107] Next, the effects of the skeletal recognition device 100 involved in this embodiment will be explained. The skeletal recognition device 100 inputs teacher data into the learning model 143 to calculate the skeletal recognition result. By comparing the skeletal recognition result with the positive value of the teacher data, it calculates the error in bone length and the error in the angle between bones, and calculates the loss value of the loss function. By adjusting the parameters of the learning model 143 to minimize the loss value of this loss function, it can improve the contribution to the loss value when some joints deviate significantly, and can learn to improve such errors when the error in some joints is large. Furthermore, by performing this learning, the skeletal recognition result of the person 15 can be obtained in one stage of processing—inputting input data into the learned learning model 143—which improves the processing speed compared to the prior art 2, which performs processing in two stages.

[0108] More specifically, the skeleton recognition device 100 calculates the joint coordinate error, joint coordinate error, and joint coordinate error respectively by comparing the positive values ​​with the teacher's data, and learns the parameters of the learning model 143 with high precision by calculating the loss function loss value.

[0109] The skeletal recognition device 100 calculates the weight w based on equation (7). l And calculate the weight w based on equation (8). r Such a weight w l w r It is utilized during the learning phase. This is achieved by determining the weights w. l w r Joint coordinate error L l Joint coordinate error L r It will never exceed the joint coordinate error L. p The order of magnitude, therefore enabling L l L r However, it is not fitted, and L l L r It has an appropriate impact on learning.

[0110] Figure 9 This diagram is used to illustrate the effects of this embodiment. Figure 9 In the first embodiment, skeletal recognition result 30a is the skeletal recognition result of prior art 1, showing an abnormal angle of the left knee. Skeletal recognition result 30b is the skeletal recognition result of the skeletal recognition device 100 involved in this embodiment, showing that the angle of the left knee has been normalized.

[0111] Next, an example of a computer hardware structure that performs the same function as the skeletal recognition device 100 shown in the above embodiment will be described. Figure 10This is a diagram illustrating an example of the hardware structure of a computer that performs the same function as the skeletal recognition device in the embodiment.

[0112] like Figure 10 As shown, the computer 200 includes a CPU 201 that performs various arithmetic operations, an input device 202 that accepts data input from the user, and a display 203. Additionally, the computer 200 includes a reading device 204 that reads programs from storage media, and a communication device 205 that exchanges data with a camera 20, other external devices, etc., via a wired or wireless network. Furthermore, the computer 200 includes RAM 206 for temporary storage of various information and a hard disk device 207. All devices 201 to 207 are connected to a bus 208.

[0113] The hard disk device 207 includes an identification processing program 207a, a loss function calculation program 207b, a learning program 207c, and a learning data processing program 207d. Additionally, the CPU 201 reads each of the programs 207a to 207d and expands them in RAM 206.

[0114] The recognition processing procedure 207a functions as recognition processing step 206a. The loss function calculation procedure 207b functions as loss function calculation step 206b. The learning procedure 207c functions as learning step 206c. The image data processing procedure 207d functions as image data processing step 206d.

[0115] The processing of recognition processing step 206a corresponds to the processing of recognition processing unit 151. The processing of loss function calculation step 206b corresponds to the processing of loss function calculation unit 152. The processing of learning step 206c corresponds to the processing of learning unit 153. The processing of image data processing step 206d corresponds to the processing of image data processing unit 154.

[0116] Furthermore, each program 207a to 207d does not need to be stored on the hard disk device 207 from the beginning. For example, each program can be stored on a "removable physical medium" such as a floppy disk (FD), CD-ROM, DVD, optical disk, or IC card that is inserted into the computer 200. Moreover, the computer 200 can also read and execute each program 207a to 207d.

[0117] Explanation of reference numerals in the attached figures

[0118] 100…Skeleton recognition device; 110…Communication unit; 120…Input unit; 130…Display unit; 140…Storage unit; 141…Teacher data; 142…Human model; 143…Learning model; 144…Image data table; 150…Control unit; 151…Recognition processing unit; 152…Loss function calculation unit; 153…Learning unit; 154…Image data processing unit.

Claims

1. A skeleton recognition device, characterized in that, have: The recognition and processing unit inputs teacher data into the learning model and calculates the output based on forward propagation. The teacher data contains skeletal information representing the position of multiple bones in a person as the positive solution value of the teacher data. The loss function calculation unit calculates the difference in joint positions based on the skeletal information and the output of the recognition processing unit. It calculates a first weight based on the difference in joint positions and the difference in bone length based on the skeletal information and the output of the recognition processing unit. It calculates a second weight based on the difference in joint positions and the difference in angles between the multiple bones based on the skeletal information and the output. The unit calculates the value of the loss function by combining the value of the difference in joint positions, the value obtained by multiplying the value of the difference in bone length by the first weight, and the value obtained by multiplying the value of the difference in angles between the bones by the second weight. The learning department adjusts the parameters of the learning model based on the value of the aforementioned loss function.

2. A skeleton recognition learning method, which is a computer-executed learning method, characterized in that, Perform the following processing: The teacher data is input into the learning model and the output based on forward propagation is calculated. The teacher data has skeletal information representing the position of multiple bones in a person as the positive solution value of the teacher data. The difference in joint positions based on the aforementioned skeletal information and the aforementioned output results is calculated. A first weight is calculated based on the difference in joint positions and the difference in bone lengths based on the aforementioned skeletal information and the aforementioned output results. A second weight is calculated based on the difference in joint positions and the difference in angles between the aforementioned multiple bones based on the aforementioned skeletal information and the aforementioned output results. The loss function is calculated by combining the value of the difference in joint positions, the value obtained by multiplying the value of the difference in bone lengths by the value obtained by multiplying the value of the difference in angles between the aforementioned bones by the value obtained by multiplying the value of the difference in angles between the aforementioned bones by the value obtained by the second weight. The parameters of the learning model are adjusted based on the value of the loss function.

3. A non-transitory computer-readable recording medium, characterized in that, The computer stores a learning program that causes it to perform the following processes: The teacher data is input into the learning model and the output based on forward propagation is calculated. The teacher data has skeletal information representing the position of multiple bones in a person as the positive solution value of the teacher data. The difference in joint positions based on the aforementioned skeletal information and the aforementioned output results is calculated. A first weight is calculated based on the difference in joint positions and the difference in bone lengths based on the aforementioned skeletal information and the aforementioned output results. A second weight is calculated based on the difference in joint positions and the difference in angles between the aforementioned multiple bones based on the aforementioned skeletal information and the aforementioned output results. The loss function is calculated by combining the value of the difference in joint positions, the value obtained by multiplying the value of the difference in bone lengths by the value obtained by multiplying the value of the difference in angles between the aforementioned bones by the value obtained by multiplying the value of the difference in angles between the aforementioned bones by the value obtained by the second weight. The parameters of the learning model are adjusted based on the value of the loss function.

Images