Annotation method and annotation apparatus

The annotation method and device address the issue of blur in training machine learning models by generating datasets that consider blur intensity and direction, enhancing joint position estimation accuracy and adaptability in varying image conditions.

JP2024157448APending Publication Date: 2024-11-07NISSAN MOTOR CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023071826
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-25
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Existing methods for training machine learning models do not account for the effect of blur in input images, which affects the accuracy of joint position estimation.

Method used

An annotation method and device that calculates blur intensity and direction in images using motion sensors to generate a learning dataset, incorporating blur information for training models, allowing them to handle blurred images effectively.

Benefits of technology

The method enables accurate joint position estimation in blurred images, improving model performance and adaptability to varying image quality and subject visibility, even when parts of the body are obscured or not fully captured.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024157448000001_ABST
    Figure 2024157448000001_ABST
Patent Text Reader

Abstract

To provide an annotation method and an annotation apparatus capable of generating a learning dataset so as to train a machine learning model according to levels and directions of blurs included in an image.SOLUTION: A processor 10 of an annotation apparatus 100 is configured to: acquire a training image V of a target person P captured using an image sensor 2; acquire joint positions of joints, joint moving speeds, and joint moving directions in the joint positions, on the basis of results detected by a motion sensor 3 which detects motions of one or more body parts of the target person P; calculate blur levels in the joint positions in the training image on the basis of the joint moving speeds; calculate blur directions D in the joint positions in the training image on the basis of the joint moving directions; calculate a training certainty distribution of an estimated joint position on the basis of the joint positions, the blur levels, and the blur directions D; and generate a training dataset including the training image and the training certainty distribution.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an annotation method and an annotation device. [Background technology]

[0002] US Patent Publication No. 2009 / 0139996 describes a learning method for training a machine learning model that identifies desired regions of interest from an input image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 059446 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method described in Patent Document 1 does not take into consideration the effect that blurring contained in the input image has on the output data.

[0005] The problem to be solved by the present invention is to provide an annotation method and annotation device capable of generating a learning dataset so as to train a machine learning model according to the intensity and direction of blur contained in an image. [Means for solving the problem]

[0006] The present invention solves the above problem by calculating a training confidence distribution of estimated joint positions based on the joint positions in training images, and the blur intensity and blur direction of the joint positions, and generating a training data set including the training images and the training confidence distribution. Effect of the Invention

[0007] According to the present invention, a learning data set can be generated so that a machine learning model can be trained according to the intensity and direction of blur contained in an image. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an annotation device according to an embodiment of the present invention. [Diagram 2] 2 is a diagram showing an example of how the image sensor and the motion sensor shown in FIG. 1 are attached to a human body. [Diagram 3] 3 is a diagram showing an example of a learning image acquired by the image sensor shown in FIG. 2. [Figure 4] 4A and 4B show the relationship between the training confidence distribution and exposure time, where FIG. 4(a) shows the training confidence distribution when the exposure time is 10 ms, and FIG. 4(b) shows the training confidence distribution when the exposure time is 20 ms. [Diagram 5] 2 is a diagram showing an example of an output image for learning generated by the annotation device shown in FIG. 1. [Figure 6] 2 is a flowchart showing the procedure of an annotation method executed by the annotation device shown in FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. 1, the annotation device 100 includes a processor 10. The annotation device 100 uses the processor 10 to generate a learning dataset as teacher data for training a machine learning model 1. The annotation device 100 also generates the learning dataset based on information acquired from an image sensor 2 and a motion sensor 3.

[0010] The machine learning model 1 is a model that outputs a certainty distribution of estimated joint positions of one or more joints of a subject based on an image (input image) of the subject. Specifically, the machine learning model 1 outputs an output image that displays an image of the subject by superimposing an image showing a certainty distribution of estimated joint positions on the image. The machine learning model 1 is, for example, a model that performs transfer learning on a posture detection model such as Openpose.

[0011] The image sensor 2 is an optical sensor such as a camera. The image sensor 2 captures an image of a part or the entire body of the training subject P. For example, as shown in FIG. 2, the image sensor 2 is a small camera attached to a helmet worn on the head of the training subject P. The image sensor 2 is not limited to this, and may be a camera installed near the training subject P or a camera built into a mobile terminal.

[0012] The motion sensor 3 is an acceleration sensor. As shown in FIG. 2, three motion sensors 3 are attached to each of both arms and both legs of the training subject P. Specifically, the motion sensors 3 are fixed to the upper arms, wrists, and backs of the hands of both arms of the training subject P with belts. The motion sensors 3 are also fixed to the thighs, lower legs, and tops of the feet of both legs of the training subject P with belts or fasteners. Furthermore, the motion sensors 3 (not shown) are fixed to the back, head, waist, and both shoulders of the training subject P with fasteners. In this embodiment, 17 motion sensors 3 are attached to the training subject P, but the number of motion sensors 3 is not limited to this. The motion sensors 3 may also be attached to the joint positions of both arms and both legs of the training subject P. The motion sensors 3 are sensors that detect the movement of one or more body parts of the training subject P. In this embodiment, an inertial motion sensor 3 is used to detect the movement of one or more body parts of the training subject P, but the present invention is not limited to this. For example, an optical sensor (camera) may be used to track the position of a marker on a body part of the training subject P. In addition, a sensor of another type may be used to detect the movement of one or more body parts of the training subject P.

[0013] The processor 10 includes a ROM (Read Only Memory) in which a program is stored, a CPU (Central Processing Unit) that executes the program stored in the ROM, and a RAM (Random Access Memory) that functions as an accessible storage device. The processor 10 includes an image acquisition unit 11, a motion information acquisition unit 12, a blur information calculation unit 13, a confidence distribution calculation unit 14, an optical flow calculation unit 15, and a learning data set generation unit 16. The processor 10 executes the functions of the image acquisition unit 11, the motion information acquisition unit 12, the blur information calculation unit 13, the confidence distribution calculation unit 14, the optical flow calculation unit 15, and the learning data set generation unit 16 by using the CPU to execute the program stored in the ROM.

[0014] The image acquisition unit 11 acquires a learning image V of the learning subject P captured by using the image sensor 2. In the example shown in FIG. 3, the learning image V is an image of both arms and face F of the learning subject P captured by the image sensor 2 located at the forehead of the learning subject P. The learning image V shows the learning subject P thrusting both arms forward and moving his / her hands. At this time, a blur B occurs in a part of both arms of the learning subject P captured in the learning image V. In addition, the learning image V shows body parts corresponding to the wrist joints K1 and elbow joints K2 of both arms. On the other hand, the body part (shoulder) corresponding to the shoulder joint K3 is hidden by the face F and does not appear in the learning image V. In addition, the joint positions of the ankle joints K4, knee joints K5, and hip joints K6 of both legs are located outside the learning image V and are not captured in the learning image V. In the example shown in Figure 3, the training subject P is standing with both legs together, so in the coordinate system of the training image V, the joint positions of the ankle joint K4, knee joint K5 and hip joint K6 are approximately the same. In addition, in the learning image V shown in FIG. 3, the motion sensor 3 attached to the body of the learning subject P and the belt for fixing the motion sensor 3 are omitted.

[0015] The motion information acquisition unit 12 acquires motion information of each joint of the learning subject P based on the detection result of the motion sensor 3 attached to the learning subject P. The motion information includes the joint position, the joint movement speed and the joint movement direction of each joint of the learning subject P. Specifically, the motion information acquisition unit 12 calculates the joint position, the joint movement speed and the joint movement direction of the joint position of the learning subject P by analyzing the detection result of the motion sensor 3 using mechanical analysis based on the skeletal information of the learning subject P. The joints from which motion information is acquired are the wrist joints K1, elbow joints K2 and shoulder joints K3 of both arms of the learning subject P, and the ankle joints K4, knee joints K5 and hip joints K6 of both legs.

[0016] The blur information calculation unit 13 includes a blur intensity calculation unit 13a and a blur direction calculation unit 13b. That is, the blur information calculation unit 13 calculates the blur intensity and blur direction D (see FIG. 3) of each joint position in the learning image V.

[0017] The blur intensity calculation unit 13a calculates the blur intensity of each joint position in the learning image V based on the exposure time of the image sensor 2 when capturing the learning image V and the joint movement speed of each joint position. The exposure time of the image sensor 2 may be set to a predetermined time in advance, or may be automatically set according to the brightness of the capturing environment. The blur intensity calculation unit 13a may calculate the blur intensity of each joint position based only on the joint movement speed of each joint position, regardless of the exposure time of the image sensor 2. The blur intensity is the distance when the movement distance of the joint position in three-dimensional space in a predetermined time (for example, the exposure time of the image sensor 2) is expressed on the two-dimensional coordinates of the learning image V.

[0018] The blur direction calculation unit 13b calculates the blur direction D of the joint position in the learning image V based on the joint movement direction of each joint position (see FIG. 3). Note that the blur direction D is the direction when the joint movement direction in the three-dimensional space is expressed on the two-dimensional coordinates of the learning image V.

[0019] The certainty distribution calculation unit 14 calculates a training certainty distribution A of the estimated joint position based on the joint position acquired by the motion information acquisition unit 12, the blur intensity calculated by the blur intensity calculation unit 13a, and the blur direction D calculated by the blur direction calculation unit 13b. The training certainty distribution A is training data indicating a distribution of certainty in an estimation range of the joint position. The range of the training certainty distribution A (the estimation range of the joint position) is an area including a coordinate position indicating the joint position in the coordinate system of the training image V. That is, the certainty distribution calculation unit 14 determines the coordinate position of the range of the training certainty distribution A (the estimation range of the joint position) in the training image V based on the joint position acquired by the motion information acquisition unit 12. The certainty distribution calculation unit 14 calculates the position of the range of the training certainty distribution A so that the joint position acquired by the motion information acquisition unit 12 coincides with the average coordinate position of the estimated joint position (the center position of the range of the training certainty distribution A).

[0020] FIG. 4(a) shows an example of the training confidence distribution A1 when the exposure time of the image sensor 2 is 10 [ms] and the moving speed of the joint position on the training image V is v [mm / ms]. At this time, the moving distance of the joint position on the training image V, i.e., the blur strength, is v×10 [mm]. The confidence distribution calculation unit 14 calculates the training confidence distribution A1 using the variance-covariance matrix M1 shown in the following formula (1). When the moving direction of the joint position is θ [rad], the direction is adjusted by multiplying it by a rotation matrix R of the angle θ. The coefficient k is adjusted according to the moving speed v.

number

[0021] On the other hand, Fig. 4(b) shows an example of the training confidence distribution A2 when the exposure time of the image sensor 2 is 20 [ms] and the moving speed of the joint position on the training image V is v [mm / ms], similar to the example shown in Fig. 4(a). At this time, the moving distance of the joint position on the training image V, i.e., the blur strength, is v × 20 [mm]. The confidence distribution calculation unit 14 calculates the training confidence distribution A2 using the variance-covariance matrix M2 shown in the following formula (2), which is obtained by doubling the coefficient k of the covariance matrix M1 in formula (1) at the reference exposure time of 10 [ms].

number

[0022] That is, as shown in the difference between the training confidence distribution A1 in FIG. 4(a) and the training confidence distribution A2 in FIG. 4(b), when the exposure time of the image sensor 2 is lengthened and the blur intensity increases by a constant factor (twice in the example shown in FIG. 4), the scale (standard deviation value) of the training confidence distribution A also increases by a constant factor (twice in the example shown in FIG. 4).

[0023] Moreover, the certainty distribution calculation unit 14 calculates the learning certainty distribution A so that the certainty of the estimated joint position increases as it approaches the center position (the joint position acquired by the motion information acquisition unit 12) of the range of the learning certainty distribution A. In addition, in Figs. 4(a) and (b), the learning certainty distributions A1 and A2 are displayed in a darker black color as the certainty increases. Moreover, as shown in Figs. 4(a) and (b), the learning certainty distributions A1 and A2 calculated as described above are both distributions with a substantially elliptical shape extending along the blur direction D. In other words, the blur strength and the blur direction D are parameters for calculating the learning certainty distribution A. When the training image V includes a plurality of training subjects P, the certainty distribution calculation unit 14 calculates the training certainty distribution A for each of the training subjects P.

[0024] Furthermore, the optical flow calculation unit 15 calculates the optical flow of the joint position based on the blur strength and the blur direction D, that is, a vector value indicating the movement of the joint position. When the learning image V includes a plurality of learning subjects P, the optical flow calculation unit 15 calculates the optical flow of the joint positions for each of the learning subjects P.

[0025] The training dataset generation unit 16 generates a training dataset including a training image V and a training confidence distribution A. Specifically, the training dataset generated by the training dataset generation unit 16 includes the training image V as input data for training. The training dataset also includes the training confidence distribution A as output data for training. That is, as shown in FIG. 5, the training dataset generation unit 16 generates a training output image V0 that displays images Z1, Z2, Z3, and Z4 indicating the training confidence distribution A superimposed on the training image V as output data for training.

[0026] The image Z1 shown in FIG. 5 shows the training certainty distribution A of the joint position of the wrist joint K1 of both arms. The image Z2 shows the training certainty distribution A of the joint position of the elbow joint K2 of both arms. The image Z3 shows the training certainty distribution A of the joint position of the shoulder joint K3 of both arms. The image Z4 shows the training certainty distribution A of the joint positions of the ankle joint K4, knee joint K5, and hip joint K6 of both legs. In addition, in the training output image V0, the joint positions of the ankle joint K4, knee joint K5, and hip joint K6 of both legs are almost the same. In addition, the images Z1, Z2, Z3, and Z4 have shapes that show the training certainty distribution A of each joint position. That is, the images Z2, Z3, and Z4 corresponding to the elbow joint K2, shoulder joint K3, ankle joint K4, knee joint K5, and hip joint K6 that do not have blur B in the training image V shown in FIG. 3 are approximately circular. On the other hand, the icon Z1 corresponding to the wrist joint K1 where the blur B occurs in the learning image V is approximately elliptical. The learning certainty distribution A is displayed by the shade or hue of the color displayed inside the icons Z1, Z2, Z3, and Z4. In the example shown in FIG. 5, the closer to the center position of each icon Z1, Z2, Z3, and Z4, the darker the color and the higher the certainty. Also, for example, the closer the color applied to the inside of the icons Z1, Z2, Z3, and Z4 is to a warm color (red), the higher the certainty, and the closer the color is to a cool color (blue), the lower the certainty.

[0027] Here, the training image V shown in FIG. 3 does not show the shoulders (body parts corresponding to the shoulder joint K3) of the training subject P, as they are hidden by the face F of the training subject P. In addition, the training image V does not show both legs (body parts corresponding to the joint positions of the ankle joint K4, the knee joint K5, and the hip joint K6) of the training subject P. On the other hand, the training output image V0 shown in FIG. 5 displays an image Z1 corresponding to the wrist joint K1 and an image Z2 corresponding to the elbow joint K2 that are shown in the training image V. In addition, the training output image V0 displays an image Z3 corresponding to the shoulder joint K3 that is not shown in the training image V. In addition, the training output image V0 also displays an image Z4 corresponding to the ankle joint K4, the knee joint K5, and the hip joint K6 that are not shown in the training image V. That is, when a body part including at least one joint position is not included in the learning image V, the learning data set generation unit 16 generates the learning output image V0 by displaying the images Z3 and Z4 related to the joint positions of the body part in the coordinate system of the learning image V, similarly to the images Z1 and Z2. That is, even when a part of a body part of the training subject P is not shown in the training image V because it is hidden by an obstruction or protrudes from the training image V, the images Z3 and Z4 related to the joint positions of the body part not shown in the training image V are displayed in the training output image V0. The learning data set generation unit 16 can identify the joint positions of the body parts not included in the training image V by converting the position information of the skeleton of the training subject P in the three-dimensional coordinates obtained by the mechanical analysis into position information in the two-dimensional coordinates in the training image V.

[0028] Furthermore, the learning dataset generation unit 16 displays, in the learning output image V0, a ​​movement trajectory R of the joint position based on the optical flow calculated by the optical flow calculation unit 15. That is, the learning dataset generated by the learning dataset generation unit 16 includes the learning output image V0 displaying the movement trajectory R based on the optical flow.

[0029] The machine learning model 1 trained by the learning dataset generated by the learning dataset generation unit 16 outputs an output image that displays an image showing the confidence distribution of the estimated joint positions superimposed on the input image, similar to the learning output image V0 shown in Figure 5.

[0030] Next, the procedure of the annotation method executed by the processor 10 of the annotation device 100 will be described with reference to the flowchart of FIG. First, in step S1, the image acquisition unit 11 acquires a learning image V of the learning subject P captured by using the image sensor 2. Next, in step S2, the motion information acquisition unit 12 acquires motion information of each joint based on the detection result of the motion sensor 3 attached to the learning subject P. The motion information includes the joint position, the joint movement speed of the joint position, and the joint movement direction.

[0031] Next, in step S3, the blur intensity calculation unit 13a calculates the blur intensity of each joint position in the learning image V based on the joint movement speed. Further next, in step S4, the blur direction calculation unit 13b calculates the blur direction D of each joint position in the learning image V based on the joint movement direction. The process of step S3 may be executed after the process of step S4, or the process of step S3 and the process of step S4 may be executed simultaneously.

[0032] Next, in step S5, the certainty distribution calculation unit 14 calculates a learning certainty distribution A based on the joint positions, the shaking strength, and the shaking direction. Further next, in step S6, the optical flow calculation unit 15 calculates the optical flow of the joint position based on the blur strength and the blur direction. The process of step S5 may be executed after the process of step S6, or the process of step S5 and the process of step S6 may be executed simultaneously.

[0033] Next, in step S7, the learning data set generation unit 16 generates a learning output image V0 indicating a learning certainty distribution A of the estimated joint positions and a movement trajectory R of the joint positions based on the optical flow. The process of step S1 may be executed after the processes of steps S2 to S7, or the process of step S1 and the processes of steps S2 to S7 may be executed in parallel with each other.

[0034] Furthermore, in step S9, the learning dataset generation unit 16 generates a learning dataset including the learning image V and the learning output image V0.

[0035] From the above, the annotation device 100 according to the present embodiment calculates the blur strength and blur direction of the joint position in the learning image V based on the motion information of the joint acquired based on the detection result of the motion sensor 3. Then, the annotation device 100 calculates the learning confidence distribution A of the estimated joint position based on the joint position, the blur strength, and the blur direction. Then, the annotation device 100 generates a learning data set including the learning image V and the learning confidence distribution A. Thereby, the annotation device 100 can generate a learning data set so that the machine learning model can be trained according to the strength and direction of the blur included in the input image. That is, the annotation device 100 can perform annotation taking into account the uncertainty caused by the blur included in the input image. Therefore, the machine learning model 1 trained using the learning data set generated by the annotation device 100 can accurately estimate the joint position of the subject even if the input image is blurred. That is, the machine learning model 1 can accurately estimate the joint position of the subject regardless of the performance of the image sensor (camera) that acquires the input image. In addition, since blurred learning images V can be mixed as noise into the learning data set generated by the annotation device 100, the machine learning model 1 can perform learning with varying degrees of difficulty, thereby improving the accuracy of estimating joint positions.

[0036] Furthermore, the annotation device 100 calculates the blur intensity based on the exposure time and the joint movement speed of the image sensor 2 when capturing the learning image V. As a result, the annotation device 100 can calculate a larger blur intensity as the exposure time of the image sensor 2 is longer, and can calculate the learning confidence distribution A with high accuracy.

[0037] Furthermore, when the training image V includes multiple training subjects P, the annotation device 100 calculates the training confidence distribution A for each training subject P. This allows the annotation device 100 to generate a training dataset even when multiple training subjects P appear in the training image V. Furthermore, the machine learning model 1 can improve the estimation accuracy of the joint positions by being trained using the training image V in which multiple training subjects P appear.

[0038] Furthermore, the annotation device 100 generates a learning dataset including a learning output image V0 in which images Z1, Z2, Z3, and Z4 indicating the learning confidence distribution A are superimposed on a learning image V. Here, the machine learning model 1 is a model that outputs an output image in which an image is superimposed with images indicating the confidence distribution. This allows the annotation device 100 to generate a learning dataset for training the machine learning model 1 that outputs an output image indicating the learning confidence distribution A.

[0039] Furthermore, when a body part including at least one joint position is not included in the learning image V, the annotation device 100 generates a learning output image V0 by displaying images Z3 and Z4 relating to the joint positions of the body parts not included in the learning image V on the coordinate system of the learning image V. This allows the annotation device 100 to perform annotation even when a body part corresponding to a joint part of the learning subject P is not shown in the learning image V. Therefore, the machine learning model 1 trained using the learning dataset generated by the annotation device 100 can estimate each joint position even when some of the subject's body parts are not shown in the input image.

[0040] Furthermore, the annotation device 100 calculates the optical flow of the joint position based on the blur strength and the blur direction, and generates a learning data set including a learning output image V0 displaying a movement trajectory R based on the optical flow. As a result, the machine learning model 1 trained using the learning data set can estimate the movement trajectory R based on the optical flow of each joint position based on one input image. Furthermore, when the image sensor 2 acquires a video of the subject as input data, the machine learning model 1 can estimate the movement trajectory R of each joint position regardless of the FPS (frame rate) of the image sensor 2. That is, even if the FPS (frame rate) of the image sensor 2 is low, the machine learning model 1 can smoothly sample the movement coordinates of the joint position based on the movement trajectory R of the joint position. In other words, the machine learning model 1 can achieve consistency of the joint position between each frame based on the joint position, the joint movement speed, and the joint movement direction. Furthermore, even if the image sensor 2 is an asynchronous network camera and the FPS (frame rate) is unstable, the machine learning model 1 can determine whether each frame contained in the video of the input data can be used for processing involving time-based actions such as gesture recognition by ensuring consistency of joint movement as described above.

[0041] In this embodiment, the annotation device 100 does not need to have the optical flow calculation unit 15. In other words, the learning data set generated by the annotation device 100 only needs to include the learning image V and the learning confidence distribution A, and the learning output image V0 does not need to include the movement trajectory R of the joint position based on the optical flow. [Explanation of symbols]

[0042] 100...Annotation device 1. Machine learning model 2. Image sensor 3...Motion sensor (a sensor that detects the movement of body parts) 10…Processor 11...Image acquisition section 12…Motion information acquisition section 13a... Shake strength calculation unit 13b... Shake direction calculation section 14...Confidence distribution calculation unit 16…Learning dataset generation section A…Learnable confidence distribution D…Shake direction R…Movement trajectory V…Learning image V0: Output image for learning Z1, Z2, Z3, Z4…Images

Claims

1. 1. A method of annotation for generating a learning dataset for training a machine learning model using a processor, comprising: the machine learning model is a model that outputs a confidence distribution of estimated joint positions of one or more joints of a subject based on an image of the subject; The processor, Acquire a learning image of the learning subject captured using an image sensor; acquiring a joint position of the joint, a joint movement speed and a joint movement direction of the joint position based on a detection result of a sensor that detects a movement of one or more body parts of the training subject; calculating a blurring intensity of the joint position in the learning image based on the joint movement speed; calculating a blur direction of the joint position in the learning image based on the joint movement direction; calculating a training certainty distribution of the estimated joint positions based on the joint positions, the blur magnitude, and the blur direction; generating the training dataset including the training images and the training belief distributions.

2. The processor, The annotation method according to claim 1 , further comprising: calculating the blur intensity based on an exposure time of the image sensor when capturing the learning image and the joint movement speed.

3. The processor, The annotation method according to claim 1 , wherein, when the training image includes a plurality of the training subjects, the training confidence distribution is calculated for each of the training subjects.

4. The machine learning model is a model that outputs an output image that displays an image showing the belief distribution superimposed on the image, The annotation method according to claim 1 , wherein the processor generates the training data set including a training output image in which the image indicating the training belief distribution is superimposed on the training image.

5. The processor, 5. The annotation method according to claim 4, wherein, if the training image does not include a body part including at least one of the joint positions, the training output image is generated by displaying the image relating to the joint position of the body part not included in the training image on the coordinate system of the training image.

6. The machine learning model is a model that outputs an output image that displays a movement trajectory of the estimated joint position at a predetermined time on the image, The processor, Calculating an optical flow of the joint position based on the blur strength and the blur direction; The annotation method according to claim 1 , further comprising generating the training data set including training output images that display the movement trajectory based on the optical flow.

7. 1. An annotation device that generates a learning dataset for training a machine learning model using a processor, comprising: the machine learning model is a model that outputs a confidence distribution of estimated joint positions of one or more joints of a subject based on an image of the subject; The processor, an image acquisition unit that acquires a learning image of a learning subject captured by an image sensor; a motion information acquisition unit that acquires a joint position of the joint, a joint movement speed of the joint position, and a joint movement direction of the joint based on a detection result of a sensor that detects a movement of one or more body parts of the training subject; a blur intensity calculation unit that calculates a blur intensity of the joint position in the learning image based on the joint movement velocity; a blur direction calculation unit that calculates a blur direction of the joint position in the learning image based on the joint movement direction; a certainty distribution calculation unit that calculates a learning certainty distribution of the estimated joint positions based on the joint positions, the blurring magnitude, and the blurring direction; and a training dataset generation unit that generates the training dataset including the training images and the training confidence distribution.

Citation Information

Patent Citations

  • Learning device and learning method

    WO2020059446A1