Image analysis device and image analysis method
The image analysis device uses a smartphone to accurately determine front-back and vertical movements by analyzing skeletal position changes, overcoming the need for specialized devices and improving movement differentiation.
Patent Information
- Application Number
- JP2023079588
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-05-12
AI Technical Summary
Existing systems require specialized devices like laser radar for estimating a person's skeleton and posture, making it difficult to use a single imaging device such as a smartphone for accurate movement determination, particularly distinguishing between front-back and vertical movements.
An image analysis device that analyzes moving image data to determine movements in the front-back and vertical directions using a smartphone, employing a skeletal estimator to identify changes in skeletal positions and trigonometric calculations to differentiate between these movements.
Enables accurate determination of movements in the front-back direction using a single photographing device, even when complex movements occur, by analyzing skeletal position changes and correcting for depth variations.
Smart Images

Figure 2024163735000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image analysis device and an image analysis method. [Background technology]
[0002] In recent years, systems that use artificial intelligence technology to estimate a person's bone structure and posture have been attracting attention, and the results of such estimations are being used for providing guidance on sports and other exercises.
[0003] Patent Document 1 discloses a judgment system that calculates a target person's skeleton, joints, and past movement trajectory based on the detection results of a detection device that detects the distance and direction to an object, such as a laser radar device, and judges the target person's play based on the calculation results. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2020-31406 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, the technology disclosed in Patent Document 1 has a problem in that a special device such as a laser radar device is required as a detection device for detecting the distance and direction to an object. Therefore, it is difficult to easily estimate a person's bone structure and posture using a single image capture device such as a smartphone. In particular, when a single image capture device is used, it is difficult to distinguish between a person's forward / backward movement and upward / downward movement (jump).
[0006] An object of the present disclosure is to provide an image analysis device and an image analysis method that are capable of appropriately determining forward and backward movement of a subject even when a single imaging device is used. [Means for solving the problem]
[0007] An image analysis device according to one aspect of the present disclosure is an image analysis device that analyzes the movement of a subject captured in video data, and has an estimation unit that analyzes the video data and, when an amount of change in the vertical direction in a first judgment part of the subject exceeds a certain value, estimates that a judgment target movement has occurred that includes at least one of forward / backward movement and upward / downward movement of the subject, and a judgment unit that, when the judgment target movement has occurred, determines whether the judgment target movement includes forward / backward movement by determining, based on the video data, whether a forward / backward change ratio, which is the ratio of change in a judgment distance defined by a plurality of second judgment parts of the subject before and after the target movement, is equal to or greater than a predetermined value. Effect of the Invention
[0008] According to the present invention, even when a single image capturing device is used, it is possible to appropriately determine the forward and backward movement of a subject. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a block diagram illustrating a functional configuration of a skeleton position estimation correction device according to an embodiment of the present disclosure. [Diagram 2] FIG. 11 is a diagram for explaining an example of a skeleton position estimation process. [Diagram 3] FIG. 13 is a diagram showing an example of a skeleton position estimation result. [Figure 4] FIG. 11 is a diagram showing an example of skeleton position time-series data. [Diagram 5] 11A and 11B are diagrams for explaining an example of a change in a skeleton position estimation result. [Figure 6] 10A and 10B are diagrams illustrating an example of the relationship between the backward movement distance of a subject and the reduction ratio of the subject within a frame. [Figure 7] FIG. 11 is a diagram for explaining an example of a determination distance. [Figure 8] 11 is a diagram for explaining calculation of a moving distance of a subject in the forward / backward direction; FIG. [Figure 9] 11 is a diagram for explaining a process performed by an acceleration calculation unit. FIG. [Figure 10] 11 is a diagram for explaining a process performed by an acceleration calculation unit. FIG. [Figure 11] FIG. 13 is a diagram for explaining an example of compound movement. [Figure 12] FIG. 13 is a diagram illustrating an example of compound movement. [Figure 13] 10 is a flowchart illustrating an example of a process performed by the skeleton position estimation correction device. [Figure 14] 11 is a flowchart illustrating an example of a forward / backward movement determination process. [Figure 15] 10 is a flowchart illustrating an example of a travel distance estimation process. [Figure 16] 10 is a flowchart illustrating an example of an initial velocity calculation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below do not limit the invention according to the claims, and all of the elements and combinations thereof described in the embodiments are not necessarily essential to the solution of the invention. EXAMPLES
[0011] FIG. 1 is a block diagram illustrating a functional configuration of a skeleton position estimation and correction device according to an embodiment of the present disclosure.
[0012] The skeletal position estimation and correction device shown in Fig. 1 is an image analysis device that analyzes the movement of a subject captured in video image data, and includes a camera 100, an image analysis unit 101, a skeletal estimation unit 102, a trigonometric calculation unit 103, an acceleration calculation unit 104, and a depth correction unit 105. The skeletal position estimation and correction device is realized, for example, by a device with a camera function such as a smartphone. The skeletal position estimation and correction device also includes, for example, a processor (computer) and a memory (both not shown), and each of the components and functions described below may be realized, for example, by the processor reading a computer program and executing the read program.
[0013] The camera 100 is a photographing device that photographs a subject who is exercising or working. A plurality of people may be photographed as the subject. For example, the subjects may include an instructor 106 or a trained worker, and a user 107 who moves by imitating the instructor's exercise or work.
[0014] The image analysis unit 101 generates video image data showing a subject from the results of shooting by the camera 100, and outputs the video image data.
[0015] The skeleton estimation unit 102 is an estimation unit that analyzes the video data output from the image analysis unit 101 and generates skeleton position time series data, which is time series data that estimates the transition of the positions of multiple parts of the subject. Hereinafter, the parts whose transitions are estimated by the skeleton estimation unit 102 may be called skeleton parts. The method of estimating the positions of the skeleton parts is not particularly limited, and examples thereof include a method using machine learning such as OpenPose. In addition, the skeleton position estimation process for estimating the positions of the skeleton parts may be performed by batch processing or real time processing. In addition, when there are multiple subjects, the subjects may be divided into subjects whose skeleton position estimation process is performed by batch processing and subjects whose skeleton position estimation process is performed by real time processing. For example, the skeleton position estimation process may be performed by batch processing for the instructor, and the skeleton position estimation process may be performed by real time processing for the user. The results of chronological estimation of the positions of major joints such as the skeleton and parts of the human body are stored in a skeleton position estimation result time series data set 108 on a memory or an external storage device.
[0016] Furthermore, the skeleton estimation unit 102 judges whether or not the vertical change amount, which is the amount of change in the vertical direction in a predetermined first judgment part of the subject, exceeds a certain value based on the skeleton position time series data. When the vertical change amount exceeds a certain value, the skeleton estimation unit 102 estimates that a judgment target movement including at least one of a movement in the front-back direction and a movement (jump) in the vertical direction of the subject has occurred. In this embodiment, the first judgment part is the lowest part of the subject. For example, the skeleton estimation unit 102 may determine the position of the toe or heel of the subject as the lowest part. Furthermore, the vertical change amount is, for example, the amount of change between two frames with different timestamps (two consecutive frames in this embodiment).
[0017] The trigonometry calculation unit 103 is a determination unit that reads data from the skeleton position estimation result time series dataset 108, and when a movement to be determined occurs, determines whether or not the movement to be determined includes a movement in the forward / backward direction, based on the skeleton position time series data generated by the skeleton estimation unit 102. The trigonometry calculation unit 103 may also estimate the movement distance in the forward / backward direction.
[0018] The acceleration calculation unit 104 is a calculation unit that reads data from the skeleton position estimation result time series data set 108, and when a movement to be determined occurs, determines whether or not the movement to be determined includes a movement in the vertical direction, based on the skeleton position time series data generated by the skeleton estimation unit 102. The acceleration calculation unit 104 may also estimate the distance of movement in the vertical direction.
[0019] When the movement to be judged includes both forward / backward movement and upward / downward movement, the depth correction unit 105 corrects the forward / backward movement distance estimated by the trigonometric calculation unit 103 based on the skeleton position time series data generated by the skeleton estimation unit 102.
[0020] 2 is a diagram for explaining an example of a skeletal position estimation process by the skeleton estimation unit 2. In the skeletal position estimation process, the skeleton estimation unit 102 estimates, for each frame, a skeleton position, which is the position of each of a plurality of skeletal parts of the subject in that frame, in pixel units. In this embodiment, the horizontal direction in the frame is the X direction, the vertical direction (height direction) is the Z direction, and the depth direction is the Y direction, and the skeleton position is indicated by the X coordinate and the Z coordinate.
[0021] Also, numbers and names are set in advance for the skeletal parts from which the skeletal positions are estimated. In the example of Fig. 2, the numbers and names of the skeletal parts are as follows: 0 Nose, 1 Neck, 2 R Shoulder, 3 REbow, 4 R Wrist, 5 L Shoulder, 6 L Ebow, 7 L Wrist, 8 Mid Hip, 9 R Hip, 10 R Knee, 11 R Ankle, 12 L Hi p (left buttock), 13LKnee (left knee), 14LAnkle (left ankle), 15REye (right eye), 16LEye (left eye), 17REar (right ear), 18LEar (left ear), 19LBigToe (left big toe), 20LSmallToe (left little toe), 21LHeel (left heel), 22RBigtoe (right big toe), 23RSmallToe (right little toe), 24RHeel (right heel).
[0022] 3 is a diagram showing an example of a skeleton position estimation result, which is a detection result of the skeleton position estimation process in a single frame. A skeleton position estimation result 300 shown in FIG.
[0023] Field 301 stores a number for identifying a skeletal part. Field 302 stores the name of the skeletal part. Fields 303 and 304 store the skeletal position, which is the position of the skeletal part. Specifically, field 303 stores the X coordinate of the skeletal position, and field 304 stores the Z coordinate of the skeletal position. Field 305 stores the reliability (accuracy) indicating the estimation accuracy of the skeletal position. Note that not all skeletal positions are necessarily included in the skeletal position estimation result 300. For example, when the subject puts his / her right hand behind his / her back, the right wrist, etc., are hidden behind the body. For such skeletal parts not photographed by camera 100, the position of the skeletal part is not estimated, and the values of the X coordinate and Z coordinate of the skeletal part may not be stored. In this case, the reliability is 0. However, the skeletal position of a skeletal part not photographed by camera 100 may be estimated from the skeletal position of another skeletal part.
[0024] 4 is a diagram showing an example of skeleton position time series data 400. The skeleton position time series data 400 shown in FIG.
[0025] Field 401 stores the shooting time when the frame was shot as a timestamp. Field 402 stores the skeletal position and reliability of each skeletal part in the frame of the timestamp. In the example of FIG. 4, field 402 includes fields 402_0 to 402_24 that store the skeletal position and reliability of each of skeletal parts 0 to 24 shown in FIG. 2. Note that in video data, the number of frames contained in one second is determined by the shooting speed. When the shooting speed is 24 fps, 24 frames can be shot per second, and when the shooting speed is 30 fps, 30 frames can be shot per second.
[0026] 5 is a diagram for explaining an example of a change in a skeleton position estimation result when a subject moves forward or backward. In the example of FIG. 5, it is assumed that the subject moves backward.
[0027] In the example of Fig. 5, the height 502 of triangle 501 connecting the three points of skeletal parts 1Neck, 13LKnee, and 10RKnee in subject 500 before moving backwards is 54.3 pixels, and the height 502 of the same triangle 501 in subject 500 after moving backwards is 48.2 pixels. In other words, when the subject moves backwards, the subject in the frame is reduced in size, and the reduction ratio of the subject in the frame changes according to the distance the subject moves backwards. On the other hand, when the subject jumps upwards, the size of the subject does not change.
[0028] FIG. 6 is a diagram for explaining an example of the relationship between the backward movement distance of the subject and the reduction ratio of the subject in the frame. In the example of FIG. 6, the distance L between the subject 500 and the camera 100 before the movement is 200 cm, the backward movement distance l of the subject 500 is 30 cm, and the height H of the subject is 170 cm. If the ratio of the subject's height before and after the movement is H:(H+h), h is h=(170×(200+30)) / 200-170=25.5 cm, so the subject 500 appears to be reduced by about 13% after the movement. Although humans cannot recognize such a difference in size, the depth, which is the distance to the subject, is determined by the difference in the field of view of both eyes. On the other hand, in this embodiment, the forward and backward movement of the subject is determined from a subtle change in size of the subject 500 that humans cannot recognize.
[0029] Specifically, when a movement to be judged occurs, the trigonometry calculation unit 103 determines whether the movement to be judged includes movement in the forward / backward direction by determining whether the front-to-back change ratio, which is the ratio of change in the judgment distance defined by multiple second judgment parts of the subject before and after the movement to be judged, is greater than or equal to a predetermined value based on the skeletal position time series data.
[0030] The second judgment part may be, for example, a skeletal part in which the change in the judgment distance is small when a person jumps, and may be a predetermined skeletal part in which the judgment distance does not change by a specified value or more in a unit time before the judgment target movement, or a predetermined number of skeletal parts selected from those skeletal parts in which the change in the judgment distance is large. The judgment distance is, for example, the height of a triangle formed by the three skeletal parts. The trigonometric calculation unit 103 judges that the judgment target movement includes a movement in the forward and backward directions when a front-back change ratio, which is a change ratio of the judgment distance, is equal to or greater than a predetermined value.
[0031] FIG. 7 is a diagram for explaining an example of the determination distance.
[0032] When exercising or working, the subject may perform complex movements such as bending the body or lifting one leg high. For this reason, if the judgment distance is calculated from a single combination of multiple skeletal parts (for example, the triangle connecting the three points of 1Neck, 13LKnee, and 10RKnee described above), accurate judgment may not be possible. Therefore, in this embodiment, as shown in FIG. 7, the trigonometric calculation unit 103 selects multiple combinations of three skeletal parts as the second judgment part, and sets the average value of the change ratio of the judgment distance for each combination as the before-after change ratio used for judgment. At this time, the trigonometric calculation unit 103 may calculate the standard deviation σ of the change ratio of each judgment distance, and exclude the judgment distance of the combination whose change ratio is an outlier. For example, the trigonometric calculation unit 103 may exclude the judgment distance that is out of ±2σ.
[0033] In the example of FIG. 7, the second determination portion and the determination distance are set as follows. 700: Nose-heel correction (Height1: Height of the triangle formed by 0Nose, 21Lheel and 24RHeel) 701: Neck and heel correction (Height2: Height of the triangle formed by 1Neck, 21Lheel and 24Rheel) 702: Nose-knee correction (Height3: Height of the triangle formed by 0Nose, 13Lknee and 10RKnee) 703: Neck-knee correction (Height4: Height of the triangle formed by 1Neck, 13Lknee and 10RKnee) 704: Shoulder and knee correction (Height5: Average height of triangle made by 5Lshoulder, 13Lknee and 10R Knee and triangle made by 2Rshoulder, 13Lknee and 10R Knee)
[0034] FIG. 8 is a diagram for explaining calculation of the movement distance of the subject in the forward / backward direction.
[0035] 8(a), the elevation angle of the line connecting camera 100 and the lowest point of the subject is set to 0 degrees. In this case, the change ratio of the determination distance before and after the movement is 1:x, and if the distance between the subject and camera 100 before the movement is L as described above, then the backward movement distance l of the subject is l=L / xL.
[0036] As shown in FIG. 8(b), whether the elevation angle of the line connecting the camera 100 and the lowest point of the subject is positive or negative, the movement distance l is similarly expressed as l=L(H+h) / HL.
[0037] 9 and 10 are diagrams for explaining the processing by the acceleration calculation unit 104. FIG.
[0038] When the vertical change amount exceeds a certain value and the front-to-back change ratio is less than a specified value, it is considered that the subject only moved up and down without moving forward and backward. However, there are actions that result in a body position similar to a jump, which is a vertical movement. There are various types of jumps, such as a jump where only the knees are raised and a jump where both heels are stretched upwards, but FIG. 9 shows a V-shaped split jump 900 and a split-leg forward bend 901, which can sometimes result in a body position similar to the V-shaped split jump 900.
[0039] Although the V-shaped split jump 900 and the split forward bend 901 are clearly different movements, they may result in similar body positions, which may lead to erroneous determination. It is considered that the Y coordinates of the heels and knees increase after the V-shaped split jump 900 movement, and the Y coordinates of the heels and knees decrease after the split forward bend 901 movement. Therefore, by detecting this difference, it is possible to determine whether or not a jump has been made. However, in this embodiment, in order to make a more accurate determination, the acceleration calculation unit 104 determines whether or not a jump has been made based on the initial movement, which is the movement before the jump.
[0040] As shown in Fig. 10, before a human jumps against the gravitational acceleration of the earth, he or she crouches down to some extent to create a reaction force, that is, shortens the distance between the waist (or buttocks) and the toes (or heels) as an initial movement. The acceleration calculation unit 104 calculates the speed of this initial movement as the initial velocity, and judges whether or not the subject has jumped from the initial velocity.
[0041] The gravitational acceleration on Earth is g = 9.80665 (m / s 2 ) and the relationships between velocity, acceleration, time, displacement and initial velocity during the jump are as follows: Velocity (m / s) = Acceleration (m / s 2 ) x time (s) + initial velocity (m / s) Displacement (m) = Initial velocity (m / s) x Time (s) + (1 / 2) x Acceleration (m / s 2 ) × time (s) 2 Speed (m / s) 2 -Initial velocity (m / s) 2 = 2 x acceleration x displacement
[0042] Therefore, for example, in order for a human to jump 10 cm (0.1 m), an initial velocity of √(2×9.8×0.1)=1.4 m / s is required. For example, when video data is captured at a shooting speed of 30 frames per second, a frame is captured at a rate of about once every 33 ms. In this case, if the hips or knees move upward by 1.4 m÷30 frames=4.67 cm between frames in which the initial movement is started, the initial velocity is equivalent to 1.4 m / s. In the present embodiment shown in FIG. 5, etc., 4.67 cm corresponds to about 2.54 pixels. If the unit time is 0.1 seconds, the change in the skeletal position during the unit time of about 0.1 seconds can be measured by comparing the nth frame with the n+3th frame.
[0043] 11 is a diagram for explaining an example of a compound movement that is a movement to be determined that includes both forward and backward movement and upward and downward movement. In FIG. 11, a compound movement in which a user jumps backwards in a parabolic curve is shown.
[0044] 11, the subject steps back, increasing the Z coordinate and appearing to shrink. Also, when the subject jumps, the Z coordinate increases in the first half of the jump (until the highest point is reached), but decreases in the second half of the jump (from the highest point to landing). In this case, depth correction unit 105 performs the following process.
[0045] The change in depth (forward / backward movement distance l) immediately before the start and end of a jump can be calculated using trigonometry. The timing of the start of a jump is the frame in which the Z coordinate of the subject's lowest point increases, and the timing of the end of the jump is the frame in which the Z coordinate of the subject's lowest point stops decreasing. The initial speed of the jump can be found by comparing the frame several frames before the start of the jump with the frame at the time of the jump, but if the jump is not vertical, the jump angle θ must be taken into account. In other words, the initial speed V of the jump is calculated by multiplying the initial speed V in the horizontal (depth) direction by the initial speed V y and the initial vertical velocity V zIn the method of measuring the velocity in a few frames before the heel is released, the only thing that can be observed by skeletal measurement using AI technology is the initial vertical velocity V z , so it is unclear how much kick is being produced horizontally.
[0046] Initial velocity V and initial horizontal velocity V y , initial vertical velocity V z and the jump angle θ are expressed by the following equation. V z = Vsinθ V y = Vcosθ
[0047] The distance traveled in the forward and backward directions, l, is calculated by the initial horizontal velocity, V y and the jump time (t) product l=V y t. Since the horizontal distance l and the jump time t can be calculated, the initial horizontal velocity Vy is V y = l / t. On the other hand, the initial vertical velocity V z The height h can also be calculated as h=V z 2 This can be calculated as / 2g.
[0048] However, in the case of compound movement, since the jump is accompanied by a backward movement, it is necessary to take into account that the height on the Z axis shrinks according to the distance traveled on the Y axis. z Once this is determined, the acceleration due to gravity is constant, so the height h that can be jumped is constant; however, if you jump vertically in the same place, it can be observed that you have jumped to a height of h, but if you jump backwards in a parabolic trajectory, the height h will be observed to be reduced by the amount that you jump backwards.
[0049] FIG. 12 is a diagram showing an example of compound movement. In the example of FIG. 12, subject 500 of height H jumps backwards from a position distance L away from camera 100 and lands at a position a distance l below. Here, height H of subject 500 is the length from the lowest point of the subject to the top of the head. Also, the height of the jump is assumed to be h. In this case, since the subject is moving backwards, the jump height is observed to be shorter than h. For example, when L=200cm, l=30cm, H=160cm, and H=40cm, z1: The distance from the ground to the top of the head when a subject of height H jumps to height h on the spot z2: When a subject with height H jumps a height of h while descending a distance l, the distance from the ground to the top of the head when the subject descends a distance l / 2 to the highest point is z3: Distance from the ground to the top of the head of the subject at height H z4: Distance from the ground to the top of the head when a subject of height H steps back by distance l z5: The distance from the ground to the feet of a subject of height H who jumps to height h on the spot z6: When a subject with height H jumps a height h while descending a distance l, the distance from the ground to the feet when the subject reaches the highest point by descending a distance l / 2. Then, z1=h+H=200cm z2=L(h+H) / (L+l / 2)=148.84cm z3=H=160cm z4=HL / (L+l)=160*200 / (200+30)=139.13cm z5=h=40cm z6=hL / (L+l / 2)=40*200 / (200+30 / 2)=37.2cm It becomes.
[0050] Therefore, if you jump in place, the Z coordinate will increase to a height of z1 (200 cm), but in the example in Figure 12, because you have moved back a distance of l / 2, the actually observed Z coordinate will appear to shrink to z2 (148.84 cm). Similarly, if you are just standing there, you will be observed to be at height H, or z3 (160 cm), but if you move back a distance of l, you will be observed to have shrunk to z4 (139.13 cm). Furthermore, if you jump in place by a height of h, the Z coordinate of your feet will increase by height h, or z5 (40 cm), but because you have moved back a distance of l / 2, the actually observed Z coordinate will appear to have shrunk to z6 (37.2 cm).
[0051] FIG. 13 is a flowchart illustrating an example of processing performed by the skeleton position estimation and correction device.
[0052] First, the camera 100 captures an image of a subject. Then, the image analysis unit 101 generates and outputs video data of the subject from the image captured by the camera 100 (step S1301). The subject here includes the instructor and the user.
[0053] The skeleton estimation unit 102 extracts frames from the video data output from the image analysis unit 101, estimates the position of the skeleton of the subject appearing in the frames, and generates a skeleton position estimation result (step S1302).The skeleton estimation unit 102 then adds the generated skeleton position estimation result to the skeleton position time series data and stores it (step S1303).However, if real-time performance is required, the skeleton estimation unit 102 loads the skeleton position time series data in memory.
[0054] The skeleton estimation unit 102 specifies the relationship between the actual size of the subject and the size in the video data (step S1304). For example, the skeleton estimation unit 102 specifies a size correspondence relationship between the actual values (cm) of the body dimensions of each skeletal part of the subject and the number of pixels of the video data based on the height of the subject and the size (number of pixels) of the subject calculated from the skeletal position time-series data. The body dimensions include height, sitting height, crotch height, head circumference, head width, neck circumference, shoulder width, bust, waist, hand length, wrist circumference, hand width, foot length, foot width, thigh circumference, and calf circumference, and the body dimensions of a standard build can be obtained based on statistical information from the height. The height of the subject may be set in advance, or if not set, the average height may be used.
[0055] Skeleton estimation unit 102 measures the position of the lowest point of the subject in each of the target frame n and the next frame n+1, and calculates the difference between these positions in the Y direction as the amount of vertical change (step S1305).
[0056] The skeleton estimation unit 102 determines whether the vertical change amount exceeds a certain value (step S1306). If the vertical change amount does not exceed the certain value, the skeleton estimation unit 102 returns to the process of step S1305 with the next frame n+1 as the target frame. On the other hand, if the vertical change amount exceeds the certain value, the skeleton estimation unit 102 estimates that a determination target movement of the subject has occurred (step S1307).
[0057] When the subject moves to be determined, trigonometry calculation unit 103 executes forward / backward movement determination processing (see FIG. 14) for determining whether the subject has moved forward / backward (step S1308). If the subject has moved forward / backward, trigonometry calculation unit 103 executes movement distance estimation processing (see FIG. 15) for estimating the forward / backward movement distance of the subject (step S1309).
[0058] After that, the acceleration calculation unit 104 executes an initial velocity calculation process (FIG. 16) for calculating an initial vertical velocity that is an initial velocity in the vertical direction of the judgment target movement (step S1310). Then, the acceleration calculation unit 104 determines whether the initial vertical velocity exceeds a reference value (step S1311).
[0059] If the initial vertical velocity exceeds the reference value, acceleration calculation unit 104 determines that the subject has jumped (step S1312). On the other hand, if the initial vertical velocity does not exceed the reference value, acceleration calculation unit 104 determines that the subject has not jumped (step S1313).
[0060] When step S1312 is completed, the depth correction unit 105 may correct the forward / backward movement distance estimated by the trigonometric calculation unit 103 using the method described with reference to FIGS.
[0061] FIG. 14 is a flowchart for explaining an example of the forward / backward movement determination process in step S1304 of FIG.
[0062] In the forward / backward movement determination process, the trigonometric calculation unit 103 calculates a plurality of determination distances D1 to D2 defined by a plurality of second determination parts in the target frame n. k (Step S1400). The trigonometric calculation unit 103 calculates a plurality of determination distances d1 to d k (Step S1401). The trigonometric calculation unit 103 calculates the change ratios C1 to C2 of the determination distance for each determination distance. k (Step S1402). That is, the change ratio C i =d i / D i Next, the trigonometric calculation unit 103 calculates the change ratios C1 to C k The standard deviation σ is calculated (step S1403).
[0063] The trigonometric calculation unit 103 removes outliers from the change ratios C1 to Ck of the judgment distance based on the standard deviation σ (step S1404). For example, the trigonometric calculation unit 103 removes change ratios that deviate by ±σ or more as outliers. Then, the skeleton estimation unit 102 calculates the average value of the change ratios C1 to Ck excluding the outliers as the front-back change ratio (step S1405).
[0064] Trigonometry calculation unit 103 judges whether the subject has moved forward or backward by determining whether the absolute value of the front-to-back change ratio is equal to or greater than a predetermined value (step S1406). At this time, if the absolute value of the front-to-back change ratio is equal to or greater than the predetermined value and the average value is negative, it can be determined that the subject has moved backward, and if the absolute value of the front-to-back change ratio is equal to or greater than the predetermined value and the front-to-back change ratio is positive, it can be determined that the subject has moved forward.
[0065] FIG. 15 is a flowchart illustrating an example of the travel distance estimation process in step S1308 of FIG.
[0066] The trigonometric calculation unit 103 determines that the target movement has ended when the change in the front-back change ratio has stopped, and starts a movement distance estimation process to estimate the movement distance due to the target movement based on the skeleton position time-series data from the target frame n to the frame where it is determined that the target movement has ended. Note that if the subject is not moving forward or backward, the process of step S1308 may be skipped.
[0067] Specifically, trigonometry calculation unit 103 sets distance L from camera 100 to the subject (step S1500). Distance L may be specified by the user, or a predetermined standard value (e.g., 200 cm) may be used.
[0068] Next, trigonometry calculation unit 103 sets the height H of the subject (step S1501). The height H may be specified by the user, or a predetermined standard value (for example, 165 cm) may be used. Note that height H may be the height or the length between major parts of the body.
[0069] The trigonometric calculation unit 103 sets the height h seen by the camera 100 after moving forward or backward (step S1502). The trigonometric calculation unit 103 calculates the change ratio H Δ =H / h (step S1503). Then, assuming that the distance is l, the trigonometric calculation unit 103 calculates l=(H Δ -1) L (step S1504).
[0070] FIG. 16 is a flowchart for explaining an example of the initial velocity calculation process in step S1309 of FIG.
[0071] First, the acceleration calculation unit 104 calculates the distance L between the waist (or buttocks) and the toes (or heels) in the target frames n and n-1. n Next, the acceleration calculation unit 104 measures the above-mentioned distance L in frame n+k, which is a predetermined number of frames ahead of the target frame n. n+k The acceleration calculation unit 104 measures the difference in distance d k =L n+k -L n is calculated (step S1601).
[0072] The acceleration calculation unit 104 calculates the difference in distance d k is negative (d k <0) (step S1602). k If the difference in distance d is not negative, the acceleration calculation unit 104 returns to the process of step S1601. k If is negative, the acceleration calculation unit 104 determines that the subject's waist is beginning to sink (is beginning to move downward), and calculates the current distance difference d k is the difference in distance from the previous k-1 Less than (d k <d k-1 ) (step S1603). k-1 d k-1 =L n+k-1 -L n-1 d k <d k-1 If not, the acceleration calculation unit 104 returns to the process of step S1601.
[0073] d k <d k-1 In this case, the acceleration calculation unit 104 determines that the subject's waist has reached the lowest point and n+k The frame advanced by a specified number n+k+m The above distance L n+k+m The acceleration calculation unit 104 measures the difference in distance d m =L n+k+m -L n+k is calculated (step S1604).
[0074] The acceleration calculation unit 104 calculates the difference in distance d m is positive (d m >0) (step S1605). m If the difference in distance d is not positive, the acceleration calculation unit 104 returns to the process of step S1604. m If is positive, the acceleration calculation unit 104 determines that the subject's waist has started to move upward, and calculates the initial velocity V Z Specifically, the acceleration calculation unit 104 calculates the initial velocity V Z V Z =d m Calculated from ×F / m.
[0075] Then, the acceleration calculation unit 104 calculates the initial velocity V Z is a specific speed V Δ If the gravitational acceleration is g, the jump height h is expressed as h=V z 2 Here, if the gravitational acceleration g is 9.8 (m / s2) and the subject is defined as jumping when it is 0.05 mm or more above the ground, then the specific speed V Δ is (0.05 x 2g) 1 / 2 ≒0.99.
[0076] As described above, according to this embodiment, the skeleton estimation unit 102 analyzes the video data, and when the amount of change in the vertical direction in the first determination portion of the subject exceeds a certain value, it estimates that a determination target movement including at least one of forward / backward movement and vertical movement of the subject has occurred. When a determination target movement has occurred, the trigonometric calculation unit 103 determines whether or not the determination target movement includes forward / backward movement by determining whether or not a forward / backward change ratio, which is a change ratio of a determination distance defined by a plurality of second determination portions of the subject before and after the target movement, is equal to or greater than a predetermined value based on the video data. Therefore, even when a single imaging device is used, it is possible to appropriately determine the forward / backward movement of the subject.
[0077] In addition, in this embodiment, the trigonometric calculation unit 103 estimates the movement distance of the subject in the forward and backward directions based on the change ratio of the determination distance before and after the target movement, which makes it possible to estimate the movement distance more accurately.
[0078] In this embodiment, the determination distance is the height of a triangle formed by the three determination portions, which makes it possible to more appropriately determine the forward and backward movement of the subject.
[0079] In this embodiment, the trigonometric calculation unit 103 selects a plurality of combinations of three determination parts and calculates the average value of the change ratio of the determination distance for each combination as the front-rear change ratio. This makes it possible to more appropriately determine the forward-rear movement of the subject.
[0080] In this embodiment, the acceleration calculation unit 104 judges whether the movement to be judged includes an upward movement based on position time-series data that estimates the transition of the positions of multiple parts of the subject before the judgment target movement occurs. This makes it possible to appropriately judge the upward movement (jump) of the subject.
[0081] In addition, in this embodiment, when the target movement includes both forward and backward movement and upward movement, the depth correction unit 105 corrects the forward and backward movement distance based on the position time series data, which makes it possible to estimate the movement distance more accurately.
[0082] The above-described embodiments of the present disclosure are illustrative examples of the present disclosure, and are not intended to limit the scope of the present disclosure to only these embodiments. A person skilled in the art may implement the present disclosure in various other forms without departing from the scope of the present disclosure. [Explanation of symbols]
[0083] 100: Camera 100 101: Image analysis unit 102: Skeleton estimation unit 103: Trigonometry calculation unit 104: Acceleration calculation unit 105: Depth correction unit 106: Instructor 107: User 108: Skeleton position estimation result time series data set
Claims
1. An image analysis device that analyzes a motion of a subject captured in moving image data, comprising: an estimation unit that analyzes the video data and estimates that a determination target movement including at least one of a forward / backward movement and an upward / downward movement of the subject has occurred when an amount of change in a vertical direction of the first determination portion of the subject exceeds a certain value; The image analysis device has a judgment unit that, when the movement to be judged occurs, judges whether the movement to be judged includes movement in the forward / backward direction by determining whether a front-to-back change ratio, which is the ratio of change in the judgment distance defined by a plurality of second judgment parts of the subject before and after the object movement, is equal to or greater than a predetermined value based on the video image data.
2. The image analyzing device according to claim 1 , wherein the determining unit estimates the movement distance in the front-rear direction based on the front-rear change ratio.
3. The image analyzing device according to claim 1 , wherein the determination distance is a height of a triangle formed by three of the second determination portions.
4. The image analysis device according to claim 3 , wherein the determination unit selects a plurality of combinations of three of the second determination portions, and calculates an average value of a change ratio of the determination distance for each of the combinations as the front-to-rear change ratio.
5. The image analysis device according to claim 2, further comprising a calculation unit that, when the movement of the object to be determined occurs, determines whether the movement of the object to be determined includes an upward movement based on position time series data that estimates the change in position of multiple parts of the subject before the movement of the object to be determined occurs.
6. The image analysis device according to claim 5 , further comprising a correction unit that corrects the forward / backward movement distance based on the position time series data when the movement to be determined includes both the forward / backward movement and the upward movement.
7. 1. An image analysis method using an image analysis device that analyzes a motion of a subject captured in moving image data, comprising: analyzing the video data, and when an amount of change in the vertical direction of the first determination portion of the subject exceeds a certain value, estimating that a determination target movement including at least one of a forward / backward movement and a vertical movement of the subject has occurred; An image analysis method in which, when the object movement occurs, based on the video image data, it is determined whether the ratio of change in the judgment distance defined by multiple second judgment parts of the subject before and after the object movement is equal to or greater than a predetermined value, thereby determining whether the object movement includes movement in the forward / backward direction.
Citation Information
Patent Citations
Driver state estimation device and driver state estimation method
JP2018151931A
Fitness support method and electronic device
JP2022546453A
Full skeletal 3D pose recovery from monocular camera
WO2022043834A1
Determination system and determination method
JP2020031406A
