Image analysis device and image analysis method

Abstract

To provide an image analysis device that is able to properly determine forward / backward and upward / downward movement distances of a subject even when a single imaging device is used.SOLUTION: A single imaging device captures a subject 106, who is an instructor or a skilled operator engaged in motion, and a subject 107, who is also in motion. Both sets of imaging data are analyzed using technology such as artificial intelligence to estimate skeleton positions. Perspective differences in the displacement of the skeleton positions in specific parts are analyzed by trigonometry to properly determine forward / backward movement distances, while quick accelerations in the skeleton positions of the specific parts are detected to properly determine upward / downward movement distances during jumps and other actions. It is also applicable to correct grasping of differences in motion between the subjects 106 and 107.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an image analysis apparatus and an image analysis method.

Background Art

[0002] In recent years, systems for estimating a person's skeleton and posture using artificial intelligence technology and the like have attracted attention, and the estimation results are used for guidance of sports and other movements.

[0003] Patent Document 1 discloses a determination system that calculates a target person's skeleton, joints, and past movement trajectory based on the detection results of a detection device that detects the distance and direction to an object, such as a lidar device, and determines the play of the target person based on the calculation results.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technique disclosed in Patent Document 1 has a problem that a special device such as a lidar device is required as a detection device for detecting the distance and direction to an object. Therefore, it is difficult to easily estimate a person's skeleton and posture using a single imaging device. In particular, when using a single imaging device, it is difficult to distinguish between the forward and backward movement of a person and the vertical movement (jump).

[0006] An object of the present disclosure is to provide an image analysis apparatus and an image analysis method capable of appropriately discriminating the forward and backward movement of a subject even when using a single imaging device.

Means for Solving the Problems

[0007] An image analysis device according to one aspect of the present disclosure is an image analysis device for analyzing the movement of a subject captured in moving image data, comprising: an estimation unit that analyzes the moving image data and estimates that a target movement to be determined has occurred, which includes at least one of movement in the front-to-back direction and movement in the up-to-down direction of the subject, when the amount of change in the vertical direction of a first determination part of the subject exceeds a certain value; and a determination unit that, when the target movement to be determined has occurred, determines whether the target movement to be determined includes movement in the front-to-back direction by determining, based on the moving image data, whether the front-to-back change ratio, which is the ratio of change in the determination distance before and after the target movement defined by a plurality of second determination parts of the subject, is greater than or equal to a predetermined value. [Effects of the Invention]

[0008] According to the present invention, even when using a single imaging device, it becomes possible to appropriately determine the forward and backward movement of a subject. [Brief explanation of the drawing]

[0009] [Figure 1] This block diagram shows the functional configuration of a skeletal position estimation and correction device according to an embodiment of the present disclosure. [Figure 2] This is a diagram illustrating an example of skeletal position estimation processing. [Figure 3] This figure shows an example of the results of skeletal position estimation. [Figure 4] This figure shows an example of skeletal position time series data. [Figure 5] This figure illustrates an example of how skeletal position estimation results can change. [Figure 6] This diagram illustrates an example of the relationship between the distance a subject moves backward and the reduction ratio of the subject within the frame. [Figure 7] This is a diagram illustrating an example of a distance used for determination. [Figure 8] This diagram illustrates the calculation of the distance traveled by a subject in the forward and backward direction. [Figure 9] This is a diagram illustrating the processing performed by the acceleration calculation unit. [Figure 10] This is a diagram illustrating the processing performed by the acceleration calculation unit. [Figure 11] This is a diagram illustrating an example of compound movement. [Figure 12] This figure shows an example of a compound movement. [Figure 13] This is a flowchart illustrating an example of the processing performed by the skeletal position estimation and correction device. [Figure 14] This is a flowchart illustrating an example of forward / backward movement detection processing. [Figure 15] This is a flowchart illustrating an example of the distance estimation process. [Figure 16] This is a flowchart illustrating an example of the initial velocity calculation process. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below are not intended to limit the claims of the invention, and not all elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention. [Examples]

[0011] Figure 1 is a block diagram showing the functional configuration of a skeletal position estimation correction device according to an embodiment of the present disclosure.

[0012] The skeletal position estimation and correction device shown in Figure 1 is an image analysis device that analyzes the movement of a subject captured in video data, and includes a camera 100, an image analysis unit 101, a skeletal estimation unit 102, a trigonometric calculation unit 103, an acceleration calculation unit 104, and a depth correction unit 105. The skeletal position estimation and correction device can be implemented, for example, in a device with a camera function, such as a smartphone. The skeletal position estimation and correction device may also include, for example, a processor (computer) and memory (neither of which are shown), and each component and function described below may be implemented, for example, by the processor reading a computer program and executing the program it has read.

[0013] Camera 100 is a filming device that films a subject, which is a person performing an exercise or work. Multiple people may be filmed as subjects. For example, a supervisor, such as an instructor 106 or a skilled worker, and a user 107 who is mimicking the supervisor's movements or work may be filmed as subjects.

[0014] The image analysis unit 101 generates and outputs motion image data of the subject from the shooting results captured by the camera 100.

[0015] The skeleton estimation unit 102 is an estimation unit that analyzes the dynamic image data output from the image analysis unit 101 and generates skeleton position time series data, which is time series data that estimates the changes in the positions of multiple parts of the subject. Hereinafter, the parts whose changes are estimated by the skeleton estimation unit 102 may also be referred to as skeletal parts. The method for estimating the positions of skeletal parts is not particularly limited, but examples include methods using machine learning such as OpenPose. Furthermore, the skeleton position estimation process for estimating the positions of skeletal parts may be performed in batch processing or in real time processing. Also, if there are multiple subjects, some subjects may be subjected to skeleton position estimation processing in batch processing, and others to skeleton position estimation processing in real time processing. For example, skeleton position estimation processing may be performed in batch processing for instructors and in real time processing for users. The results of estimating the positions of major joints and parts of the human body, such as the skeleton, in time series are stored in the skeleton position estimation result time series dataset 108 on memory or an external storage device.

[0016] Furthermore, the skeletal estimation unit 102 determines, based on the skeletal position time-series data, whether the vertical change amount, which is the amount of vertical change at a predetermined first determination part of the subject, exceeds a certain value. If the vertical change amount exceeds a certain value, the skeletal estimation unit 102 estimates that a determination target movement has occurred, which includes at least one of movement in the front-to-back direction and movement in the up-and-down direction (jump) of the subject. In this embodiment, the first determination part is the lowest point of the subject. The skeletal estimation unit 102 may, for example, use the position of the subject's toes or heel as the lowest point. Also, the vertical change amount is, for example, the amount of change between two frames with different timestamps (in this embodiment, two consecutive frames).

[0017] The trigonometric calculation unit 103 reads data from the skeletal position estimation result time series dataset 108 and, when a movement of the object to be judged occurs, determines whether or not the movement of the object to be judged includes movement in the forward and backward direction, based on the skeletal position time series data generated by the skeletal estimation unit 102. The trigonometric calculation unit 103 may also estimate the distance of movement in the forward and backward direction.

[0018] The acceleration calculation unit 104 reads data from the skeleton position estimation result time series dataset 108 and, when a movement of the target object occurs, calculates whether or not vertical movement is included in that movement based on the skeleton position time series data generated by the skeleton estimation unit 102. The acceleration calculation unit 104 may also estimate the vertical movement distance.

[0019] The depth correction unit 105 corrects the forward-backward movement distance estimated by the triangulation calculation unit 103 based on the skeleton position time series data generated by the skeleton estimation unit 102, when the movement of the object to be judged includes both forward-backward movement and up-down movement.

[0020] Figure 2 is a diagram illustrating an example of the skeletal position estimation process by the skeletal position estimation unit 2. In the skeletal position estimation process, the skeletal position estimation unit 102 estimates the position of each of the multiple skeletal parts of the subject within each frame, on a pixel-by-pixel basis. In this embodiment, the horizontal direction within the frame is defined as the X direction, the vertical direction (height direction) as the Z direction, and the depth direction as the Y direction, and the skeletal position is indicated by the X and Z coordinates.

[0021] Furthermore, the skeletal parts whose positions are estimated are assigned numbers and names in advance. In the example in Figure 2, the numbers and names of the skeletal parts are: 0 Nose, 1 Neck, 2 Right Shoulder, 3 Right Elbow, 4 Right Wrist, 5 Left Shoulder, 6 Left Elbow, 7 Left Wrist, 8 Mid Hip, 9 Right Hip, 10 Right Knee, 11 Right Ankle, 12 Left Hip p (left hip), 13LKnee (left knee), 14LAnkle (left ankle), 15REye (right eye), 16LEye (left eye), 17REar (right ear), 18LEar (left ear), 19LBigToe (left big toe), 20LSmallToe (left little toe), 21LHeel (left heel), 22RBigtoe (right big toe), 23RSmallToe (right little toe), 24RHeel (right heel).

[0022] Figure 3 shows an example of the skeletal position estimation result, which is the detection result of the skeletal position estimation process in a single frame. The skeletal position estimation result 300 shown in Figure 3 includes fields 301 to 305.

[0023] Field 301 stores a number to identify the skeletal part. Field 302 stores the name of the skeletal part. Fields 303 and 304 store the skeletal position, which is the location of the skeletal part. Specifically, field 303 stores the X coordinate of the skeletal position, and field 304 stores the Z coordinate of the skeletal position. Field 305 stores the confidence level (accuracy) indicating the accuracy of the skeletal position estimation. Note that the skeletal position estimation result 300 does not necessarily include all skeletal positions. For example, if the subject puts their right hand behind their back, their right wrist and other parts may be hidden behind their body. For skeletal parts that are not captured by camera 100 in this way, the position of the skeletal part is not estimated, and the X and Z coordinate values ​​of the skeletal part do not need to be stored. In this case, the confidence level will be 0. However, the skeletal position of a skeletal part not captured by camera 100 may be estimated from the skeletal position of another skeletal part, for example.

[0024] Figure 4 shows an example of skeletal position time series data. The skeletal position time series data 400 shown in Figure 4 includes fields 401 to 402.

[0025] Field 401 stores the time the frame was captured as a timestamp. Field 402 stores the skeletal position and confidence level of each skeletal part in the frame with the given timestamp. In the example in Figure 4, field 402 includes fields 402_0 to 402_24, which store the skeletal position and confidence level of each of the skeletal parts 0 to 24 shown in Figure 2. Note that in video data, the number of frames included per second is determined by the shooting speed. If the shooting speed is 24fps, 24 frames can be captured per second, and if the shooting speed is 30fps, 30 frames can be captured per second.

[0026] Figure 5 illustrates an example of how the skeletal position estimation results change when the subject moves in the forward and backward directions. In the example in Figure 5, the subject is assumed to have moved backward.

[0027] In the example in Figure 5, the height 502 of the triangle 501 formed by connecting the three skeletal parts 1Neck, 13LKnee, and 10RKnee of subject 500 before it moves backward is 54.3 pixels, and the height 502 of the same triangle 501 of subject 500 after it moves backward is 48.2 pixels. In other words, when a subject moves backward, the subject in the frame shrinks, and the degree of shrinkage of the subject in the frame changes depending on the distance the subject moves backward. On the other hand, if the subject jumps upward, the size of the subject does not change.

[0028] Figure 6 illustrates an example of the relationship between the distance a subject moves backward and the reduction ratio of the subject within the frame. In the example in Figure 6, the distance L between subject 500 and camera 100 before movement is 200 cm, the distance l of subject 500 moving backward is 30 cm, and the height H of the subject is 170 cm. If the ratio of the subject's height before and after movement is H:(H+h), then h = (170 × (200 + 30)) / 200 - 170 = 25.5 cm, so subject 500 appears to be reduced by approximately 13% after movement. Humans cannot perceive this level of size difference, but they determine the depth, which is the distance to the subject, based on the difference in the field of view of both eyes. On the other hand, in this embodiment, the movement of the subject in the forward and backward direction is determined from the subtle change in the size of subject 500, which is imperceptible to humans.

[0029] Specifically, when movement of the object to be judged occurs, the trigonometric calculation unit 103 determines whether the movement of the object to be judged includes movement in the forward and backward direction by determining, based on the skeletal position time series data, whether the forward and backward change ratio, which is the ratio of change of the determination distance defined by multiple second determination parts of the subject before and after the movement of the object to be judged, is greater than or equal to a predetermined value.

[0030] The second determination area is, for example, a skeletal area where the change in the determination distance is small when a person jumps. This area may be predetermined, or it may be a skeletal area where the change in the determination distance did not change by more than a predetermined value in the unit time before the movement of the object to be determined, or it may be a predetermined number of skeletal areas selected from among those skeletal areas that have a large change in the determination distance. The determination distance is, for example, the height of the triangle formed by the three skeletal areas. The trigonometric calculation unit 103 determines that the movement of the object to be determined includes movement in the forward and backward directions if the forward and backward change ratio, which is the ratio of the change in the determination distance, is greater than or equal to a predetermined value.

[0031] Figure 7 is a diagram illustrating an example of a distance used for determination.

[0032] When the subject is exercising or working, they may perform complex movements such as bending their body or raising one leg high. Therefore, if the determination distance is calculated from a single combination of multiple skeletal parts (for example, the triangle formed by connecting the three points 1Neck, 13LKnee, and 10RKnee mentioned above), there is a risk that the determination will not be accurate. For this reason, in this embodiment, as shown in Figure 7, the trigonometric calculation unit 103 selects multiple combinations of three skeletal parts as the second determination part, and uses the average of the change ratio of the determination distance for each combination as the before-and-after change ratio used for determination. At this time, the trigonometric calculation unit 103 may calculate the standard deviation σ of the change ratio of each determination distance and exclude the determination distance of combinations whose change ratio is an outlier. For example, the trigonometric calculation unit 103 may exclude determination distances that are outside of ±2σ.

[0033] In the example shown in Figure 7, the second determination area and determination distance are set as follows. 700: Nose-heel correction (Height 1: Height of the triangle formed by 0 Nose, 21 L Heel, and 24 R Heel) 701: Neck and heel correction (Height2: Height of the triangle formed by 1 Neck, 21 L Heel, and 24 R Heel) 702: Nose-knee correction (Height 3: Height of the triangle formed by 0 Nose, 13 L Knee, and 10 R Knee) 703: Neck and knee correction (Height 4: Height of the triangle formed by 1 Neck, 13 L Knee, and 10 R Knee) 704: Shoulder-Knee Correction (Height 5: Average height of the triangle formed by 5L shoulder, 13L knee, and 10RK knee, and the triangle formed by 2R shoulder, 13L knee, and 10RK knee)

[0034] Figure 8 is a diagram illustrating the calculation of the distance traveled by the subject in the forward and backward direction.

[0035] In Figure 8(a), the elevation angle of the line connecting camera 100 and the lowest point of the subject is set to 0 degrees. In this case, the ratio of the change in the distance used for determination before and after movement is 1:x, and as described above, if the distance between the subject and camera 100 before movement is L, then the distance l that the subject moves backward is l = L / xL.

[0036] As shown in Figure 8(b), the distance traveled, l, remains the same regardless of whether the elevation angle of the line connecting camera 100 and the lowest point of the subject is positive or negative: l = L(H+h) / HL.

[0037] Figures 9 and 10 are diagrams illustrating the processing performed by the acceleration calculation unit 104.

[0038] If the vertical movement exceeds a certain value and the front-to-back movement ratio is less than a predetermined value, it is considered that the subject moved only vertically without moving forward or backward. However, there are movements that result in a body position similar to jumping, which is vertical movement. Note that there are various types of jumps, such as jumps where only the knees are raised, and jumps where both heels are also raised and the body is stretched upward. Figure 9 shows the V-split jump 900 and the split forward bend 901, which sometimes results in a body position similar to the V-split jump 900.

[0039] The V-split jump 900 and the split-leg forward bend 901 are clearly different movements, but they can result in similar body positions, which may lead to misjudgment. It is assumed that after the V-split jump 900, the Y-coordinates of the heels and knees increase, while after the split-leg forward bend 901, the Y-coordinates of the heels and knees decrease. Therefore, by detecting this difference, it is possible to determine whether or not a jump occurred. However, in this embodiment, to determine more accurately, the acceleration calculation unit 104 determines whether or not a jump occurred based on the initial movement, which is the movement before the jump.

[0040] As shown in Figure 10, before jumping against the Earth's gravitational acceleration, a person performs an initial movement to create momentum by crouching slightly, that is, shortening the distance between their hips (or buttocks) and toes (or heels). The acceleration calculation unit 104 calculates the speed of this initial movement as the initial velocity and determines whether or not the subject has jumped based on that initial velocity.

[0041] The acceleration due to gravity on Earth is g = 9.80665 m / s². 2 The relationship between velocity, acceleration, time, displacement, and initial velocity during a jump is as follows: Velocity (m / s) = Acceleration (m / s 2 ) x time (s) + initial velocity (m / s) Displacement (m) = Initial velocity (m / s) × Time (s) + (1 / 2) × Acceleration (m / s) 2 ) × time (s) 2 Speed ​​(m / s) 2 -Initial velocity (m / s) 2 = 2 × acceleration × displacement

[0042] Therefore, for example, in order for a person to jump 10 cm (0.1 m), an initial velocity of √(2 × 9.8 × 0.1) = 1.4 m / s is required. For example, if video data is acquired at a shooting speed of 30 frames per second, a frame is acquired approximately once every 33 ms. In this case, if the hips or knees move upward by 1.4 m ÷ 30 frames = 4.67 cm between the frames in which the initial movement begins, the initial velocity will be equivalent to 1.4 m / s. In this embodiment shown in Figure 5, 4.67 cm corresponds to approximately 2.54 pixels. If the unit time is 0.1 seconds, the change in skeletal position that moved during the unit time of approximately 0.1 seconds can be measured by comparing the nth frame with the (n+3)th frame.

[0043] Figure 11 illustrates an example of a compound movement, which is a movement to be judged that includes both forward / backward and vertical movement. In Figure 11, a compound movement is shown that involves jumping backward in a parabolic trajectory.

[0044] In the example in Figure 11, as the subject moves backward, the Z-coordinate increases and at the same time, the subject's size appears to shrink. Also, when the subject jumps, the Z-coordinate increases during the first half of the jump (until reaching the highest point), but decreases during the second half of the jump (from the highest point until landing). In this case, the depth correction unit 105 performs the following processing.

[0045] The change in depth (distance l in the forward / backward direction) immediately before and after a jump can be calculated using trigonometry. The timing of the jump start is the frame in which the Z coordinate of the lowest point of the subject increases, and the timing of the jump end is the frame in which the decrease in the Z coordinate of the lowest point of the subject ends. The initial velocity of the jump can be determined by comparing the frame in which the jump occurs with the frame in which the jump occurs, but if the jump is not vertical, the angle θ of the jump must be taken into consideration. In other words, the initial velocity V of the jump is equal to the initial velocity V in the horizontal (depth) direction. y And the initial velocity V in the vertical direction zIt becomes the synthesis of. In the method of measuring the speed in the few frames before the heel leaves, what can be observed by skeletal measurement using AI technology is the initial velocity V in the vertical direction z However, it is not known how much it is kicked out in the horizontal direction.

[0046] The initial velocity V, the initial velocity V in the horizontal direction y and the initial velocity V in the vertical direction z The relationship with the jump angle θ is expressed by the following formula. V z = Vsinθ V y = Vcosθ

[0047] Also, the moving distance l in the front-back direction is the product of the initial velocity V in the horizontal direction y and the time (t) of jumping, l = V y t. Since the horizontal distance l and the jumping time t can be calculated, the initial velocity Vy in the horizontal direction is V y = l / t. On the other hand, the initial velocity V in the vertical direction z can also be calculated, and the height h can be calculated as h = V z 2 / 2g.

[0048] However, in the case of complex movement, since it is moving backward simultaneously with the jump, it is necessary to consider that the height in the Z-axis direction shrinks according to the moving distance in the Y-axis direction. Once the initial velocity V in the vertical direction z is determined, since the gravitational acceleration is constant, the jumpable height h is constant. However, when jumping vertically at the same place, even if it can be observed that the jump reaches the height h, when jumping in a parabolic shape backward, the height h will also be observed to shrink by the amount it has dropped backward.

[0049] Figure 12 shows an example of compound movement. In the example in Figure 12, a subject 500 with height H jumps backward from a distance L from camera 100 and lands at a distance l. Here, the height H of the subject 500 is the length from the lowest point of the subject to the top of its head. The jump height is also denoted as h. In this case, because it is moving backward, the jump height is observed to be smaller than h. For example, if L=200cm, l=30cm, H=160cm, and H=40cm, z1: The distance from the ground to the top of the head of a subject at height H if it jumps from its current position by height h. z2: When a subject at height H jumps a distance of h while descending a distance l, the distance from the ground to the top of the head when it has descended a distance of l / 2 and reached its highest point. z3: Distance from the ground to the top of the head of the subject at height H z4: The distance from the ground to the top of the head of a subject at height H when it moves back by distance l. z5: The distance from the ground to the feet of a subject at height H if it were to jump from its current position by height h. z6: When a subject at height H jumps a distance of h while descending a distance l, the distance from the ground to the feet when the subject descends a distance of l / 2 and reaches its highest point. So, z1 = h + H = 200 cm z² = L(h+H) / (L+l / 2) = 148.84 cm z3=H=160cm z4=HL / (L+l)=160*200 / (200+30)=139.13cm z5=h=40cm z6=hL / (L+l / 2)=40*200 / (200+30 / 2)=37.2cm This is the result.

[0050] Therefore, if you jump in place, the Z coordinate increases to a height of z1 (200 cm), but in the example in Figure 12, you move backward by a distance of l / 2, so the Z coordinate that is actually observed appears to have shrunk to z2 (148.84 cm). Similarly, if you are just standing in place, the height is observed to be H, i.e., z3 (160 cm), but if you move backward by a distance of l, it appears to have shrunk to z4 (139.13 cm). Furthermore, if you jump in place by a height of h, the Z coordinate at your feet increases by a height of h, i.e., z5 (40 cm), but because you move backward by a distance of l / 2, the Z coordinate that is actually observed appears to have shrunk to z6 (37.2 cm).

[0051] Figure 13 is a flowchart illustrating an example of the processing performed by the skeletal position estimation and correction device.

[0052] First, camera 100 photographs the subject. Then, image analysis unit 101 generates and outputs video data of the subject from the shooting results captured by camera 100 (step S1301). In this case, the subject includes the instructor and the user.

[0053] The skeleton estimation unit 102 extracts a frame from the video data output from the image analysis unit 101, estimates the position of the skeleton of the subject in that frame, and generates a skeleton position estimation result (step S1302). The skeleton estimation unit 102 then stores the generated skeleton position estimation result in addition to the skeleton position time series data (step S1303). However, if real-time performance is required, the skeleton estimation unit 102 loads the skeleton position time series data into memory.

[0054] The skeletal estimation unit 102 identifies the relationship between the actual size of the subject and the size in the video data (step S1304). For example, based on the subject's height and the subject's size (number of pixels) calculated from the skeletal position time-series data, the skeletal estimation unit 102 identifies a size correspondence relationship, which is the correspondence between the actual value (cm) and the number of pixels in the video data for each skeletal part of the subject. Human body dimensions include height, sitting height, inseam height, head circumference, head width, neck circumference, shoulder width, bust, waist, arm length, wrist circumference, hand width, foot length, foot width, thigh circumference, and calf circumference. The human body dimensions of a standard build can be obtained from height based on statistical information. The subject's height may be set in advance, or if not set, an average height may be used.

[0055] The skeleton estimation unit 102 measures the position of the lowest point of the subject in the target frame n and the next frame n+1, and calculates the difference in their positions in the Y direction as the vertical change (step S1305).

[0056] The skeletal estimation unit 102 determines whether the amount of vertical change exceeds a certain value (step S1306). If the amount of vertical change does not exceed a certain value, the skeletal estimation unit 102 returns to the process in step S1305, using the next frame n+1 as the target frame. On the other hand, if the amount of vertical change exceeds a certain value, the skeletal estimation unit 102 estimates that a movement of the subject has occurred (step S1307).

[0057] If the subject moves as determined, the trigonometric calculation unit 103 performs a forward / backward movement determination process (see Figure 14) to determine whether the subject has moved forward or backward (step S1308). Then, if forward / backward movement has occurred, the trigonometric calculation unit 103 performs a movement distance estimation process (see Figure 15) to estimate the distance the subject has moved in the forward / backward direction (step S1309).

[0058] Subsequently, the acceleration calculation unit 104 performs an initial velocity calculation process (Figure 16) to calculate the initial vertical velocity, which is the initial vertical velocity of the movement to be judged (step S1310). Then, the acceleration calculation unit 104 determines whether or not the initial vertical velocity exceeds a reference value (step S1311).

[0059] If the initial vertical velocity exceeds the reference value, the acceleration calculation unit 104 determines that the subject has jumped (step S1312). On the other hand, if the initial vertical velocity does not exceed the reference value, the acceleration calculation unit 104 determines that the subject has not jumped (step S1313).

[0060] Furthermore, if step S1312 is completed, the depth correction unit 105 may correct the forward / backward travel distance estimated by the trigonometric calculation unit 103 using the method described in Figures 11 and 12.

[0061] Figure 14 is a flowchart illustrating an example of the forward / backward movement determination process in step S1304 of Figure 13.

[0062] In the forward / backward movement determination process, the trigonometric calculation unit 103 determines multiple determination distances D1 to D defined by multiple second determination points in the target frame n. k The trigonometric calculation unit 103 calculates multiple determination distances d1~d in the next frame n+1. k The trigonometric calculation unit 103 calculates the change ratio of the determination distance C1 to C for each determination distance. k Calculate (step S1402). That is, change ratio C i =d i / D i Next, the trigonometric calculation unit 103 calculates the change ratio C1 to C for each determination distance. k Calculate the standard deviation σ (step S1403).

[0063] The trigonometric calculation unit 103 removes outliers from the change ratios C1 to Ck of the determination distances based on the standard deviation σ (step S1404). For example, the trigonometric calculation unit 103 removes change ratios that deviate by ±σ or more as outliers. Then, the skeletal estimation unit 102 calculates the average value of the change ratios C1 to Ck, excluding the outliers, as the before-and-after change ratio (step S1405).

[0064] The trigonometric calculation unit 103 determines whether the subject has moved forward or backward by determining whether the absolute value of the front-to-back change ratio is greater than or equal to a predetermined value (step S1406). In this case, if the absolute value of the front-to-back change ratio is greater than or equal to a predetermined value and the average value is negative, it can be determined that the subject has moved backward, and if the absolute value of the front-to-back change ratio is greater than or equal to a predetermined value and the front-to-back change ratio is positive, it can be determined that the subject has moved forward.

[0065] Figure 15 is a flowchart illustrating an example of the distance estimation process in step S1308 of Figure 13.

[0066] The trigonometric calculation unit 103 determines that the movement of the object to be judged has ended when the change in the front-to-back change ratio stops, and starts a movement distance estimation process to estimate the distance moved due to the movement of the object to be judged based on the skeletal position time series data from the target frame n to the frame in which the movement of the object to be judged was determined to have ended. Note that if the subject has not moved front to back, the process in step S1308 may be skipped.

[0067] Specifically, the trigonometry calculation unit 103 sets the distance L from the camera 100 to the subject (step S1500). The distance L may be specified by the user, or a predetermined standard value (for example, 200 cm) may be used.

[0068] Next, the trigonometric calculation unit 103 sets the height H of the subject (step S1501). The height H may be specified by the user, or a predetermined standard value (for example, 165 cm) may be used. Note that the height H may be the person's height or the distance between major body parts.

[0069] The trigonometry calculation unit 103 sets the height h visible from the camera 100 after moving back and forth (step S1502). The trigonometry calculation unit 103 calculates the change ratio H in the Y-axis direction. Δ =Calculate H / h (step S1503). The trigonometric calculation unit 103 then calculates l=(H Δ -1) It can be calculated using L (step S1504).

[0070] Figure 16 is a flowchart illustrating an example of the initial velocity calculation process in step S1309 of Figure 13.

[0071] The acceleration calculation unit 104 first calculates the distance L between the waist (or buttocks) and toes (or heels) in the target frames n and n-1. n (Step S1600) is measured. Subsequently, the acceleration calculation unit 104 calculates the above distance L in frame n+k, which is obtained by advancing the target frame n by a predetermined number of minutes. n+k The acceleration calculation unit 104 measures the difference in distance d. k =L n+k -L n Calculate (step S1601).

[0072] The acceleration calculation unit 104 calculates the difference in distance d k is negative (d k Determine whether the difference in distance d is <0 (step S1602). k If the value is not negative, the acceleration calculation unit 104 returns to the process of step S1601. Meanwhile, the difference in distance d k If the value is negative, the acceleration calculation unit 104 determines that the subject's waist has started to sink (start to move downwards) and calculates the difference in distance d. k The difference in distance from the previous one, d k-1 Less than (d k <d k-1 ) or not is determined (step S1603). Note that the difference in distance d from the previous step is considered. k-1 is d k-1 =L n+k-1 -L n-1 d k <d k-1 Otherwise, the acceleration calculation unit 104 returns to the process of step S1601.

[0073] d k <d k-1 In this case, the acceleration calculation unit 104 determines that the subject's waist has reached its lowest point and frames n+k The frame advanced by the specified number of minutes. n+k+m The above distance L n+k+m The acceleration calculation unit 104 measures the difference in distance d. m =L n+k+m -L n+k Calculate (step S1604).

[0074] The acceleration calculation unit 104 calculates the difference in distance d m is correct (d m Determine whether the difference in distance d is >0 (step S1605). m If the result is not positive, the acceleration calculation unit 104 returns to the process in step S1604. Meanwhile, the difference in distance d m If the value is positive, the acceleration calculation unit 104 determines that the subject's waist has started to move upward and calculates the initial vertical velocity V. Z The acceleration calculation unit 104 calculates the initial velocity V, assuming that the frame rate of the moving image data is F. Z V Z =d m It is calculated from ×F / m.

[0075] Then, the acceleration calculation unit 104 calculates the initial velocity V Z is a specific speed V Δ By determining whether or not the subject has jumped (step S1607), the process is terminated. Note that if the acceleration due to gravity is g, the jump height h is given by h = V z 2 This is expressed as / 2g. Here, assuming that the acceleration due to gravity g is 9.8 (m / s²) and that a jump is defined as the subject being 0.05 mm or more above the ground, then the specific velocity V Δ (0.05 × 2g) 1 / 2 This is approximately 0.99.

[0076] As described above, according to this embodiment, the skeletal estimation unit 102 analyzes the motion image data and estimates that a movement of the subject to be judged has occurred, which includes at least one of the movement in the front-to-back direction and the movement in the up-to-down direction, if the amount of vertical change in the first determination part of the subject exceeds a certain value. When a movement of the subject to be judged occurs, the triangulation calculation unit 103 determines, based on the motion image data, whether the front-to-back change ratio, which is the ratio of the change in the determination distance before and after the movement of the subject defined by a plurality of second determination parts of the subject, is greater than or equal to a predetermined value, thereby determining whether the movement of the subject to be judged includes the movement in the front-to-back direction. Therefore, even when using a single imaging device, it becomes possible to appropriately determine the movement of the subject in the front-to-back direction.

[0077] Furthermore, in this embodiment, the trigonometric calculation unit 103 estimates the distance traveled in the front-to-back direction of the subject based on the ratio of change in the determination distance before and after the subject's movement. This makes it possible to estimate the distance traveled more accurately.

[0078] Furthermore, in this embodiment, the determination distance is the height of the triangle formed by the three determination parts. This makes it possible to more accurately determine the movement of the subject in the front-to-back direction.

[0079] Furthermore, in this embodiment, the trigonometric calculation unit 103 selects multiple combinations of three determination points and calculates the average value of the change ratio of the determination distance for each combination as the front-to-back change ratio. This makes it possible to more appropriately determine the movement of the subject in the front-to-back direction.

[0080] Furthermore, in this embodiment, the acceleration calculation unit 104 determines whether the movement to be judged includes upward movement based on position time-series data that estimates the changes in the positions of multiple parts of the subject before the movement to be judged occurs. This makes it possible to appropriately determine upward movement (jumping) of the subject.

[0081] Furthermore, in this embodiment, if the movement of the object to be judged includes both movement in the front-to-back direction and movement in the upward direction, the depth correction unit 105 corrects the movement distance in the front-to-back direction based on the position time series data. This makes it possible to estimate the movement distance more accurately.

[0082] The embodiments of the Disclosure described above are illustrative for illustrative purposes and are not intended to limit the scope of the Disclosure to those embodiments only. Those skilled in the art can implement the Disclosure in various other forms without departing from the scope of the Disclosure. [Explanation of symbols]

[0083] 100: Camera 100 101: Image Analysis Unit 102: Skeleton Estimation Unit 103: Trigonometry Calculation Unit 104: Acceleration Calculation Unit 105: Depth Correction Unit 106: Instructor 107: User 108: Skeleton Position Estimation Result Time Series Dataset

Claims

1. An image analysis device that analyzes the movement of a subject captured in video data, An estimation unit analyzes the aforementioned video data and estimates that if the amount of vertical change in the first determination area of ​​the subject exceeds a certain value, a determination target movement has occurred that includes at least one of the forward / backward movement and vertical movement of the subject. An image analysis device having a determination unit that, when the movement of the object to be determined occurs, determines whether the movement of the object to be determined includes movement in the front-to-back direction by determining, based on the moving image data, whether the front-to-back change ratio, which is the ratio of change of the determination distance defined by a plurality of second determination parts of the subject before and after the movement of the object, is greater than or equal to a predetermined value.

2. The image analysis apparatus according to claim 1, wherein the determination unit estimates the distance traveled in the front-rear direction based on the front-rear change ratio.

3. The image analysis device according to claim 1, wherein the distance for determination is the height of the triangle formed by the three second determination parts.

4. The image analysis device according to claim 3, wherein the determination unit selects a plurality of combinations of the three second determination parts and calculates the average value of the change ratio of the determination distance for each combination as the before-and-after change ratio.

5. The image analysis apparatus according to claim 2, further comprising a calculation unit that, when the movement of the object to be determined occurs, determines whether the movement of the object to be determined includes upward movement, based on position time-series data obtained by estimating the changes in the positions of multiple parts of the subject before the movement of the object to be determined occurred.

6. The image analysis apparatus according to claim 5, further comprising a correction unit that corrects the distance of movement in the front-to-back direction based on the position time series data when the movement of the object to be determined includes both movement in the front-to-back direction and movement in the upward direction.

7. An image analysis method using an image analysis device that analyzes the movement of a subject captured in video data, By analyzing the aforementioned video data, if the amount of vertical change in the first determination area of ​​the subject exceeds a certain value, it is estimated that a determination target movement has occurred, which includes at least one of the forward / backward movement and vertical movement of the subject. An image analysis method that, when the aforementioned movement of the object to be judged occurs, determines whether the change ratio of the determination distance defined by a plurality of second determination parts of the subject before and after the movement of the object is greater than or equal to a predetermined value, based on the moving image data, thereby determining whether the movement of the object to be judged includes movement in the forward and backward direction.

Images