Joint point estimation system, joint point estimation device, joint point estimation program, and joint point estimation method

Abstract

This invention provides a joint point estimation system that can effectively improve the robustness of image-based skeletal estimation without increasing computational costs. [Solution] An articulation point estimation system comprising: an image acquisition unit that acquires captured images; an object region information detection unit that detects region information of a region containing an object based on the acquired images; a skeleton information detection unit that detects skeleton information of an object based on the acquired images; a threshold range calculation unit that calculates a threshold range based on the detected region information; and a determination unit that determines whether or not an articulation point in the skeleton information is an articulation point of an object based on the threshold range calculated by the threshold range calculation unit and the skeleton information detected by the skeleton information detection unit.

Description

Technical Field

[0001] The present invention relates to a joint point estimation system, a joint point estimation device, a joint point estimation program, and a joint point estimation method.

Background Art

[0002] There is known a technique for detecting skeletal information based on each image captured in a time series and estimating human behavior or the like based on the skeletal information.

[0003] In such a technique, in order to improve the estimation accuracy of human behavior or the like, it is necessary to improve the estimation accuracy of skeletal information, the tracking accuracy of the skeleton, and the like.

[0004] The following prior art is disclosed in Patent Document 1 below. The position of joint points and visible feature amounts, which are image features of a local range centered on the joint points, are detected from time-series images and stored as a joint point tracking flow. The confidence level of each joint point detected from the image corresponding to each joint point flow is estimated based on the similarity of the respective positions and visible feature amounts. Then, a skeleton is estimated for each person based on the combination of the joint point and the joint point tracking flow having the highest confidence level. Thereby, even if a joint point being tracked cannot be temporarily detected from the frame image due to occlusion or the like, highly accurate tracking can be continued to realize skeleton tracking.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the aforementioned prior art has a problem in that it can be difficult to distinguish between individuals when people with similar image features such as color and shape approach each other in an image, potentially reducing the accuracy of joint point detection and tracking. Furthermore, expanding the range of visible image features to improve the accuracy of joint point detection and tracking increases computational costs and reduces the real-time processing capability.

[0007] This invention was made to solve these problems. Specifically, it aims to provide a joint point estimation system, joint point estimation device, joint point estimation program, and joint point estimation method that can effectively improve the robustness of image-based skeletal estimation without increasing computational costs. [Means for solving the problem]

[0008] The above-mentioned problems of the present invention are solved by the following means.

[0009] (1) An image acquisition unit that acquires the captured image, An articulation point estimation system comprising: an object region information detection unit that detects region information of a region containing an object based on the acquired image; a skeleton information detection unit that detects skeleton information of the object based on the acquired image; a threshold range calculation unit that calculates a threshold range based on the detected region information; and a determination unit that determines whether or not an articulation point in the skeleton information is an articulation point of the object based on the threshold range calculated by the threshold range calculation unit and the skeleton information detected by the skeleton information detection unit.

[0010] (2) The joint point estimation system according to (1) above, further comprising a joint point distance calculation unit that calculates the distance between joint points based on the skeletal information detected by the skeletal information detection unit, the threshold range calculation unit that calculates the size of the region including the object based on the detected region information, calculates the threshold range based on the calculated size of the region, and the determination unit that determines whether the joint point in the skeletal information is the joint point of the object based on the threshold range calculated by the threshold range calculation unit and the distance between joint points calculated by the joint point distance calculation unit.

[0011] (3) The joint point estimation system according to (1) above, further comprising a determination execution reception unit that receives whether or not the determination unit has performed a determination, wherein the determination unit performs a determination of whether or not the joint point is the joint point of the object according to the received determination of whether or not the determination unit has performed the determination.

[0012] (4) The joint point estimation system according to (2) above, further comprising a joint point movement amount calculation unit that calculates the amount of movement of the joint point in time series based on the detected skeletal information, wherein the determination unit determines whether the joint point is the joint point of the object based on the threshold range calculated by the threshold range calculation unit, the skeletal information detected by the skeletal information detection unit, and the calculation result by the joint point movement amount calculation unit.

[0013] (5) The joint point estimation system according to (1) above, wherein the object region information detection unit detects the region information of the region surrounding half or the whole body of the object as the region information.

[0014] (6) The joint point estimation system according to (2) above, wherein the threshold range calculation unit calculates the length or area of ​​the region surrounding the object detected as region information by the object region information detection unit as the size of the region.

[0015] (7) The joint point estimation system according to (2) above, wherein the threshold range calculation unit calculates a first threshold by multiplying or adding a predetermined first coefficient to the size of the calculated region, and the determination unit determines whether the joint point in the skeletal information is the joint point of the object by comparing the calculated first threshold with the calculated distance between the joint points.

[0016] (8) The joint point estimation system described in (2) above, wherein the joint point distance calculation unit calculates the distance between the shoulder and the elbow, the elbow and the wrist, the waist and the knee, and the knee and the ankle.

[0017] (9) The joint point estimation system according to (2) above, wherein the threshold range calculation unit comprises an individual first threshold calculation unit that calculates an individual first threshold for each frame of the image by multiplying or adding a predetermined first coefficient to the calculated size of the region, and a first threshold calculation unit that calculates a moving weighted average of the individual first thresholds of the frames in a time series as the first threshold by changing a weighting coefficient according to the change in the size of the region of the current frame compared to past frames, and the determination unit determines whether the joint point in the skeletal information is the joint point of the object by comparing the calculated first threshold with the calculated distance between the joint points.

[0018] (10) The joint point estimation system described in (4) above, wherein the joint point displacement calculation unit calculates the displacement of the elbow, wrist, knee, and ankle.

[0019] (11) The joint point estimation system according to (4) above, wherein the threshold range calculation unit comprises an individual second threshold calculation unit that calculates an individual second threshold for each frame of the image by multiplying or adding a predetermined second coefficient to the calculated size of the region, and a second threshold calculation unit that calculates a moving weighted average of the individual second thresholds of the frames in a time series as the second threshold by changing a weighting coefficient according to the change in the size of the region of the current frame compared to past frames, and the determination unit determines whether the joint point is correct or not by comparing the calculated second threshold with the calculated joint point movement amount.

[0020] (12) The joint point estimation system according to (1) above further includes an off-screen determination unit that determines whether a part of the joint points included in the skeleton information exists outside the frame of the image based on the detected skeleton information, and the determination unit determines the validity of the joint points based on the determination result by the off-screen determination unit.

[0021] (13) An image acquisition unit that acquires a captured image, a skeleton information detection unit that detects skeleton information of an object based on the acquired image, an off-screen determination unit that determines whether a part of the joint points included in the skeleton information exists outside the frame of the image based on the detected skeleton information, and a determination unit that determines whether the joint points in the skeleton information are the joint points of the object based on the determination result by the off-screen determination unit.

[0022] (14) When the off-screen determination unit detects the neck, shoulder, or waist at the upper end of the frame of the screen by the skeleton information detection unit, it determines that the eyes, or the eyes and nose exist outside the screen. When the neck, shoulder, or waist is detected at the horizontal end of the screen, it determines that the wrists, or the wrists and elbows exist outside the screen. When the neck, shoulder, or waist is detected at the lower end of the screen, it determines that the ankles, or the knees and ankles exist outside the screen. The joint point estimation system according to (12) or (13) above.

[0023] (15) The joint point estimation system according to (1) or (13) above further includes an information control unit that associates the information of the joint points determined by the determination unit not to be the joint points of the object in the skeleton information with the skeleton information in which the joint points determined not to be the joint points of the object are detected.

[0024] (16) The determination execution reception unit selectively displays whether the determination by the determination unit is to be executed. The determination unit executes a determination as to whether the joint point is the joint point of the object according to whether the execution of the selected joint point is to be performed. The joint point estimation system according to (3) above.

[0025] (17) The joint point estimation system according to (7) above further includes a determination severity reception unit that receives the severity of the determination of whether the joint point is correct. The determination unit switches the magnitude of the predetermined first coefficient according to the received severity of the determination.

[0026] (18) A joint point estimation apparatus having an image acquisition unit that acquires a captured image, an object region information detection unit that detects region information of a region including an object based on the acquired image, a skeleton information detection unit that detects skeleton information of the object based on the acquired image, a threshold range calculation unit that calculates a threshold range based on the detected region information, and a determination unit that determines whether a joint point in the skeleton information is the joint point of the object based on the threshold range calculated by the threshold range calculation unit and the skeleton information detected by the skeleton information detection unit.

[0027] (19) A joint point estimation program for causing a computer to execute a process having a step (a) of acquiring a captured image, a step (b) of detecting region information of a region including an object based on the acquired image, a step (c) of detecting skeleton information of the object based on the acquired image, a step (d) of calculating a threshold range based on the detected region information, and a step (e) of determining whether a joint point in the skeleton information is the joint point of the object based on the threshold range calculated in step (d) and the skeleton information detected in step (c).

[0028] (20) A method for estimating joint points, performed by an articulation point estimation system, comprising: (a) acquiring an image; (b) detecting region information of a region including an object based on the acquired image; (c) detecting skeletal information of the object based on the acquired image; (d) calculating a threshold range based on the detected region information; and (e) determining whether an articulation point in the skeletal information is an articulation point of the object based on the threshold range calculated in step (d) and the skeletal information detected in step (c). [Effects of the Invention]

[0029] The system detects the object's region and skeletal information based on the image. It then determines the accuracy of the joint points in the skeletal information based on a threshold range calculated from the region information and the skeletal information itself. This effectively improves the robustness of image-based skeletal estimation without increasing computational costs. [Brief explanation of the drawing]

[0030] The advantages and features provided by one or more embodiments of the present invention will be better understood from the following detailed description and accompanying drawings. However, these are for illustrative purposes only and are not intended to limit the present invention. [Figure 1] This is a diagram illustrating the schematic configuration of the joint point estimation system. [Figure 2] This is a block diagram showing the hardware configuration of the joint point estimation device. [Figure 3] This is a diagram showing the articular points. [Figure 4] This figure shows the reference joint point and the first coefficient set for each joint point subject to evaluation. [Figure 5] This is an explanatory diagram showing the relationship between each joint point, the bounding box, the joint point to be judged, and the first threshold. [Figure 6]This diagram illustrates the relationship between the joint point to be judged and the first threshold when a joint point is incorrectly detected due to multiple people approaching it. [Figure 7] This figure shows a selection screen that allows the user to choose whether or not to perform a check to determine the correctness of the joint points. [Figure 8] This is a flowchart showing the operation of the joint point estimation system. [Figure 9] This is a diagram illustrating the schematic configuration of the joint point estimation system. [Figure 10] This is an explanatory diagram showing the amount of movement of the joint points. [Figure 11] This figure shows the second coefficient set for each joint point subject to evaluation. [Figure 12] This diagram illustrates the relationship between the joint point to be judged when it is incorrectly detected due to multiple people approaching it, and the first and second thresholds. [Figure 13] This diagram illustrates the relationship between the joint points to be evaluated when joint points are correctly detected, and the first and second thresholds. [Figure 14] This is a flowchart showing the operation of the joint point estimation system. [Figure 15] This is a diagram illustrating the schematic configuration of the joint point estimation system. [Figure 16] This is an explanatory diagram illustrating how the correctness of joint points can be misjudged when the body contracts due to a fall. [Figure 17] This is an explanatory diagram illustrating how the correctness of joint points can be determined when the body contracts due to a fall. [Figure 18] This diagram illustrates a case where, when the human domain size increases, the contribution of the individual first threshold of the current frame is reduced, and the contribution of the individual first threshold of past frames is increased when calculating the first threshold, resulting in an incorrect determination of whether the joint points are correct or incorrect. [Figure 19] This diagram illustrates how, when the size of the human domain increases, the contribution of the individual first threshold in the current frame is increased, and the contribution of the individual first threshold in past frames is decreased, thereby correctly determining the validity of the joint points. [Figure 20]This is a diagram illustrating the schematic configuration of the joint point estimation system. [Figure 21] This is a diagram illustrating the schematic configuration of the joint point estimation system. [Figure 22] This figure shows examples of articulation points estimated to be located outside the frame of the image. [Modes for carrying out the invention]

[0031] The following describes, with reference to the attached drawings, an articular point estimation system, an articular point estimation device, an articular point estimation program, and an articular point estimation method according to embodiments of the present invention. However, the scope of the present invention is not limited to the disclosed embodiments. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0032] (First Embodiment) Figure 1 is a schematic diagram of the joint point estimation system 1. Figure 2 is a block diagram showing the hardware configuration of the joint point estimation device 10.

[0033] The joint point estimation system 1 includes a joint point estimation device 10 and an imaging device 20. The joint point estimation system 1 may also consist of the joint point estimation device 10 alone.

[0034] The joint point estimation device 10 and the imaging device 20 can be connected to each other in a way that allows them to communicate with one another.

[0035] The joint point estimation device 10 includes a control unit 100, a storage unit 200, a display unit 300, an input unit 400, and a communication unit 500. These components are interconnected via a bus 600. The joint point estimation device 10 is configured as a computer. The joint point estimation device 10 is configured as a computer, which may be, for example, a PC (Personal Computer) or a server.

[0036] The control unit 100 is composed of a CPU (Central Processing Unit) and performs control and calculation processing of each part of the joint point estimation system 1 according to the program. The functions of the control unit 100 will be described later.

[0037] The storage unit 200 may consist of RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. The RAM temporarily stores programs and data as a working area for the control unit 100. The ROM stores various programs and data in advance. The flash memory stores various programs and data, including the operating system.

[0038] The display unit 300 is, for example, a liquid crystal display, which displays various information.

[0039] The input unit 400 is comprised of, for example, a touch panel and various keys. The input unit 400 is used for various operations and inputs.

[0040] The communication unit 500 is an interface for communicating with external devices. Communication standards such as Ethernet (registered trademark), USB, MIPI (Mobile Industry Processor Interface), IEEE 1394, Bluetooth (registered trademark), and IEEE 802.11 can be used for communication.

[0041] The imaging device 20 may be a camera that captures monochrome or color images. The imaging device 20 can generate a two-dimensional image, which is a still image, by decoding the captured image (moving image). This allows for a sequence of images (still images) in time. The imaging device 20 is installed in a predetermined location and captures images of people, for example. The imaging device 20 may be, for example, a bullet camera or a ceiling-mounted 360-degree camera.

[0042] An image can be a photograph of an object. The object may include a person. The object may also be an animal other than a person. For simplicity, the following explanation will use the example of the object being a person.

[0043] Referring to Figure 1, the functions of the control unit 100 will be described. By executing a program, the control unit 100 functions as an image acquisition unit 110, a skeletal information detection unit 120, an object region information detection unit 130, a tracking processing unit 140, an object region size calculation unit 150, an inter-joint point distance calculation unit 160, an inter-joint point correctness determination unit 170, a determination execution reception unit 180, and a determination strictness / looseness reception unit 190. The object region size calculation unit 150 and the inter-joint point correctness determination unit 170 constitute a threshold range calculation unit and a determination unit.

[0044] The image acquisition unit 110 can acquire images by receiving images captured by the imaging device 20 from the imaging device 20 via the communication unit 500. Alternatively, the image acquisition unit 110 may acquire images by reading images that have been previously captured by the imaging device 20 and stored in the storage unit 200.

[0045] The skeletal information detection unit 120 detects human skeletal information based on the acquired image. Skeletal information may be the coordinates of each joint point 121 (see Figure 3) on the image. Hereinafter, the coordinates of the joint point 121 will also be simply referred to as "joint point 121". The skeletal information detection unit 120 can detect the joint points 121 using known methods. For example, the skeletal information detection unit 120 can detect the joint points 121 using Deep Pose.

[0046] Figure 3 shows the joint point 121. In addition to the joint point 121, a person in the image is shown as a silhouette in Figure 3.

[0047] The joint points 121 include the shoulder joint point 121a, the elbow joint point 121b, the wrist joint point 121c, the hip joint point 121d, the knee joint point 121e, and the ankle joint point 121f. These are the locations detected as joint points 121 for each individual. Joint points 121 may also include joint points of the head and neck. Note that the right joint point 121 and the left joint point 121 are detected as separate joint points 121. Specifically, for example, the right shoulder joint point 121 and the left shoulder joint point 121 are detected as separate joint points 121. In Figure 3, only the left shoulder joint point 121a, the left elbow joint point 121b, the left wrist joint point 121c, the left hip joint point 121d, the left knee joint point 121e, and the left ankle joint point 121f are labeled. For the sake of simplicity, the following will also refer to the shoulder joint point 121a as "shoulder," the elbow joint point 121b as "elbow," the wrist joint point 121c as "wrist," the hip joint point 121d as "hip," the knee joint point 121e as "knee," and the ankle joint point 121f as "ankle."

[0048] The object region information detection unit 130 detects human region information based on the acquired image. Human region information is the region surrounding half or the entire body of a person in the image. Human region information may be the coordinates of the vertices of a rectangle surrounding half or the entire body of a person in the image. For simplicity, the following explanation will use the example where human region information is the coordinates of the vertices of a rectangle surrounding half of a person. Hereafter, the region surrounding half of a person, which is human region information, will also be referred to as the "bounding box 131". The object region information detection unit 130 can detect the bounding box 131 using known methods. For example, the object region information detection unit 130 can detect the bounding box 131 using YOLO (You Only Look Once).

[0049] The tracking processing unit 140 performs tracking processing to associate joint points 121 detected from past images with joint points 121 detected from current images as joint points 121 of the same person. The tracking processing unit 140 performs tracking processing by associating joint points 121 detected from current images with IDs assigned to joint points 121 detected from past images. Specifically, the tracking processing unit 140 determines whether joint points 121 detected from past images and joint points 121 detected from current images belong to the same person. Then, the tracking processing unit 140 performs tracking processing by assigning the same ID to joint points 121 determined to belong to the same person. In this way, joint points 121 detected from past images and joint points 121 detected from current images are associated as joint points 121 of the same person. The tracking processing unit 140 may perform tracking processing using known methods. The tracking processing unit 140 may also perform tracking processing based on the degree of proximity of joint points 121 detected from past and current images, respectively. The tracking processing unit 140 may perform tracking processing based on the position information of the bounding box 131 detected from past and present images by the object region information detection unit 130, and the degree of similarity of the image data within the bounding box 131. The tracking processing unit 140 may also perform tracking processing using a Kalman filter.

[0050] The object area size calculation unit 150 calculates the size of the area surrounding the person based on the bounding box 131 detected by the object area information detection unit 130. The object area size calculation unit 150 may calculate the average of the side lengths of the bounding box 131 as the size of the area surrounding the person. The object area size calculation unit 150 may also calculate the area of ​​the bounding box 131 as the size of the area surrounding the person. For the sake of simplicity, the following explanation will use the example where the size of the area surrounding the person is the average of the side lengths of the bounding box 131. Hereafter, the size of the area surrounding the person will also be referred to as the "person area size".

[0051] The joint point distance calculation unit 160 calculates the distance between joint points 121 based on the joint points 121 detected by the skeletal information detection unit 120. The joint point distance calculation unit 160 can only calculate the distance between the joint points 121 that are subject to judgment by the joint point correctness judgment unit 170 and the reference joint points. For example, the joint point distance calculation unit 160 calculates the distance between the "shoulder" and the "elbow", the distance between the "elbow" and the "wrist", the distance between the "waist" and the "knee", and the distance between the "knee" and the "ankle". Specifically, the distance between the "shoulder" and the "elbow" means the distance between the "right shoulder" and the "right elbow", and the distance between the "left shoulder" and the "left elbow". Specifically, the distance between the "elbow" and the "wrist" means the distance between the "right elbow" and the "right wrist", and the distance between the "left elbow" and the "left wrist". The distance between the "waist" and the "knee" specifically refers to the distance between the "right waist" and the "right knee," and the distance between the "left waist" and the "left knee." The distance between the "knee" and the "ankle" specifically refers to the distance between the "right knee" and the "right ankle," and the distance between the "left knee" and the "left ankle."

[0052] The joint point correctness determination unit 170 determines whether the joint point 121 in the skeletal information detected by the skeletal information detection unit 120 is a joint point of an object, based on the calculation result by the object region size calculation unit 150 and the calculation result by the joint point distance calculation unit 160. Specifically, the joint point correctness determination unit 170 determines the correctness of the joint point 121 in the skeletal information detected by the skeletal information detection unit 120, based on the calculation result by the object region size calculation unit 150 and the calculation result by the joint point distance calculation unit 160. In other words, the joint point correctness determination unit 170 determines that the detection result of the joint point to be determined is "correct" if the distance between the joint point to be determined and the reference joint point is less than or equal to the first threshold. A determination of "correct" constitutes a "positive" determination. The joint point correctness determination unit 170 determines that the detection result of the joint point to be determined is "incorrect" if the distance between the joint point to be determined and the reference joint point is greater than the first threshold. A determination of "incorrect" constitutes a "negative" determination. A reference joint point can be set in advance for each joint point to be determined. Hereinafter, the joint point 121 to be determined will also be referred to as the "joint point to be determined". The first threshold constitutes a threshold range.

[0053] The joint point accuracy determination unit 170 determines whether a joint point 121 in the skeletal information is a joint point 121 of an object, based on a first threshold and skeletal information detected by the skeletal information detection unit 120. For example, in cases where a person is sleeping and there is relatively little movement, the determination of whether a joint point 121 in the skeletal information is a joint point 121 of an object may be made by comparing the first threshold with the coordinates of the joint point in a predetermined direction.

[0054] Figure 4 shows the reference joint point and the first coefficient set for each joint point to be judged. In Figure 4, four example settings of combinations of joint points to be judged and reference joint points are shown, but in these examples, four more combinations can be set by reversing the joint points to be judged and the reference joint points. Also, as mentioned above, for example, the distance between the "elbow" and the "wrist" means the distance between the "right elbow" and the "right wrist" and the distance between the "left elbow" and the "left wrist". Therefore, considering the joint points on both the left and right sides, 16 combinations of joint points to be judged and reference joint points can be set.

[0055] In the example shown in Figure 4, for instance, when the joint point to be judged is the "ankle," the reference joint point is set to the "knee," and the first coefficient should be 0.6.

[0056] Figure 5 is an explanatory diagram showing the relationship between each joint point 121, the bounding box 131, the joint point to be judged, and the first threshold 175. The joint point to be judged is joint point 121f of the right ankle. The first threshold 175 is shown as a dashed circle with a radius of the first threshold 175, centered at joint point 121e of the right knee.

[0057] In the example shown in Figure 5, the distance between the joint point 121f of the right ankle, which is the joint point to be judged, and the joint point 121e of the right knee, which is the reference joint point, is less than or equal to the first threshold of 175. In this case, the detection result for the joint point 121f of the right ankle, which is the joint point to be judged, is determined to be "correct".

[0058] Figure 6 is an explanatory diagram showing the relationship between the joint point to be judged and the first threshold 175 when joint point 121 is incorrectly detected due to multiple people approaching it. In Figure 6, the joint point 121 detected in frames (1) to (3) of the images captured in time series is shown.

[0059] In frame (1), one person is being imaged, and the distance between the joint point 121f of the right ankle, which is the joint point to be judged, and the joint point 121e of the right knee, which is the reference joint point, is less than or equal to the first threshold of 175. Therefore, the detection result for the joint point 121f of the right ankle, which is the joint point to be judged, is determined to be "correct".

[0060] Subsequently, in frame (2), multiple people are captured without overlapping, and the distance between the joint point 121f of the right ankle, which is the joint point to be determined for the tracked person, and the joint point 121e of the right knee, which is the reference joint point, is less than or equal to the first threshold of 175. Therefore, the detection result of the joint point 121f of the right ankle, which is the joint point to be determined, is judged to be "correct".

[0061] Subsequently, in frame (3), multiple people are captured overlapping due to their proximity, and the distance between the joint point 121f of the right ankle, which is the joint point to be determined for the tracked person, and the joint point 121e of the right knee, which is the reference joint point, is greater than the first threshold 175. Therefore, the detection result of the joint point 121f of the right ankle, which is the joint point to be determined, is determined to be "incorrect". In this case, the control unit 100 may exclude the joint point 121f of the right ankle, which has been determined to be "incorrect", from the joint points 121 of the tracked person detected by the skeletal information detection unit 120. The behavior estimation unit (not shown) performs behavior estimation based on each joint point 121 from which the joint point to be determined as "incorrect" has been excluded.

[0062] The judgment execution reception unit 180 receives whether or not the joint point validity determination unit 170 has performed a validity determination of the joint point 121. Specifically, for example, the judgment execution reception unit 180 displays the option to perform a validity determination of the joint point 121 and accepts the selection of whether or not to perform a validity determination of the joint point 121. As a result, the judgment execution reception unit 180 receives whether or not to perform a validity determination of the joint point 121. The joint point validity determination unit 170 determines the validity of the joint point 121 according to whether or not the validity determination of the joint point 121 has been performed, as received by the judgment execution reception unit 180.

[0063] Figure 7 shows a selection screen that accepts the user's choice of whether or not to perform a determination of the validity of the joint point 121. The selection screen can be displayed on the display unit 300 by the determination execution acceptance unit 180. When the user checks the "Enabled" checkbox on the selection screen, the determination execution acceptance unit 180 accepts the choice to perform a determination of the validity of the joint point 121. When the "Enabled" checkbox is not checked, the determination execution acceptance unit 180 accepts the choice not to perform a determination of the validity of the joint point 121.

[0064] The judgment strictness / looseness reception unit 190 receives the strictness / looseness of the judgment regarding the pass / fail status of the joint point 121. Specifically, for example, the judgment strictness / looseness reception unit 190 displays the option to select the strictness / looseness of the judgment regarding the pass / fail status of the joint point 121, and receives the selection of the strictness / looseness of the judgment regarding the pass / fail status of the joint point 121. As a result, the judgment strictness / looseness reception unit 190 receives the strictness / looseness of the judgment regarding the pass / fail status of the joint point 121. The judgment execution reception unit 180 switches the magnitude of the first coefficient described above according to the strictness / looseness of the judgment received by the judgment strictness / looseness reception unit 190.

[0065] As shown in Figure 7, the selection screen further accepts a choice of strictness or leniency for determining whether the joint point 121 is correct or incorrect. When the user checks the "leniency" checkbox on the selection screen, the strictness / leniency judgment acceptance unit 190 accepts a choice to determine whether the joint point 121 is correct or incorrect with leniency. In this case, the joint point correct / incorrect determination unit 170 makes the first threshold 175 relatively large by making the magnitude of the first coefficient relatively large. When the user checks the "strict" checkbox on the selection screen, the strictness / leniency judgment acceptance unit 190 accepts a choice to determine whether the joint point 121 is correct or incorrect with strictness or leniency. In this case, the joint point correct / incorrect determination unit 170 makes the first threshold 175 relatively small by making the magnitude of the first coefficient relatively small. When the user checks the "medium" checkbox on the selection screen, the strictness / leniency judgment acceptance unit 190 accepts a choice to determine whether the joint point 121 is correct or incorrect with a strictness that is in between strict and leniency. In this case, the joint point correctness determination unit 170 sets the magnitude of the first coefficient to an intermediate size between the magnitude of the first coefficient when "loose" is selected and the magnitude of the first coefficient when "strict" is selected.

[0066] The operation of the joint point estimation system 1 will be explained.

[0067] Figure 8 is a flowchart showing the operation of the joint point estimation system 1. This flowchart can be executed by the control unit 100 according to a program.

[0068] The control unit 100 acquires the image captured by the imaging device 20 (S101).

[0069] The control unit 100 detects the bounding box 131 and joint points 121 based on the acquired image (S102). The bounding box 131 may be a region surrounding half of a person's body.

[0070] The control unit 100 calculates the human area size based on the detected bounding box 131 (S103). The human area size may be the average of the side lengths of the bounding box 131.

[0071] The control unit 100 calculates the distance between the joint points 121 based on the detected joint points 121 (S104). Specifically, the control unit 100 can calculate the distance between the joint point 121 that is subject to correctness determination and the reference joint point.

[0072] The control unit 100 determines whether a joint point 121 is correct or incorrect based on the calculated human area size and the distance between the joint points 121 (S105). Specifically, the control unit 100 determines whether a joint point 121 is correct or incorrect by comparing a first threshold 175, obtained by multiplying the human area size by a first coefficient, with the distance between the joint point 121 to be determined and a reference joint point.

[0073] (modified version) A modified example of the first embodiment will be described.

[0074] The control unit 100 may associate the information of a joint point 121 that has been determined to be "incorrect" by the joint point correctness determination unit 170 with the joint point 121 in the image in which the joint point 121 that has been determined to be "incorrect" was detected. Specifically, for example, the control unit 100 adds the information of a joint point 121 that has been determined to be "incorrect" as attribute information to the joint point 121 in the image in which the joint point 121 that has been determined to be "incorrect" was detected. This allows information such as the coordinates of the joint point 121 that was deemed to have been misdetected to be used for preliminary re-detection and re-verification of important actions such as falls, even if a temporary abnormality of a joint point 121 occurs. The control unit 100 constitutes an information control unit.

[0075] (Second Embodiment) A second embodiment will now be described. The differences between this embodiment and the first embodiment are as follows. In the first embodiment, the correctness of the joint points 121 is determined based on the size of the human area and the distance between the joint points 121. On the other hand, in this embodiment, the correctness of the joint points 121 is determined based on the size of the human area, the distance between the joint points 121 and the amount of movement of the joint points 121 over time. As other points are the same as in the first embodiment, redundant explanations will be omitted or simplified.

[0076] Figure 9 shows a schematic configuration of the joint point estimation system 1.

[0077] Referring to Figure 9, the functions of the control unit 100 will be explained. By executing a program, the control unit 100 functions as an image acquisition unit 110, a skeletal information detection unit 120, an object region information detection unit 130, a tracking processing unit 140, an object region size calculation unit 150, an inter-joint point distance calculation unit 160, an inter-joint point movement amount calculation unit 165, an inter-joint point correct / false determination unit 170, a determination execution reception unit 180, and a determination strict / lenient reception unit 190.

[0078] The joint point movement amount calculation unit 165 calculates the amount of joint point movement over time based on the joint points 121 detected by the skeletal information detection unit 120. Specifically, the joint point movement amount calculation unit 165 calculates the amount of movement between frames of the time-series images for each joint point 121 that has been assigned the same ID, using tracking processing by the tracking processing unit 140. In other words, the joint point movement amount calculation unit 165 calculates the amount of movement between frames of the images for each joint point 121 detected from the same person, for each joint point. The joint point movement amount calculation unit 165 can calculate the movement amounts of the "elbow," "wrist," "knee," and "ankle."

[0079] Figure 10 is an explanatory diagram showing the amount of movement of joint point 121. In Figure 10, for the sake of simplicity, the silhouettes of people in each frame of images in chronological order are shown. The dashed silhouettes are the silhouettes of people in past frames, and the solid silhouettes are the silhouettes of people in the current frame in which the joint point to be determined is detected.

[0080] In Figure 10, the amount of movement of each joint point 121 is shown as the length of the arrow. Joint points 121 detected from past frames are shown as dashed circles. Joint points 121 detected from the current frame are shown as black circles. The amount of movement of joint point 121f of the "right ankle" is exemplified by the length of arrow 121fm.

[0081] The joint point correctness determination unit 170 determines the correctness of a joint point 121 based on the human domain size, the distance between joint points 121, and the amount of movement of the joint points 121 over time. The joint point correctness determination unit 170 may consist of a primary determination unit and a secondary determination unit. The primary determination unit, similar to the first embodiment, determines whether a joint point 121 is "correct" based on the calculation result by the object domain size calculation unit 150 and the calculation result by the joint point distance calculation unit 160. Specifically, the primary determination unit calculates a first threshold 175 by multiplying or adding a first coefficient to the human domain size. The primary determination unit then determines whether a joint point to be determined is "correct" by comparing the distance between the joint point to be determined and a reference joint point with the first threshold 175. The primary determination unit determines that the detection result of the joint point to be determined is "correct" if the distance between the joint point to be determined and a reference joint point is less than or equal to the first threshold 175. The primary determination unit, if the distance between the joint point to be determined and the reference joint point is greater than the first threshold of 175, does not determine that the detection result of the joint point to be determined is "correct," and instead prompts the secondary determination unit to determine whether the joint point to be determined is correct or not.

[0082] The secondary determination unit determines whether the joint point 121 is correct or incorrect based on the calculation results from the object region size calculation unit 150 and the calculation results from the joint point movement amount calculation unit 165. Specifically, the secondary determination unit calculates a second threshold 176 by multiplying or adding a second coefficient to the human region size. Then, the secondary determination unit determines whether the joint point to be determined is correct or incorrect by comparing the movement amount of the joint point to be determined with the second threshold 176. If the movement amount of the joint point to be determined is less than or equal to the second threshold 176, the secondary determination unit determines that the detection result of the joint point to be determined is "correct". If the movement amount of the joint point to be determined is greater than the second threshold 176, the secondary determination unit determines that the detection result of the joint point to be determined is "incorrect".

[0083] Figure 11 shows the second coefficient set for each joint point to be evaluated. In Figure 11, four example settings of combinations of joint points to be evaluated and the second coefficient are shown. For example, "elbow" can refer to both the "right elbow" and the "left elbow". Therefore, considering the left and right sides of joint point 121, eight combinations of joint points to be evaluated and the second coefficient can be set.

[0084] In the example shown in Figure 11, for example, when the joint point to be judged is the "ankle," the second coefficient is set to 0.7.

[0085] Figure 12 is an explanatory diagram showing the relationship between the joint point to be judged when joint point 121 is incorrectly detected due to multiple people approaching, and the first threshold 175 and the second threshold 176. In Figure 12, the joint point 121 detected in frames (1) to (3) of images captured in time series is shown.

[0086] In frame (2), the distance between the joint point 121f(2) of the right ankle, which is the joint point to be determined for the person being tracked, and the joint point 121e(2) of the right knee, which is the reference joint point, is less than or equal to the first threshold of 175. Therefore, the primary determination unit determines that the detection result of the joint point 121f of the right ankle, which is the joint point to be determined, is "correct". If the primary determination unit determines that the detection result of the joint point 121f of the right ankle, which is the joint point to be determined, is "correct", the determination by the secondary determination unit is not performed. Note that in Figure 11, for the sake of simplicity, the second threshold 176 used for the determination by the secondary determination unit is shown. The second threshold 176 in frame (2) is shown by a dashed line with the radius of the coordinates of the joint point 121f(1) of the right ankle detected in frame (1) as the second threshold 176.

[0087] Subsequently, in frame (3), as a result of multiple people approaching each other, they are captured overlapping, and the distance between the joint point 121f(3) of the right ankle, which is the joint point to be determined for the tracked person, and the joint point 121e(3) of the right knee, which is the reference joint point, is greater than the first threshold 175. In this case, the primary determination unit causes the secondary determination unit to determine whether the joint point to be determined is correct or incorrect. In frame (3), the amount of movement of the joint point 121f(3) of the right ankle, which is the joint point to be determined for the tracked person, from the joint point 121f(2) of the right ankle in frame (2) is greater than the second threshold 176. Therefore, the second determination unit determines that the detection result for the joint point 121f(3) of the right ankle, which is the joint point to be determined, is "incorrect". In this case, the control unit 100 can exclude the joint point 121f(3) of the right ankle, which has been determined to be "incorrect", from the joint points 121 of the tracked person detected by the skeletal information detection unit 120. The behavior estimation unit (not shown) performs behavior estimation based on each joint point 121 from which the joint points that were determined to be "errors" have been excluded.

[0088] Figure 13 is an explanatory diagram showing the relationship between the joint point to be judged when joint point 121 is correctly detected, and the first threshold 175 and the second threshold 176. In Figure 13, the joint point 121 detected in frames (1) to (3) of the images acquired in time series is shown.

[0089] In frame (2), the distance between the joint point 121f(2) of the right ankle, which is the joint point to be determined for the subject being tracked, and the joint point 121e(2) of the right knee, which is the reference joint point, is less than or equal to the first threshold of 175. Therefore, the primary determination unit determines that the detection result of the joint point 121f of the right ankle, which is the joint point to be determined, is "correct". If the primary determination unit determines that the detection result of the joint point 121f of the right ankle, which is the joint point to be determined, is "correct", the determination by the secondary determination unit is not performed. Note that in Figure 13, for the sake of simplicity, the second threshold of 176 used for the determination by the secondary determination unit is shown. Furthermore, the amount of movement of the joint point 121f(2) of the right ankle, which is the joint point to be determined for the subject being tracked, from the joint point 121f(1) of the right ankle in frame (1) is greater than the second threshold of 176. Therefore, if a judgment were to be made by the secondary judgment unit, the detection result of the joint point 121f(3) of the right ankle, which is the joint point to be judged by the second judgment unit would be judged as "incorrect". However, as mentioned above, the judgment by the secondary judgment unit is not performed. Therefore, even if the amount of movement of the joint point 121f(2) of the right ankle, which is the joint point to be judged of the tracked person in frame (2), is greater than the second threshold 176, it does not affect the judgment result that the detection result of the joint point 121f of the right ankle, which is the joint point to be judged in frame (2), is "correct".

[0090] Subsequently, in frame (3), the distance between the joint point 121f(3) of the right ankle, which is the joint point to be determined for the subject of tracking, and the joint point 121e(3) of the right knee, which is the reference joint point, is greater than the first threshold 175. For this reason, the primary determination unit does not determine that the joint point 121f(3) of the right ankle, which is the joint point to be determined, is "correct," and instead has the secondary determination unit determine whether the joint point to be determined is correct or not. In frame (3), the amount of movement of the joint point 121f(3) of the right ankle, which is the joint point to be determined for the subject of tracking, from the joint point 121f(2) of the right ankle in frame (2) is less than or equal to the second threshold 176. For this reason, the second determination unit determines that the detection result for the joint point 121f(3) of the right ankle, which is the joint point to be determined, is "correct."

[0091] The operation of the joint point estimation system 1 will be explained.

[0092] Figure 14 is a flowchart showing the operation of the joint point estimation system 1. This flowchart can be executed by the control unit 100 according to a program.

[0093] The control unit 100 acquires the image captured by the imaging device 20 (S201).

[0094] The control unit 100 detects the bounding box 131 and joint points 121 based on the acquired image (S202). The bounding box 131 may be a region surrounding half of a person's body.

[0095] The control unit 100 calculates the human area size based on the detected bounding box 131 (S203). The human area size may be the average of the side lengths of the bounding box 131.

[0096] The control unit 100 calculates the distance between the joint points 121 based on the detected joint points 121 (S204). Specifically, the control unit 100 can calculate the distance between the joint point 121 to be judged and the reference joint point.

[0097] The control unit 100 determines whether the joint point 121 is "correct" based on the calculated human domain size and the distance between the joint points 121 (S205). Specifically, the control unit 100 determines whether the joint point 121 to be judged is "correct" by comparing a first threshold 175 obtained by multiplying the human domain size by a first coefficient with the distance between the joint point 121 to be judged and a reference joint point.

[0098] If the control unit 100 determines that the joint point 121 subject to judgment is "correct" (S206: YES), it terminates processing for the joint point 121 subject to judgment.

[0099] If the control unit 100 does not determine that the joint point 121 subject to judgment is "correct" (S206: NO), it calculates the amount of movement of the joint point 121 subject to judgment based on the detected joint point 121 (S207).

[0100] The control unit 100 determines whether the joint point 121 is correct or incorrect based on the calculated human domain size and the amount of movement of the joint point 121 (S208). Specifically, the control unit 100 determines whether the joint point 121 is correct or incorrect by comparing a second threshold 176, obtained by multiplying the human domain size by a second coefficient, with the amount of movement of the joint point 121 to be judged.

[0101] (Third embodiment) A third embodiment will now be described. The differences between this embodiment and the first embodiment are as follows. In the first embodiment, the size of the human area is calculated for each image frame, and the first threshold 175 is calculated by multiplying the calculated human area size by a first coefficient. On the other hand, in this embodiment, the size of the human area is calculated for each frame, and an individual first threshold is calculated by multiplying the calculated human area size by a first coefficient. Then, the weighting coefficient is changed according to the change in the size of the human area of ​​the current frame compared to past frames, and the moving weighted average of the individual first thresholds of the time-series frames is calculated as the first threshold 175. As other points are the same as in the first embodiment, redundant explanations will be omitted or simplified.

[0102] Figure 15 shows a schematic configuration of the joint point estimation system 1.

[0103] Referring to Figure 15, the functions of the control unit 100 will be described. By executing a program, the control unit 100 functions as an image acquisition unit 110, a skeletal information detection unit 120, an object region information detection unit 130, a tracking processing unit 140, an object region size calculation unit 150, an inter-joint point distance calculation unit 160, an inter-joint point movement amount calculation unit 165, an inter-joint point correctness determination unit 170, a determination execution reception unit 180, and a determination strictness / looseness reception unit 190. The inter-joint point correctness determination unit 170 includes an individual first threshold calculation unit 171 and a first threshold calculation unit 172.

[0104] The individual first threshold calculation unit 171 included in the joint point correctness determination unit 170 calculates an individual first threshold for each image frame by multiplying or adding a first coefficient to the human region size calculated by the object region size calculation unit 150.

[0105] The first threshold calculation unit 172 changes the weighting coefficient according to the change in the size of the human region of the current frame in which the joint point 121 to be judged is detected relative to past frames, and calculates the moving weighted average of the individual first thresholds of the time-series frames as the first threshold 175. The past frames used to calculate the first threshold 175 may be one or more. For the sake of simplicity, the following explanation will be given as an example where only one past frame is used to calculate the first threshold 175. That is, the first threshold 175 will be given as an example where it is the moving weighted average of the individual first thresholds calculated from the current frame and the past frames adjacent to the current frame.

[0106] The first threshold of 175 can be calculated by the following formula (1) or (2), depending on the change in the size of the human area in the current frame compared to past frames.

[0107] First threshold = α UP × Individual first threshold of the current frame + (1-α UP ) × Individual first threshold of past frames ···(1) First threshold = α DOWN × Individual first threshold of the current frame + (1-α DOWN ) × Individual first threshold of past frames ···(2) In equation (1), the weighting coefficient α UP For example, it is set to 0.9. In equation (2), the weighting coefficient α DOWN For example, this is set to 0.1.

[0108] Equation (1) is used to calculate the first threshold of 175 when the size of the human area in the current frame is larger than the size of the human area in previous frames. Equation (2) is used to calculate the first threshold of 175 when the size of the human area in the current frame is less than or equal to the size of the human area in previous frames.

[0109] The joint point correctness determination unit 170 determines whether the joint point 121 is correct by comparing the calculated first threshold 175 with the distance between the joint points 121 calculated by the joint point distance calculation unit 160.

[0110] Figure 16 is an explanatory diagram illustrating a case where the correctness of joint point 121 is incorrectly determined when the body contracts due to a fall. In Figure 16, joint point 121 is shown in frames (1) to (3) of images acquired in time series.

[0111] Frame (2) is the frame immediately before the subject falls. In frame (2), the distance between the joint point 121f(2) of the right ankle, which is the joint point to be judged, and the joint point 121e(2) of the right knee, which is the reference joint point, is less than or equal to the first threshold of 175. Therefore, the detection result of the joint point 121f of the right ankle, which is the joint point to be judged, can be judged as "correct".

[0112] Frame (3) is the frame when the subject fell. In frame (3), the distance between the joint point 121f(3) of the right ankle, which is the joint point to be judged, and the joint point 121e(3) of the right knee, which is the reference joint point, is greater than the first threshold 175. For this reason, the detection result of the joint point 121f of the right ankle, which is the joint point to be judged, is incorrectly judged as "error". The reason why the distance between the joint point 121f(3) of the right ankle and the joint point 121e(3) of the right knee, which is the reference joint point, is greater than the first threshold 175 is that the size of the human area used to calculate the first threshold 175 has decreased due to the fall, resulting in a smaller first threshold 175. In such cases, even though the distance between the joint point 121f(3) of the right ankle and the reference joint point 121e(3) of the right knee is normal, there is a high probability that the detection result for the joint point 121f of the right ankle will be incorrectly judged as "error".

[0113] Figure 17 is an explanatory diagram showing that, according to this embodiment, the correctness of the joint point 121 is correctly determined when the body curls up due to a fall. In Figure 17, the joint point 121 detected in frames (1) to (3) of images captured in time series is shown.

[0114] Frame (2) is the frame immediately before the subject falls. In frame (2), the distance between the joint point 121f(2) of the right ankle, which is the joint point to be judged, and the joint point 121e(2) of the right knee, which is the reference joint point, is less than or equal to the first threshold of 175. Therefore, the detection result of the joint point 121f of the right ankle, which is the joint point to be judged, can be judged as "correct".

[0115] Frame (3) is the frame in which the subject fell. In frame (3), the distance between the joint point 121f(3) of the right ankle, which is the joint point to be judged, and the joint point 121e(3) of the right knee, which is the reference joint point, is less than or equal to the first threshold of 175. For this reason, the detection result of the joint point 121f of the right ankle, which is the joint point to be judged, is correctly determined to be "correct". This is because the size of the human area used to calculate the first threshold of 175 is reduced due to the fall, but the degree to which the first threshold of 175 is calculated to be smaller is mitigated by using the above formula (2). Specifically, this is because the contribution of the individual first threshold of the current frame to the calculation of the first threshold of 175 is reduced, and the contribution of the individual first threshold of past frames is increased.

[0116] Figure 18 is an explanatory diagram illustrating a case in which the validity of an articular point 121 is incorrectly determined when the size of the human region increases, by lowering the contribution of the individual first threshold of the current frame and increasing the contribution of the individual first threshold of past frames to calculate the first threshold 175. Specifically, Figure 18 is an explanatory diagram illustrating a case in which the validity of an articular point 121 is incorrectly determined when the size of the human region increases as the person being tracked approaches the imaging device 20, by calculating the first threshold 175 using the above formula (2). In Figure 18, the articular points 121 detected in frames (1) to (2) of the images captured in time series are shown.

[0117] Frame (1) is a frame captured just before the subject being tracked approached the imaging device 20. In frame (1), the distance between the joint point 121c(1) of the right wrist, which is the joint point to be determined for the subject being tracked, and the joint point 121b(2) of the right elbow, which is the reference joint point, is less than or equal to the first threshold of 175. Therefore, the detection result for the joint point 121c(1) of the right wrist, which is the joint point to be determined, can be judged as "correct".

[0118] Frame (2) is a frame captured when the subject being tracked approached the imaging device 20. As the subject being tracked approaches the imaging device 20, the size of the human area increases. In frame (2), the first threshold 175 is calculated using the above formula (2), and the distance between the joint point 121c(2) of the right wrist, which is the joint point to be determined, and the joint point 121b(2) of the right elbow, which is the reference joint point, is greater than the first threshold 175. For this reason, the detection result for the joint point 121f(2) of the right wrist, which is the joint point to be determined, is incorrectly judged as "error". The reason why the distance between the joint point 121c(2) of the right wrist and the joint point 121c(2) of the right elbow, which is the reference joint point, is greater than the first threshold 175 is as follows: The size of the human area used to calculate the first threshold 175 increases as the subject being tracked approaches the imaging device 20. However, by reducing the contribution of the individual first threshold for the current frame and increasing the contribution of the individual first threshold for past frames, the first threshold 175 became smaller than the distance between the joint point 121c(2) of the right wrist and the reference joint point, the joint point 121c(2) of the right elbow. In such a case, even though the distance between the joint point 121c(2) of the right wrist and the reference joint point, the joint point 121b(2) of the right elbow, is normal, there is a high possibility that the detection result for the joint point 121c of the right wrist will be misjudged as "incorrect".

[0119] Figure 19 is an explanatory diagram illustrating how the validity of articular points 121 can be correctly determined by increasing the contribution of the individual first threshold in the current frame and decreasing the contribution of the individual first threshold in past frames when the size of the human area increases. Specifically, Figure 19 is an explanatory diagram illustrating how the validity of articular points 121 can be correctly determined by calculating the first threshold 175 using the above formula (1) when the size of the human area increases as the person being tracked approaches the imaging device 20. In Figure 19, articular points 121 detected in frames (1) to (2) of images captured in time series are shown.

[0120] Frame (1) is a frame captured just before the subject being tracked approached the imaging device 20. In frame (1), the distance between the joint point 121c(1) of the right wrist, which is the joint point to be determined for the subject being tracked, and the joint point 121b(2) of the right elbow, which is the reference joint point, is less than or equal to the first threshold of 175. Therefore, the detection result for the joint point 121c(1) of the right wrist, which is the joint point to be determined, can be judged as "correct".

[0121] Frame (2) is a frame captured when the subject being tracked approached the imaging device 20. As the subject being tracked approaches the imaging device 20, the size of the human area increases. In frame (2), the first threshold 175 is calculated using the above formula (1), and the distance between the joint point 121c(2) of the right wrist, which is the joint point to be determined, and the joint point 121b(2) of the right elbow, which is the reference joint point, is less than or equal to the first threshold 175. Therefore, the detection result of the joint point 121f(2) of the right wrist, which is the joint point to be determined, is incorrectly judged as "correct". The reason why the distance between the joint point 121c(2) of the right wrist and the joint point 121c(2) of the right elbow, which is the reference joint point, is less than or equal to the first threshold 175 is as follows: The size of the human area used to calculate the first threshold 175 increases as the subject being tracked approaches the imaging device 20. However, in calculating the first threshold of 175, the contribution of the individual first threshold of the current frame was increased, while the contribution of the individual first threshold of past frames was decreased. As a result, the distance between the joint point 121c(2) of the right wrist and the reference joint point 121c(2) of the right elbow became smaller than the first threshold of 175.

[0122] (modified version) A modified example of the third embodiment will now be described.

[0123] Figure 20 shows a schematic configuration of the joint point estimation system 1.

[0124] The joint point correctness determination unit 170 may also include an individual second threshold calculation unit 173 and a second threshold calculation unit 174.

[0125] The individual second threshold calculation unit 173 calculates an individual second threshold for each image frame by multiplying or adding a predetermined second coefficient to the calculated human area size. The second threshold calculation unit 174 changes the weighting coefficient according to the change in the size of the human area of ​​the current frame compared to past frames, and calculates the moving weighted average of the individual second thresholds of the frames in the time series as the second threshold 176. The joint point correctness determination unit 170 determines the correctness of the joint point by comparing the calculated second threshold 176 with the joint point movement amount calculated by the joint point movement amount calculation unit 165. The weighting coefficient when calculating the moving weighted average of the individual second thresholds can be set appropriately by experiment from the viewpoint of the detection accuracy of the joint point 121.

[0126] (Fourth Embodiment) A fourth embodiment will now be described. The differences between this embodiment and the second embodiment are as follows. In this embodiment, based on the joint points 121 detected from the image frame, it is further determined whether a portion of the joint points 121 contained in the joint points 121 are outside the frame's display area, and the validity of the joint points 121 is determined based on the determination result. As other points are the same as in the second embodiment, redundant explanations will be omitted or simplified.

[0127] Figure 21 shows a schematic configuration of the joint point estimation system 1.

[0128] The screen-out-of-screen determination unit 195 determines, based on each joint point 121 detected by the skeletal information detection unit 120, whether a portion of the joint points 121 included in the joint points 121 exist outside the screen of the image frame.

[0129] The out-of-screen detection unit 195 determines that if the "neck," "shoulders," or "waist" are detected at the top edge of the screen, then the "eyes," or "eyes" and "nose" are outside the screen. The out-of-screen detection unit 195 determines that if the "neck," "shoulders," or "waist" are detected at the side edge of the screen, then the "wrist," or "wrist" and "elbow" are outside the screen. The out-of-screen detection unit 195 determines that if the "neck," "shoulders," or "waist" are detected at the bottom edge of the screen, then the "ankle," or "knee" and "ankle" are outside the screen.

[0130] The joint point correctness determination unit 170 determines whether a joint point 121 is correct or incorrect based on the determination result from the out-of-screen determination unit 195. If a joint point 121 that has been determined to be out of screen by the out-of-screen determination unit 195 is detected by the skeletal information detection unit 120, the joint point correctness determination unit 170 determines that the joint point 121 is "incorrect".

[0131] Figure 22 shows an example of an articular point 121 that was estimated to be located outside the frame of the image.

[0132] Figure 22 shows that the ankle f, which is actually located outside the frame 111 of the image, is detected by the skeletal information detection unit 120 as the joint point 121f of the "ankle" located within the frame 111. The out-of-screen determination unit 195 likely determined that the joint point 121f of the "ankle" is located outside the screen because the joint point 121d of the "hip" is detected at the bottom edge of the screen.

[0133] The embodiment provides the following effects.

[0134] The system detects the object's region and skeletal information based on the image. It then determines the accuracy of the joint points in the skeletal information based on a threshold range calculated from the region information and the skeletal information itself. This effectively improves the robustness of image-based skeletal estimation without increasing computational costs.

[0135] The system detects the object's region and skeletal information based on the image. It then determines the accuracy of the joints in the skeletal information based on the calculated region size and the distance between joints calculated from the skeletal information. This effectively improves the robustness of image-based skeletal estimation without increasing computational costs.

[0136] Furthermore, the system accepts whether or not a correct / incorrect judgment should be performed, and determines the correctness of the joint points according to the accepted judgment. This allows for flexible improvement of the accuracy of skeletal estimation depending on the system's operating environment.

[0137] Furthermore, based on the skeletal information, the amount of joint point movement over time is calculated, and the accuracy of the joint points is determined based on the object's region size calculated based on the region information, as well as the distance between joint points and the amount of joint point movement calculated based on the skeletal information. This further improves the robustness of image-based skeletal estimation.

[0138] Furthermore, as region information, it detects information about the area surrounding half or the entire body of an object. This allows for more accurate and easier determination of whether or not joint points are correct.

[0139] Furthermore, the object's region size is calculated as the length or area of ​​the region surrounding the detected object. This allows for a more accurate and simpler determination of whether an articulation point is correct.

[0140] Furthermore, the correctness of a joint point is determined by comparing a first threshold, calculated by multiplying or adding a predetermined first coefficient to the calculated size of the region, with the calculated distance between joint points. This allows the first threshold to be set flexibly and appropriately depending on the scene, such as a crowd scene or a scene with intense movement.

[0141] Furthermore, the distances between joint points are calculated as the distances between the shoulder and elbow, elbow and wrist, hip and knee, and knee and ankle. By focusing on the extremity joints, where misestimation of joint points is likely to occur, the accuracy of skeletal estimation can be significantly improved with the minimum necessary computational effort.

[0142] Furthermore, a first threshold value is calculated for each image frame by multiplying or adding a predetermined first coefficient to the calculated size of the object's region. The weighting coefficient is changed according to the change in the size of the object's region in the current frame compared to past frames, and the moving weighted average of the individual first threshold values ​​for the time-series frames is calculated as the first threshold value. This enables highly accurate skeletal estimation even when the size of the object's region changes rapidly due to object movement, etc.

[0143] Furthermore, the movement amounts of the elbows, wrists, knees, and ankles are calculated as joint point displacements. By focusing on the extremity joints, where joint point misestimation is prone to occur, the accuracy of skeletal estimation can be significantly improved with the minimum necessary computational effort.

[0144] Furthermore, a separate second threshold is calculated for each image frame by multiplying or adding a predetermined second coefficient to the calculated size of the object's region. The weighting coefficient is changed according to the change in the size of the object's region in the current frame compared to past frames, and the moving weighted average of the individual second thresholds of the time-series frames is calculated as the second threshold. This enables highly accurate skeletal estimation even when the size of the object's region changes rapidly due to object movement, etc.

[0145] Furthermore, based on the detected skeletal information, it is determined whether some of the joint points included in the skeletal information are located outside the frame of the image, and the correctness of the joint points is determined based on the result of this determination. This allows for highly accurate skeletal estimation even when part of the object is located outside the frame.

[0146] Furthermore, if the neck, shoulders, or waist are detected at the top edge of the frame's screen, it is determined that the eyes, or eyes and nose, are outside the screen. If the neck, shoulders, or waist are detected at the side edge of the screen, it is determined that the wrists, or wrists and elbows, are outside the screen. If the neck, shoulders, or waist are detected at the bottom edge of the screen, it is determined that the ankles, or knees and ankles, are outside the screen. This makes it easy to determine whether any part of an object is outside the frame's screen.

[0147] Furthermore, information on joint points that are determined to be false is associated with the skeletal information in which those joint points were detected. This allows for the suppression of false positives by using information such as the coordinates of the falsely detected joint point 121 to perform preliminary re-detection and re-verification of important actions such as falls, even if a temporary abnormality occurs at joint point 121.

[0148] Furthermore, the system displays an option to choose whether or not to perform the joint point verification, and the verification is performed according to the selected option. This allows for flexible and appropriate improvement of the detectability of skeletal information.

[0149] Furthermore, the system accepts requests for stricter or less strict judgments regarding the accuracy of joint points, and switches the magnitude of a predetermined first coefficient according to the accepted strictness or less strictness of the judgment. This allows for optimization of the first coefficient according to the installation location of the imaging device and how people appear in the installation area.

[0150] The present invention is not limited to the embodiments described above.

[0151] For example, in this embodiment, some or all of the processing performed by the program may be replaced by hardware such as circuits.

[0152] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are for illustrative purposes only and are not limiting. The scope of the present invention should be interpreted in accordance with the language of the appended claims. [Explanation of Symbols]

[0153] 1. Joint point estimation system, 10. Joint point estimation device, 20 Imaging device, 100 Control unit, 110 Image acquisition unit, 120 Skeletal information detection unit, 121 joint points, 121a Joint point of the shoulder, 121b Joint point of the elbow, 121c The joint point of the wrist, 121d Lumbar joint point, 121e Joint point of the knee, 121f Ankle joint point, 130 Object area information detection unit, 140 Tracking Processing Unit, 150 Object region size calculation unit, 160 Inter-joint distance calculation unit, 165 Joint point movement amount calculation unit, 170 Joint point correctness determination unit, 171 Individual first threshold calculation unit, 172 First threshold calculation unit, 173 Individual second threshold calculation unit, 174 Second threshold calculation unit, 175 First threshold, 176 Second threshold, 180 Judgment Execution Reception Unit, 190 Reception Department for Strict and Relaxed Judgments, 195 Off-screen determination section, 200 storage section, 300 display section, 400 Input section, 500 Communications Department, 600 buses.

Claims

1. An image acquisition unit that acquires the captured image, Based on the acquired image, an object region information detection unit detects region information of a region containing an object, A skeleton information detection unit detects the skeleton information of the object based on the acquired image, A threshold range calculation unit calculates a threshold range based on the detected region information, A determination unit determines whether or not the joint points in the skeletal information are the joint points of the object, based on the threshold range calculated by the threshold range calculation unit and the skeletal information detected by the skeletal information detection unit. A joint point estimation system having the following features.

2. The system further includes an inter-joint distance calculation unit that calculates the distance between joint points based on the skeletal information detected by the skeletal information detection unit, The threshold range calculation unit calculates the size of the region including the object based on the detected region information, and calculates the threshold range based on the calculated size of the region. The joint point estimation system according to claim 1, wherein the determination unit determines whether the joint point in the skeletal information is the joint point of the object based on the threshold range calculated by the threshold range calculation unit and the distance between the joint points calculated by the joint point distance calculation unit.

3. The system further includes a determination execution reception unit that receives information on whether or not the determination unit has performed the determination, The joint point estimation system according to claim 1, wherein the determination unit performs a determination of whether the joint point is the joint point of the object, according to whether or not the execution has been received.

4. The system further includes a joint point movement amount calculation unit that calculates the amount of movement of the joint points over time based on the detected skeletal information. The joint point estimation system according to claim 2, wherein the determination unit determines whether the joint point is the joint point of the object based on the threshold range calculated by the threshold range calculation unit, the skeletal information detected by the skeletal information detection unit, and the calculation result by the joint point movement amount calculation unit.

5. The joint point estimation system according to claim 1, wherein the object region information detection unit detects region information of the region surrounding half or the entire body of the object as region information.

6. The joint point estimation system according to claim 2, wherein the threshold range calculation unit calculates the length or area of ​​the region surrounding the object detected as region information by the object region information detection unit as the size of the region.

7. The threshold range calculation unit calculates a first threshold by multiplying or adding a predetermined first coefficient to the size of the calculated region. The joint point estimation system according to claim 2, wherein the determination unit determines whether the joint point in the skeletal information is the joint point of the object by comparing the calculated first threshold with the calculated distance between the joint points.

8. The joint point estimation system according to claim 2, wherein the joint point distance calculation unit calculates the distance between the shoulder and the elbow, the elbow and the wrist, the waist and the knee, and the knee and the ankle.

9. The threshold range calculation unit, An individual first threshold calculation unit calculates an individual first threshold for each frame of the image by multiplying or adding a predetermined first coefficient to the calculated size of the region, The system includes a first threshold calculation unit that calculates a first threshold by changing the weighting coefficient according to the change in the size of the region of the current frame relative to past frames, and by using the moving weighted average of the individual first thresholds of the frames in the time series as the first threshold. The determination unit determines whether the joint point in the skeletal information is the joint point of the object by comparing the calculated first threshold with the calculated distance between the joint points. The joint point estimation system according to claim 2.

10. The joint point estimation system according to claim 4, wherein the joint point movement amount calculation unit calculates the movement amounts of the elbow, wrist, knee, and ankle.

11. The threshold range calculation unit, An individual second threshold calculation unit calculates an individual second threshold for each frame of the image by multiplying or adding a predetermined second coefficient to the calculated size of the region, The system includes a second threshold calculation unit that calculates a second threshold by changing the weighting coefficient according to the change in the size of the region of the current frame relative to past frames, and by using a moving weighted average of the individual second thresholds of the frames in the time series. The joint point estimation system according to claim 4, wherein the determination unit determines whether the joint point is correct or incorrect by comparing the calculated second threshold with the calculated joint point movement amount.

12. The system further includes an out-of-screen determination unit that determines, based on the detected skeletal information, whether a portion of the joint points included in the skeletal information is located outside the frame of the image. The joint point estimation system according to claim 1, wherein the determination unit determines whether the joint point is correct or incorrect based on the determination result from the off-screen determination unit.

13. An image acquisition unit that acquires the captured image, A skeleton information detection unit detects the skeletal information of an object based on the acquired image, An off-screen determination unit determines whether a portion of the joint points included in the skeletal information is located outside the frame of the image, based on the detected skeletal information. A determination unit determines whether the joint point in the skeletal information is the joint point of the object based on the determination result by the off-screen determination unit, An articular point estimation system that includes this.

14. The joint point estimation system according to claim 12 or 13, wherein the out-of-screen determination unit determines that if the neck, shoulders, or waist are detected at the upper edge of the screen of the frame by the skeletal information detection unit, the eyes, or eyes and nose are located outside the screen; if the neck, shoulders, or waist are detected at the horizontal edge of the screen, the wrists, or wrists and elbows are located outside the screen; and if the neck, shoulders, or waist are detected at the lower edge of the screen, the ankles, or knees and ankles are located outside the screen.

15. The joint point estimation system according to claim 1 or 13, further comprising an information control unit that associates information of a joint point in the skeletal information that is determined by the determination unit to be not a joint point of the object with the skeletal information in which the joint point that is determined to be not a joint point of the object was detected.

16. The aforementioned determination execution reception unit displays whether or not to perform a determination by the determination unit, allowing the user to select the option. The joint point estimation system according to claim 3, wherein the determination unit determines whether the selected joint point is the joint point of the object, according to whether or not the selected joint point is executed.

17. The system further includes a judgment strictness / looseness receiving unit that receives the strictness / looseness of the judgment of whether the joint point is correct or incorrect, The joint point estimation system according to claim 7, wherein the determination unit switches the magnitude of the predetermined first coefficient according to the strictness or leniency of the determination received.

18. An image acquisition unit that acquires the captured image, Based on the acquired image, an object region information detection unit detects region information of a region containing an object, A skeleton information detection unit detects the skeleton information of the object based on the acquired image, A threshold range calculation unit calculates a threshold range based on the detected region information, A determination unit determines whether or not the joint points in the skeletal information are the joint points of the object, based on the threshold range calculated by the threshold range calculation unit and the skeletal information detected by the skeletal information detection unit. A joint point estimation device having the following features.

19. Step (a) to acquire the captured image, (b) A step of detecting region information of a region containing an object based on the acquired image, (c) A step of detecting the skeletal information of the object based on the acquired image, (d) A step of calculating a threshold range based on the detected region information, Step (e) is to determine whether the joint point in the skeletal information is the joint point of the object, based on the threshold range calculated in step (d) and the skeletal information detected in step (c), An articular point estimation program for causing a computer to perform a process that includes the following.

20. A method performed by an articular point estimation system, Step (a) to acquire the captured image, (b) A step of detecting region information of a region containing an object based on the acquired image, (c) A step of detecting the skeletal information of the object based on the acquired image, (d) A step of calculating a threshold range based on the detected region information, Step (e) is to determine whether the joint point in the skeletal information is the joint point of the object, based on the threshold range calculated in step (d) and the skeletal information detected in step (c), A method for estimating joint points.

Images