Image processing device, image processing method, and program

Abstract

The present invention provides an image processing device (10) that has: a skeletal structure detecting unit (11) that performs processing for detecting key points of a human body included in an image; a similarity degree calculating unit (12) that calculates a similarity degree between the attitude or motion of the human body detected from the image on the basis of the detected key points, and the attitude or motion of a human body represented in a previously registered template image; an identifying unit (13) that identifies a location in an image showing a human body for which the similarity degree between any attitude or motion of the human body represented in the template image is less than a first threshold; and an output unit (14) that outputs a partial image, from which is cut out the location identified from the information that indicates the identified location or from the image, as a candidate of a template image to be additionally registered in an assessment device that assesses the attitude or motion of the human body detected from the image on the basis of the attitude or motion of the human body represented in the template image.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Techniques related to the present invention are disclosed in Patent Documents 1 to 3 and Non-Patent Document 1.

[0003] Patent Document 1 discloses a technology for calculating feature amounts for each of a plurality of key points of a human body included in an image, searching for images containing human bodies with similar postures or movements based on the calculated feature amounts, and classifying images with similar postures or movements. Also, Non-Patent Document 1 discloses a technology related to human skeleton estimation.

[0004] Patent Document 2 discloses a technology in which, upon obtaining multiple images of a specified area and information indicating changes in the situation in the specified area, the multiple images are classified based on the information indicating changes in the situation in the specified area, and a classifier is trained to determine the situation in the specified area from the images using at least some of the multiple images according to the classification results.

[0005] Patent Document 3 discloses a technique for detecting a change in the state of a target person based on an input image, and determining an abnormal state when it is detected that the change in state of the target has occurred in multiple people. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2021 / 084677 [Patent Document 2] Patent Publication No. 2021-87031 [Patent Document 3] International Publication No. 2015 / 198767 [Non-patent literature]

[0007] [Non-Patent Document 1] Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, P. 7291-7299 Summary of the Invention [Problem to be solved by the invention]

[0008] According to the technology disclosed in the above-mentioned Patent Document 1, an image including a human body in a desired posture or movement is registered in advance as a template image, and thereby a human body in a desired posture or movement can be detected from an image to be processed. As a result of studying the technology disclosed in Patent Document 1, the present inventor has newly found that there is room for improvement in the workability of searching for an image including a human body in a desired posture or movement different from the posture or movement shown in a registered template image when additionally registering such an image as a template image.

[0009] None of the above Patent Documents 1 to 3 and Non-Patent Document 1 discloses the problems related to template images and the means for solving them, and therefore there is a problem in that the above problems cannot be solved.

[0010] In view of the above-mentioned problems, one example of the object of the present invention is to provide an image processing device, an image processing method, and a program that solve the problem of workability involved in registering as a template image an image that includes a human body in a desired posture or desired movement that differs from the posture or movement shown in a registered template image. [Means for solving the problem]

[0011] According to one aspect of the present invention, a skeletal structure detection means for detecting key points of a human body included in an image; a similarity calculation means for calculating a similarity between a posture or movement of the human body detected from the image and a posture or movement of the human body shown in a pre-registered template image based on the detected key points; an identification means for identifying a location in the image where a human body is captured, the similarity of which to the posture or movement of the human body shown in any of the template images is less than a first threshold; an output means for outputting information indicating the specified location or a partial image obtained by cutting out the specified location from the image as a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body indicated by the template image; An image processing apparatus is provided, comprising:

[0012] According to another aspect of the present invention, The computer The process detects key points of the human body contained in the image, Calculating a similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in a pre-registered template image based on the detected key points; Identifying a location in the image where a human body is captured, the similarity to the posture or movement of the human body shown in any of the template images being less than a first threshold; outputting information indicating the specified location or a partial image obtained by cutting out the specified location from the image as a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body indicated by the template image; An image processing method is provided.

[0013] According to another aspect of the present invention, Computer, a skeletal structure detection means for detecting key points of a human body included in an image; a similarity calculation means for calculating a similarity between a posture or movement of the human body detected from the image and a posture or movement of the human body shown in a pre-registered template image based on the detected key points; an identification means for identifying a location in the image in which a human body is captured, the similarity of which to the posture or movement of the human body shown in any of the template images being less than a first threshold; an output means for outputting information indicating the specified location or a partial image obtained by cutting out the specified location from the image as a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body indicated by the template image; A program is provided to function as a [Effects of the Invention]

[0014] According to one aspect of the present invention, an image processing device, an image processing method, and a program are provided that solve the problem of workability involved in registering as a template image an image that includes a human body in a desired posture or movement that differs from the posture or movement shown in a registered template image. [Brief explanation of the drawings]

[0015] The above-mentioned objects, as well as other objects, features, and advantages, will become more apparent from the following description of the preferred embodiments and the accompanying drawings.

[0016] [Figure 1] FIG. 1 is a diagram illustrating an example of a functional block diagram of an image processing apparatus. [Figure 2] FIG. 2 is a diagram for explaining processing contents of the image processing device. [Figure 3] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing apparatus. [Figure 4] 1 is a diagram illustrating an example of a skeletal structure of a human body model detected by an image processing device. [Figure 5] FIG. 2 is a diagram illustrating an example of a skeletal structure of a human body model detected by an image processing device. [Figure 6]FIG. 2 is a diagram illustrating an example of a skeletal structure of a human body model detected by an image processing device. [Figure 7] FIG. 2 is a diagram illustrating an example of a skeletal structure of a human body model detected by an image processing device. [Figure 8] FIG. 10 is a diagram illustrating an example of feature amounts of key points calculated by the image processing device. [Figure 9] FIG. 10 is a diagram illustrating an example of feature amounts of key points calculated by the image processing device. [Figure 10] FIG. 10 is a diagram illustrating an example of feature amounts of key points calculated by the image processing device. [Figure 11] FIG. 2 is a diagram schematically illustrating an example of information output by the image processing device. [Figure 12] 10 is a flowchart illustrating an example of a processing flow of the image processing device. [Figure 13] FIG. 2 is a diagram for explaining processing contents of the image processing device. [Figure 14] 10 is a flowchart illustrating an example of a processing flow of the image processing device. [Figure 15] FIG. 1 is a diagram illustrating an example of a functional block diagram of an image processing apparatus. [Figure 16] FIG. 2 is a diagram schematically illustrating an example of information output by the image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0018] First Embodiment 1 is a functional block diagram showing an overview of an image processing device 10 according to the first embodiment. As shown in FIG. 1, the image processing device 10 includes a skeletal structure detection unit 11, a similarity calculation unit 12, an identification unit 13, and an output unit 14.

[0019] The skeletal structure detection unit 11 performs a process to detect key points of the human body contained in the image. Based on the detected key points, the similarity calculation unit 12 calculates the similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in pre-registered template images. The identification unit 13 identifies a location in the image containing a human body whose similarity to the posture or movement of the human body shown in any template image is less than a first threshold. The output unit 14 outputs information indicating the location identified by the identification unit 13, or a partial image cut out from the image of the location identified, as a candidate template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body shown in the template images.

[0020] According to this image processing device 10, it is possible to solve the problem of workability involved in registering as a template image an image including a human body in a desired posture or movement that differs from the posture or movement shown in a registered template image.

[0021] <Second embodiment> "overview" The image processing device 10 calculates the similarity between the posture or movement of a human body contained in an image (hereinafter simply referred to as "image") that is the basis of the template image and the posture or movement of a human body shown in a pre-registered template image, and then identifies a location in the image where a human body appears, where the similarity to the posture or movement of a human body shown in any template image is less than a first threshold. The image processing device 10 then outputs information indicating the identified location, or a partial image extracted from the image of the identified location, as a candidate template image to be additionally registered for the determination device. The determination device performs detection processing using the registered template image, and if the similarity is equal to or greater than the first threshold, determines that the posture or movement of the human body detected from the image and the posture or movement of the human body shown in the template image are the same or the same type of posture or movement.

[0022] According to this image processing device 10, it is possible to identify a location in the image where a human body is detected from a group of human bodies that is not determined to have the same or the same type of posture or movement as the human body shown in any of the template images, and output information related to the identified location. This will be described in more detail with reference to FIG. 2. In the second embodiment, as shown in FIG. 2, the group of human bodies detected from the image is classified into (1) a group of human bodies that is determined to have the same or the same type of posture or movement as the human body shown in any of the template images, and (2) a group of other human bodies. The group of (2) other human bodies is a group of human bodies that is not determined to have the same or the same type of posture or movement as the human body shown in any of the template images. In this embodiment, the device identifies a location in the image where a human body included in the group of (2) other human bodies is detected, and outputs information related to the identified location.

[0023] "Hardware Configuration" Next, an example of the hardware configuration of image processing device 10 will be described. Each functional unit of image processing device 10 is realized by any combination of hardware and software, centered around a CPU (Central Processing Unit) of any computer, memory, programs loaded into memory, a storage unit such as a hard disk that stores the programs (this can store programs that are pre-loaded when the device is shipped, as well as programs downloaded from storage media such as CDs (Compact Discs) or servers on the Internet), and a network connection interface. Those skilled in the art will understand that there are many variations in the implementation methods and devices.

[0024] FIG. 3 is a block diagram illustrating an example of the hardware configuration of an image processing device 10. As shown in FIG. 3, the image processing device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The image processing device 10 does not necessarily have to have the peripheral circuit 4A. Note that the image processing device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices can have the above hardware configuration.

[0025] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to transmit and receive data among them. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and an interface for outputting information to an output device, an external device, an external server, etc. Examples of input devices include a keyboard, a mouse, a microphone, physical buttons, a touch panel, etc. Examples of output devices include a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0026] "Function Configuration" 1 is a functional block diagram showing an overview of an image processing device 10 according to the second embodiment. As shown in FIG. 1, the image processing device 10 includes a skeletal structure detection unit 11, a similarity calculation unit 12, an identification unit 13, and an output unit 14.

[0027] The skeletal structure detection unit 11 performs processing to detect key points of the human body contained in the image.

[0028] An "image" is an image that is the basis for a template image. The template image is an image that is registered in advance in the technology disclosed in the above-mentioned Patent Document 1, and is an image that includes a human body in a desired posture or desired movement (a posture or movement that the user wants to detect). The image may be a moving image made up of multiple frame images, or a single still image.

[0029] The skeletal structure detection unit 11 detects N (N is an integer equal to or greater than 2) key points of the human body contained in the image. When a moving image is to be processed, the skeletal structure detection unit 11 performs processing to detect key points for each frame image. This processing by the skeletal structure detection unit 11 is realized using the technology disclosed in Patent Document 1. Although details are omitted, the technology disclosed in Patent Document 1 detects the skeletal structure using a skeletal estimation technology such as OpenPose disclosed in Non-Patent Document 1. The skeletal structure detected by this technology consists of "key points," which are characteristic points such as joints, and "bones (bone links)," which indicate the links between key points.

[0030] Fig. 4 shows the skeletal structure of a human body model 300 detected by the skeletal structure detection unit 11, and Figs. 5 to 7 show examples of detected skeletal structures. The skeletal structure detection unit 11 detects the skeletal structure of a human body model (two-dimensional skeletal model) 300 as shown in Fig. 4 from a two-dimensional image using a skeletal estimation technique such as OpenPose. The human body model 300 is a two-dimensional model made up of key points such as a person's joints and bones connecting each key point.

[0031] The skeletal structure detection unit 11, for example, extracts feature points that can be key points from an image, and detects N key points of the human body by referring to information obtained by machine learning of the image of the key points. The N key points to be detected are determined in advance. The number of key points to be detected (i.e., the number N) and which parts of the human body are to be detected as key points vary, and any variation can be adopted.

[0032] In the following, as shown in Figure 4, the head A1, neck A2, right shoulder A31, left shoulder A32, right elbow A41, left elbow A42, right hand A51, left hand A52, right waist A61, left waist A62, right knee A71, left knee A72, right foot A81, and left foot A82 are defined as N key points (N=14) to be detected. In the human body model 300 shown in FIG. 3, the bones of the person that connect these key points are further defined as follows: bone B1 connecting the head A1 and neck A2; bone B21 and bone B22 connecting the neck A2 and the right shoulder A31 and left shoulder A32, respectively; bone B31 and bone B32 connecting the right shoulder A31 and left shoulder A32 and the right elbow A41 and left elbow A42, respectively; bone B41 and bone B42 connecting the right elbow A41 and left elbow A42 and the right hand A51 and left hand A52, respectively; bone B51 and bone B52 connecting the neck A2 and the right hip A61 and left hip A62, respectively; bone B61 and bone B62 connecting the right hip A61 and left hip A62 and the right knee A71 and left knee A72, respectively; and bone B71 and bone B72 connecting the right knee A71 and left knee A72 and the right foot A81 and left foot A82, respectively.

[0033] Fig. 5 shows an example of detecting a person standing upright. In Fig. 5, the person standing upright is imaged from the front, and bones B1, B51 and B52, B61 and B62, and B71 and B72 are detected without overlapping, and bones B61 and B71 of the right foot are slightly more bent than bones B62 and B72 of the left foot.

[0034] Fig. 6 shows an example of detecting a person in a crouching position. In Fig. 6, the image of the person crouching is captured from the right side, and bones B1, B51 and B52, B61 and B62, and B71 and B72 are detected as seen from the right side, with bones B61 and B71 of the right foot and bones B62 and B72 of the left foot being significantly bent and overlapping.

[0035] Fig. 7 shows an example of detecting a person who is lying down. In Fig. 7, the person lying down is imaged from the diagonal front left, and bones B1, B51 and B52, B61 and B62, and B71 and B72 seen from the diagonal front left are detected, with bones B61 and B71 of the right foot and bones B62 and B72 of the left foot being bent and overlapping.

[0036] Returning to Figure 1, the similarity calculation unit 12 calculates the similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in a pre-registered template image, based on the key points detected by the skeletal structure detection unit 11.

[0037] There are various ways to calculate the similarity of the posture or movement of the human body, and any technique can be adopted. For example, the technique disclosed in Patent Document 1 may be adopted. Alternatively, the same technique as that used by a determination device may be adopted, which calculates the similarity between the posture or movement of the human body shown in a template image and the posture or movement of the human body detected from the image, and detects a human body whose similarity is equal to or greater than a first threshold as the same as the human body shown in the template image, or as a human body with the same type of posture or movement. An example will be described below, but the present invention is not limited to this.

[0038] As an example, the similarity calculation unit 12 may calculate the feature of the skeletal structure indicated by the detected key points, and calculate the similarity between the feature of the skeletal structure of the human body detected from the image and the feature of the skeletal structure of the human body indicated by the template image, thereby calculating the similarity between the postures of the two human bodies.

[0039] Skeletal structure features indicate the characteristics of a person's skeleton and are used to classify a person's state (posture and movement) based on their skeleton. Typically, these features include multiple parameters. The features may be the features of the entire skeletal structure, the features of a portion of the skeletal structure, or multiple features for each part of the skeletal structure. The feature calculation method may be any method, such as machine learning or normalization, and normalization may involve finding a minimum or maximum value. Examples of feature values ​​include features obtained by machine learning of the skeletal structure, the size of the skeletal structure from the head to the feet on the image, the relative positions of multiple key points in the vertical direction of the skeletal region containing the skeletal structure on the image, and the relative positions of multiple key points in the horizontal direction of the skeletal region. The size of the skeletal structure refers to the vertical height or area of ​​the skeletal region containing the skeletal structure on the image. The vertical direction (height direction or vertical direction) refers to the up-down direction (Y-axis direction) in the image, for example, the direction perpendicular to the ground (reference plane). The left-right direction (horizontal direction) is the left-right direction in the image (X-axis direction), and is, for example, a direction parallel to the ground.

[0040] In order to perform the classification desired by the user, it is preferable to use features that are robust to the judgment process. For example, if the user wants a judgment that is not dependent on the person's orientation or body shape, features that are robust to the person's orientation and body shape may be used. By learning the skeletons of people facing in various directions in the same posture or the skeletons of people with various body shapes in the same posture, or by extracting features only in the up-down direction of the skeleton, it is possible to obtain features that are not dependent on the person's orientation or body shape. An example of a process for calculating features of a skeletal structure is disclosed in Patent Document 1.

[0041] 8 shows an example of the feature amounts of each of the multiple key points calculated by the similarity calculation unit 12. A collection of the feature amounts of the multiple key points becomes the feature amount of the skeletal structure. Note that the feature amounts of the key points exemplified here are merely examples and are not limited to these.

[0042] In this example, the feature values ​​of keypoints indicate the relative positional relationships of multiple keypoints in the vertical direction of the skeletal region containing the skeletal structure on the image. Because the neck keypoint A2 is used as the reference point, the feature value of keypoint A2 is 0.0. The feature values ​​of keypoint A31 on the right shoulder and keypoint A32 on the left shoulder, which are at the same height as the neck, are also 0.0. The feature value of keypoint A1 on the head, which is higher than the neck, is -0.2. The feature values ​​of keypoints A51 on the right hand and A52 on the left hand, which are lower than the neck, are 0.4, and the feature values ​​of keypoints A81 on the right foot and A82 on the left foot are 0.9. If the person raises their left hand from this position, as shown in Figure 9, the left hand will be higher than the reference point, and the feature value of keypoint A52 on the left hand will be -0.4. However, because normalization is performed using only the Y-axis coordinate, the feature values ​​do not change even if the width of the skeletal structure changes, as shown in Figure 10, compared to Figure 8. That is, the feature amount (normalized value) in this example indicates the feature in the height direction (Y direction) of the skeletal structure (keypoint), and is not affected by changes in the lateral direction (X direction) of the skeletal structure.

[0043] There are various methods for calculating the similarity of postures indicated by such feature amounts. For example, after calculating the similarity of feature amounts for each keypoint, the similarity of postures may be calculated based on the similarities of feature amounts of multiple keypoints. For example, the average value, maximum value, minimum value, mode, median, weighted average value, weighted sum, etc. of the similarities of feature amounts of multiple keypoints may be calculated as the similarity of postures. When calculating the weighted average value or weighted sum, the weight of each keypoint may be set by the user or may be determined in advance.

[0044] Furthermore, movement is expressed as a change in a plurality of postures over time. Therefore, the similarity calculation unit 12 may, for example, calculate the posture similarity for each combination of a plurality of corresponding frame images using the above-described method, and then calculate, as the movement similarity, a statistical value (such as an average, maximum, minimum, mode, median, weighted average, or weighted sum) of the posture similarity calculated for each combination of a plurality of frame images.

[0045] Returning to FIG. 1, the identification unit 13 identifies a location in the image containing a human body whose similarity to the posture or movement of the human body shown in any of the template images is less than a first threshold, as a candidate for a template image to be additionally registered for the determination device. Specifically, the identification unit 13 compares the similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in each of the multiple template images with a first threshold. Then, based on the result of the comparison, the identification unit 13 identifies a location in the image containing a human body whose similarity to the posture or movement of the human body shown in any of the template images is less than the first threshold.

[0046] The determination device determines the posture or movement of the human body detected from the image based on the posture or movement of the human body shown in the template image. Specifically, if the similarity is equal to or greater than a first threshold, the determination device determines that the posture or movement of the human body detected from the image and the posture or movement of the human body shown in the template image are the same or the same type of posture or movement. In other words, the identification unit 13 identifies a location in the image containing a human body that is not determined by the determination device to be the same or the same type of posture or movement as the posture or movement of the human body shown in any of the template images, among the group of human bodies detected from the image.

[0047] If the image is a still image, the "location identified by the identification unit 13" is a partial area within one still image. In this case, for each still image, the location is indicated by, for example, coordinates in a coordinate system set for the still image. On the other hand, if the image is a moving image, the "location identified by the identification unit 13" is a partial area within each of some of the frame images among the multiple frame images that make up the moving image. In this case, for each moving image, the location is indicated by, for example, information indicating some of the frame images among the multiple frame images (frame identification information, elapsed time from the beginning, etc.) and coordinates in a coordinate system set for the frame image.

[0048] The output unit 14 outputs information indicating the portion identified by the identification unit 13 or a partial image obtained by cutting out the portion identified by the identification unit 13 from the image, as a candidate for a template image to be additionally registered in the determination device. When the output unit 14 outputs a partial image, the image processing device 10 may have a processing unit that cuts out the portion identified by the identification unit 13 from the image to generate a partial image. Then, the output unit 14 may output the partial image generated by the processing unit.

[0049] The above-mentioned "location identified by the identification unit 13," i.e., a location in an image showing a human body whose similarity to the posture or movement of the human body shown in any template image is less than the first threshold, becomes a candidate for the template image. Based on the above information or the above partial image, the user can browse the above locations and select from them a location that includes a human body in a desired posture or movement as a template image.

[0050] 11 is a schematic diagram illustrating an example of information output by the output unit 14. In the example illustrated in FIG. 11, human body identification information for identifying the detected human bodies from one another and attribute information for each human body are displayed in association with one another. As an example of the attribute information, information indicating the location in the image (information indicating the location where the above-mentioned human body appears) and the date and time the image was taken are displayed. The attribute information may also include information indicating the installation location (taking position) of the camera that took the image (e.g., rear of the No. 102 bus, entrance to X Park, etc.) and attribute information of the person calculated by image analysis (e.g., gender, age group, body type, etc.).

[0051] Next, an example of the flow of processing by the image processing device 10 will be described with reference to the flowchart of FIG.

[0052] The image processing device 10 performs a process of detecting key points of the human body contained in the image (S10), and then calculates the similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in a pre-registered template image based on the detected key points (S11).

[0053] Next, the image processing device 10 identifies a location in the image containing a human body whose similarity to the posture or movement of the human body shown in any of the template images is less than a first threshold, as a candidate template image to be additionally registered for the determination device (S12). Specifically, the image processing device 10 compares the similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in each of the multiple template images with a first threshold. Then, based on the result of the comparison, the image processing device 10 identifies a location in the image containing a human body whose similarity to the posture or movement of the human body shown in any of the template images is less than the first threshold. Note that, if the similarity is equal to or greater than the first threshold, the determination device determines that the posture or movement of the human body detected from the image and the posture or movement of the human body shown in the template image are the same or the same type of posture or movement.

[0054] Then, the image processing device 10 outputs information indicating the location identified in S12, or a partial image obtained by cutting out the location identified in S12 from the image (S13).

[0055] "Action and effect" The image processing device 10 of the second embodiment achieves the same effects as those of the first embodiment. Moreover, the image processing device 10 of the second embodiment can output information about a location in an image that shows a human body that is determined by a determination device to have the same or the same type of posture or movement as the posture or movement of a human body shown in any template image, among a group of human bodies detected from an image.

[0056] This will be explained in more detail using FIG. 2. In the second embodiment, as shown in FIG. 2, a group of human bodies detected from an image is classified into (1) a group of human bodies determined by the determination device to have the same or similar type of pose or movement as the human body shown in any of the template images, and (2) a group of other human bodies. The group of (2) other human bodies is a group of human bodies not determined by the determination device to have the same or similar type of pose or movement as the human body shown in any of the template images. The image processing device 10 of the second embodiment can identify a location in an image where a human body included in the group of (2) other human bodies appears, and output information about the identified location. A user can view the identified location and select a location containing a human body with a desired pose or desired movement as a template image. As a result, the problem of ease of use in registering an image containing a human body with a desired pose or desired movement that differs from the pose or movement shown in a registered template image is resolved.

[0057] <Third embodiment> The image processing device 10 of the third embodiment specifies a part of the location in the image specified by the image processing device 10 of the second embodiment as a candidate for a template image to be additionally registered for the determination device.

[0058] In the third embodiment, as shown in Fig. 13, a group of human bodies detected from an image is classified into (1) a group of human bodies determined to have the same or the same type of posture or movement as the posture or movement of a human body shown in any of the template images, (2-1) a group of human bodies with similar postures or movements but not determined to have the same or the same type of posture or movement as the posture or movement of a human body shown in any of the template images, and (2-2) a group of other human bodies. That is, in the third embodiment, the (2) group of other human bodies (see Fig. 2) in the second embodiment is classified into (2-1) a group of human bodies with similar postures or movements but not determined to have the same or the same type of posture or movement as the posture or movement of a human body shown in any of the template images, and (2-2) a group of other human bodies.

[0059] The group of (2-2) other human bodies is a group of human bodies that are not determined to be the same as or have the same type of posture or movement as the posture or movement of the human bodies shown in any template image, and have dissimilar postures or movements. In this embodiment, a location in the image where a human body included in the group of (2-2) other human bodies appears is identified, and information about the identified location is output. This will be described in detail below.

[0060] The identification unit 13 identifies a location in an image that contains a human body (a human body belonging to the set (2-2) in Figure 13) that does not satisfy the first similarity condition with the posture or movement of a human body shown in any of the template images, among human bodies (human bodies belonging to the sets (2-1) and (2-2) in Figure 13) whose similarity with the posture or movement of a human body shown in any of the template images is less than a first threshold, as a candidate template image to be additionally registered for the determination device.

[0061] The identification unit 13 identifies human bodies belonging to the sets (2-1) and (2-2) in FIG. 13 from among human bodies detected from an image using the method described in the second embodiment. Next, the identification unit 13 determines, for each identified human body, whether the posture or movement of the human body shown in any template image satisfies a first similarity condition. Then, based on the determination result, the identification unit 13 identifies human bodies belonging to the set (2-2) in FIG. 13 and identifies the location in the image where the identified human body appears. Human bodies that satisfy the first similarity condition are identified as human bodies belonging to the set (2-1) in FIG. 13, and human bodies that do not satisfy the first similarity condition are identified as human bodies belonging to the set (2-2) in FIG. 13.

[0062] The first similarity condition is: "The degree of similarity to the posture or movement of the human body shown in the template image is equal to or greater than the second threshold and less than the first threshold." "The similarity between the posture or movement of the human body shown in the template image calculated based on some of the multiple key points (N key points) detected from each human body is equal to or greater than a third threshold value." "The similarity between the posture or movement of the human body shown in the template image calculated by taking into account the weighted values ​​assigned to each of the multiple key points detected from each human body is equal to or greater than a fourth threshold value," and "The template image includes a plurality of frame images each showing a human body in a posture where the similarity to the posture of the human body shown in each of the frame images at a predetermined rate or more among the plurality of frame images included in the template image, which is a moving image, is equal to or greater than a fifth threshold value." Contains at least one of the following:

[0063] When multiple conditions from the above examples are included, the first similarity condition can be a combination of multiple conditions connected by a logical operator such as "or." Each of the above examples will be explained below.

[0064] "The similarity to the posture or movement of the human body shown in the template image is equal to or greater than the second threshold and less than the first threshold." The "similarity" in this condition is a value calculated using the same method as that used by the similarity calculation unit 12 described in the second embodiment. The second threshold value is a value smaller than the first threshold value.

[0065] By appropriately setting the second threshold, it is possible to detect human bodies (human bodies belonging to the set (2-1) in FIG. 13) that have similar postures or movements but are not determined to be the same as or of the same type as the postures or movements of the human bodies shown in any of the template images. Then, by removing the human bodies belonging to the set (2-1) in FIG. 13 from the human bodies belonging to the sets (2-1) and (2-2) in FIG. 13 identified by the method described in the second embodiment, it is possible to identify human bodies belonging to the set (2-2) in FIG.

[0066] "The similarity between the posture or movement of the human body shown in the template image calculated based on some of the multiple key points (N key points) detected from each human body is equal to or greater than the third threshold." The "similarity" under this condition is a value calculated based on some of the multiple keypoints (N keypoints) to be detected. The similarity under this condition can be calculated using the same method as the calculation method by the similarity calculation unit 12 described in the second embodiment, except that only the feature amounts of some of the multiple keypoints (N keypoints) are used.

[0067] Which key points to use is a design decision, but for example, the user may be able to specify them. The user can specify key points for body parts that they want to emphasize (e.g., the upper body) and remove key points for body parts that they do not want to emphasize (e.g., the lower body).

[0068] By appropriately setting the third threshold, it is possible to detect human bodies (human bodies belonging to the set (2-1) in FIG. 13) that have the same or similar body part posture or movement but are not determined to be the same or the same type of posture or movement as the human bodies shown in any template image. Then, by removing the human bodies belonging to the set (2-1) in FIG. 13 from the human bodies belonging to the sets (2-1) and (2-2) in FIG. 13 identified by the method described in the second embodiment, it is possible to identify human bodies belonging to the set (2-2) in FIG.

[0069] "The degree of similarity between the posture or movement of the human body shown in the template image calculated by taking into account the weighted values ​​assigned to each of the multiple key points detected from each human body is equal to or greater than the fourth threshold value." The "similarity" in this condition is a value calculated by assigning weights to multiple key points (N key points) to be detected. For example, the similarity of the feature amount for each key point is calculated using the same calculation method as that used by the similarity calculation unit 12 described in the second embodiment, and then the weighted average or weighted sum of the similarities of the feature amounts of the multiple key points is calculated as the posture similarity using the weighted values. The weight of each key point may be set by the user or may be determined in advance.

[0070] By appropriately setting the fourth threshold, it is possible to detect human bodies (human bodies belonging to the set (2-1) in FIG. 13) that are not determined to be the same or the same type of pose or movement as the pose or movement of the human bodies shown in any template image, but that have the same or similar pose or movement when weighting is placed on a part of the body. Then, by removing the human bodies belonging to the set (2-1) in FIG. 13 from the human bodies belonging to the sets (2-1) and (2-2) in FIG. 13 identified by the method described in the second embodiment, it is possible to identify human bodies belonging to the set (2-2) in FIG.

[0071] "The template image is a moving image and includes a plurality of frame images each showing a human body in a pose where the similarity to the pose of the human body shown in each of the frame images at a predetermined rate or more is equal to or greater than a fifth threshold." This condition is used when the image and the template images are moving images, and the movement of the human body is indicated by the time-varying posture of the human body shown in each of the plurality of template images included in the moving image.

[0072] For example, if a template image is composed of M frame images, the condition will be satisfied if the frame images each contain a human body in a posture similar to the posture of the human body shown in at least a predetermined percentage (e.g., 70% or more) of the M frame images (the similarity is at least a fifth threshold). The method for calculating the posture similarity for each combination of corresponding frame images can be the method described in the second embodiment.

[0073] By appropriately setting the fifth threshold and the predetermined ratio, it is possible to detect human bodies (human bodies belonging to the group (2-1) in FIG. 13) that are not determined to be the same or have the same type of posture or movement as the human body movements shown in any of the template images but that have the same or similar movements as the human body movements in a certain time period in the template images (moving images).Then, by removing the human bodies belonging to the group (2-1) in FIG. 13 from the human bodies belonging to the groups (2-1) and (2-2) in FIG. 13 identified by the method described in the second embodiment, it is possible to identify human bodies belonging to the group (2-2) in FIG.

[0074] Next, an example of the flow of processing by the image processing device 10 will be described with reference to the flowchart of FIG.

[0075] The image processing device 10 performs a process of detecting key points of the human body contained in the image (S20), and then calculates the similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in a pre-registered template image based on the detected key points (S21).

[0076] Next, the image processing device 10 identifies, from the detected human bodies, human bodies whose similarity to the posture or movement of the human body shown in any of the template images is less than a first threshold (S22). Specifically, the image processing device 10 compares the similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in each of the multiple template images with the first threshold. Then, based on the result of the comparison, the image processing device 10 identifies human bodies whose similarity to the posture or movement of the human body shown in any of the template images is less than the first threshold.

[0077] Next, the image processing device 10 identifies a location in the image where a human body appears that does not satisfy the first similarity condition with the posture or movement of the human body shown in any of the template images among the human bodies identified in S22, as a candidate template image to be additionally registered for the determination device (S23). Specifically, the image processing device 10 determines, for each human body identified in S22, whether the first similarity condition is satisfied with the posture or movement of the human body shown in any of the template images. Then, based on the determination result, the image processing device 10 identifies a location in the image where a human body appears that does not satisfy the first similarity condition with the posture or movement of the human body shown in any of the template images among the human bodies identified in S22.

[0078] Then, the image processing apparatus 10 outputs information indicating the portion identified in S23, or a partial image obtained by cutting out the portion identified in S23 from the image (S24).

[0079] Other configurations of the image processing device 10 of the third embodiment are similar to those of the image processing device 10 of the first and second embodiments.

[0080] The image processing device 10 of the third embodiment achieves the same effects as those of the first and second embodiments. Moreover, the image processing device 10 of the third embodiment can output information about a location in an image that contains a human body that is not determined by a determination device to have the same or the same type of posture or movement as the posture or movement of the human body shown in any of the template images and that does not resemble the posture or movement of the human body shown in any of the template images.

[0081] This will be explained in more detail using FIG. 13. In the third embodiment, as shown in FIG. 13, a group of human bodies detected from an image is classified into (1) a group of human bodies determined to have the same or the same type of posture or movement as the human bodies shown in any of the template images, (2-1) a group of human bodies with similar postures or movements but not determined to have the same or the same type of posture or movement as the human bodies shown in any of the template images, and (2-2) a group of other human bodies. The (2-2) group of other human bodies is a group of human bodies that are not determined by the determination device to have the same or the same type of posture or movement as the human bodies shown in any of the template images and are dissimilar to the postures or movements of the human bodies shown in any of the template images. The image processing device 10 of the third embodiment can identify a location in an image where a human body included in the (2-2) group of other human bodies appears and output information about the identified location. A user can view the identified locations and select a location containing a human body with a desired posture or desired movement from among them as a template image. As a result, the problem of workability in registering as a template image an image including a human body in a desired posture or movement different from the posture or movement shown in a registered template image is solved.

[0082] <Fourth embodiment> The image processing device 10 of this embodiment has a function of grouping multiple human bodies that appear in a location in an image identified by the method of any one of the first to third embodiments based on the similarity of their postures or movements, and outputting the results. This will be described in detail below.

[0083] 15 shows an example of a functional block diagram of the image processing device 10 of this embodiment. As shown in the figure, the image processing device 10 includes a skeletal structure detection unit 11, a similarity calculation unit 12, an identification unit 13, an output unit 14, and a grouping unit 15.

[0084] The grouping unit 15 groups multiple human bodies appearing in the location in the image identified by the identification unit 13 based on the similarity of their postures or movements. The grouping unit 15 groups bodies with similar postures or movements together to create a group. This grouping can be achieved by using the classification technology disclosed in Patent Document 1.

[0085] The output unit 14 further outputs the results of the grouping by the grouping unit 15. Fig. 16 shows an example of information output by the output unit 14. In the example shown in the figure, multiple human bodies appearing in locations in the image identified by the identification unit 13 are classified into three groups. For example, as shown in Fig. 16, posture areas WA1 to WA3 for each posture (each group) are displayed in the display window W1, and human bodies corresponding to each posture are displayed in the posture areas WA1 to WA3.

[0086] Other configurations of the image processing device 10 of the fourth embodiment are similar to those of the image processing device 10 of the first to third embodiments.

[0087] The image processing device 10 of the fourth embodiment achieves the same effects as those of the first to third embodiments. Furthermore, the image processing device 10 of the fourth embodiment can group multiple human bodies appearing in a specified location in an image based on the similarity of their postures or movements, and output the results. Based on this information, the user can easily understand what postures and movements of human bodies are included in the candidate template images. As a result, the problem of ease of operation involved in registering, as a template image, an image including a human body with a desired posture or movement that differs from the posture or movement shown in a registered template image is resolved.

[0088] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.

[0089] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed to the extent that the content is not affected. Furthermore, the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0090] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. A skeletal structure detection means for detecting key points of the human body contained in the image; a similarity calculation means for calculating a similarity between a posture or movement of the human body detected from the image and a posture or movement of the human body shown in a pre-registered template image based on the detected key points; an identification means for identifying a location in the image where a human body is captured, the similarity of which to the posture or movement of the human body shown in any of the template images is less than a first threshold; an output means for outputting information indicating the specified location or a partial image obtained by cutting out the specified location from the image as a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body indicated by the template image; An image processing device having: 2. The image processing device described in 1, wherein the identification means identifies a location within the image that depicts a human body that does not satisfy a first similarity condition with the posture or movement of the human body shown in any of the template images, among human bodies whose similarity with the posture or movement of the human body shown in any of the template images is less than the first threshold value. 3. The image processing device according to 2, wherein the first similarity condition includes that the similarity is equal to or greater than a second threshold and less than the first threshold. 4. An image processing device according to 2 or 3, wherein the first similarity condition includes that the similarity between the posture or movement of the human body shown in the template image calculated based on some of the multiple key points detected from each human body is equal to or greater than a third threshold. 5. An image processing device described in any one of 2 to 4, wherein the first similarity condition includes that the similarity to the posture or movement of the human body shown in the template image, calculated taking into account weighting values ​​assigned to each of the multiple key points detected from each human body, is equal to or greater than a fourth threshold. 6. The image and the template image are moving images, and the movement of the human body is represented by the time change of the posture of the human body represented by each of the plurality of template images included in the moving image; 6. An image processing device according to any one of 2 to 5, wherein the first similarity condition is to include a plurality of frame images each showing a human body in a posture where the similarity to the posture of the human body shown in each of the frame images of a predetermined percentage or more of the plurality of frame images included in the template image is equal to or greater than a fifth threshold. 7. The apparatus further comprises a grouping means for grouping a plurality of human bodies appearing in the specified location based on the similarity of their postures or movements, The output means further outputs the results of the grouping. 7. An image processing device according to any one of 1 to 6. 8. The computer The process detects key points of the human body contained in the image, Calculating a similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in a pre-registered template image based on the detected key points; Identifying a location in the image where a human body is captured, the similarity to the posture or movement of the human body shown in any of the template images being less than a first threshold; outputting information indicating the specified location or a partial image obtained by cutting out the specified location from the image as a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body indicated by the template image; Image processing methods. 9. Computer a skeletal structure detection means for detecting key points of a human body included in an image; a similarity calculation means for calculating a similarity between a posture or movement of the human body detected from the image and a posture or movement of the human body shown in a pre-registered template image based on the detected key points; an identification means for identifying a location in the image in which a human body is captured, the similarity of which to the posture or movement of the human body shown in any of the template images being less than a first threshold; an output means for outputting information indicating the specified location or a partial image obtained by cutting out the specified location from the image as a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body indicated by the template image; A program that functions as a [Explanation of symbols]

[0091] 10 Image processing device 11 Skeletal structure detection unit 12 Similarity calculation unit 13 Specific section 14 Output section 15 Grouping section 1A processor 2A Memory 3A input / output I / F 4A peripheral circuit 5A Bus

Claims

1. a skeletal structure detection means for detecting key points of a human body included in an image; a similarity calculation means for calculating a similarity between a posture or movement of the human body detected from the image and a posture or movement of the human body shown in a pre-registered template image based on the detected key points; an identification means for identifying a location in the image where a human body is captured that does not satisfy a first similarity condition with respect to a posture or movement of the human body shown in any of the template images, among the human bodies whose similarity to the posture or movement of the human body shown in any of the template images is less than a first threshold value; an output means for outputting information about the identified portion as a candidate for the template image to be additionally registered; and The first similarity condition is: the similarity is equal to or greater than a second threshold and less than the first threshold; the similarity between the posture or movement of the human body shown in the template image calculated based on some of the plurality of key points detected from each human body is equal to or greater than a third threshold; The similarity between the posture or movement of the human body shown in the template image calculated by taking into consideration weighting values ​​assigned to each of the plurality of key points detected from each human body is equal to or greater than a fourth threshold value; and the template image includes a plurality of frame images each showing a human body in a posture in which the similarity to the posture of the human body shown in each of the frame images of a predetermined proportion or more of the plurality of frame images included in the template image is equal to or greater than a fifth threshold; An image processing device including at least one of the above.

2. The image processing device described in Claim 1, wherein the output means outputs information regarding the identified location as a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body indicated by the template image.

3. An image processing device as described in claim 1 or 2, wherein the output means outputs, as information regarding the identified location, information indicating the identified location or a partial image obtained by cutting out the identified location from the image.

4. The method further comprises a grouping means for grouping a plurality of human bodies appearing in the specified location based on similarity in posture or movement, The output means further outputs the results of the grouping. The image processing device according to claim 1 .

5. The computer The process detects key points of the human body contained in the image, Calculating a similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in a pre-registered template image based on the detected key points; identifying a location in the image in which a human body is captured that does not satisfy a first similarity condition with respect to a posture or movement of the human body shown in any of the template images, among the human bodies whose similarity to the posture or movement of the human body shown in any of the template images is less than a first threshold value; outputting information about the identified portion as a candidate for the template image to be additionally registered; The first similarity condition is: the similarity is equal to or greater than a second threshold and less than the first threshold; the similarity between the posture or movement of the human body shown in the template image calculated based on some of the plurality of key points detected from each human body is equal to or greater than a third threshold; The similarity between the posture or movement of the human body shown in the template image calculated by taking into consideration weighting values ​​assigned to each of the plurality of key points detected from each human body is equal to or greater than a fourth threshold value; and the template image includes a plurality of frame images each showing a human body in a posture in which the similarity to the posture of the human body shown in each of the frame images of a predetermined proportion or more of the plurality of frame images included in the template image is equal to or greater than a fifth threshold; An image processing method including at least one of the above.

6. Computer, a skeletal structure detection means for detecting key points of a human body included in an image; a similarity calculation means for calculating a similarity between a posture or movement of the human body detected from the image and a posture or movement of the human body shown in a pre-registered template image based on the detected key points; a specifying means for specifying a location in the image in which a human body is captured that does not satisfy a first similarity condition with respect to a posture or movement of the human body shown in any of the template images, among the human bodies whose similarity to the posture or movement of the human body shown in any of the template images is less than a first threshold value; an output means for outputting information about the identified portion as a candidate for the template image to be additionally registered; It functions as The first similarity condition is: the similarity is equal to or greater than a second threshold and less than the first threshold; the similarity between the posture or movement of the human body shown in the template image calculated based on some of the plurality of key points detected from each human body is equal to or greater than a third threshold; The similarity between the posture or movement of the human body shown in the template image calculated by taking into consideration weighting values ​​assigned to each of the plurality of key points detected from each human body is equal to or greater than a fourth threshold value; and the template image includes a plurality of frame images each showing a human body in a posture in which the similarity to the posture of the human body shown in each of the frame images of a predetermined proportion or more of the plurality of frame images included in the template image is equal to or greater than a fifth threshold; A program that includes at least one of the following:

Images