Image processing apparatus, image processing method, and program

Abstract

The present invention provides an image processing device (10) comprising: a skeleton structure detection unit (11) that executes processing of detecting key points of a human body included in an image; a similarity degree calculation unit (12) that uses the detected key points to calculate the degree of similarity between the posture or motion of the human body detected from the image and the posture or motion of a human body indicated by each template image registered in advance; a specification unit (13) that specifies a portion of the image with the human body for which the degree of similarity to the posture or motion of a human body indicated by any template image is smaller than a first threshold value but a first similarity condition is satisfied for the posture or motion of a human body indicated by a template image; and an output unit (14) that outputs information indicating the specified portion or a partial image obtained by extracting the portion from the image as a candidate for a template image to be additionally registered into a determination device for determining the posture or motion of a human body detected from an image on the basis of the posture or motion of a human body indicated by each template image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Technologies related to the present invention are disclosed in Patent Documents 1 to 3 and Non-Patent Document 1.

[0003] Patent Document 1 discloses a technology for calculating feature amounts of a plurality of key points of a human body included in an image, and searching for an image including a human body with a similar posture or a similar movement based on the calculated feature amounts, or classifying together those with similar postures or movements. Further, Non-Patent Document 1 discloses a technology related to human skeleton estimation.

[0004] Patent Document 2 discloses a technology for acquiring a plurality of images obtained by imaging a predetermined area and information indicating a change in the situation of the predetermined area, classifying the plurality of images based on the information indicating the change in the situation of the predetermined area, and performing learning of an identifier for determining the situation of the predetermined area from the image using at least a part of the plurality of images according to the classification result.

[0005] Patent Document 3 discloses a technology for detecting a change in the state of a target in a person based on an input image, and determining an abnormal state in response to detection of the occurrence of a change in the state of the target in a plurality of people.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Non-Patent Documents

[0007] [Non-Patent Document 1] Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, P. 7291-7299 [Summary of the Invention] [Problems to be Solved by the Invention]

[0008] According to the technology disclosed in Patent Document 1 described above, by registering in advance an image including a human body with a desired pose or desired movement as a template image, it is possible to detect a human body with a desired pose or desired movement from the images to be processed. As a result of examining the technology disclosed in such Patent Document 1, the present inventor newly found that by newly registering as a template image an image including a human body with a posture or movement that is not determined to be the same or of the same type as the posture or movement shown in the registered template image but is similar, it becomes possible to detect a human body with a desired posture or desired movement without omission. And the present inventor newly found that there is room for improvement in the workability of the operation of searching for an image including a human body with a posture or movement that is not determined to be the same or of the same type as the posture or movement shown in such a registered template image but is similar.

[0009] Since each of Patent Documents 1 to 3 and Non-Patent Document 1 described above does not disclose problems related to template images and means for solving them, there was a problem that the above problems could not be solved.

[0010] An example of an object of the present invention is to provide an image processing apparatus, an image processing method, and a program that solve the problem of workability of the work of registering, as a template image, an image including a human body in a posture or movement that is not determined to be the same or of the same type as the posture or movement shown by a registered template image but is similar, in view of the above-described problems.

Means for Solving the Problem

[0011] According to one aspect of the present invention, skeleton structure detection means for performing a process of detecting key points of a human body included in an image; similarity calculation means for calculating a similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown by a template image registered in advance, based on the detected key points; specific means for specifying a location in the image where a human body that satisfies a first similarity condition with the posture or movement of the human body shown by any of the template images appears, although the similarity with the posture or movement of the human body shown by any of the template images is less than a first threshold value; output means for outputting information indicating the specified location or a partial image obtained by cutting out the location from the image, as a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body shown by the template image; An image processing apparatus having the above is provided.

[0012] Also, according to one aspect of the present invention, a computer performs a process of detecting key points of a human body included in an image, calculates a similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown by a template image registered in advance, based on the detected key points, Although the similarity to the posture or movement of the human body shown in any of the template images is less than the first threshold value, a location within the image in which a human body satisfying the first similarity condition appears with respect to the posture or movement of the human body shown in any of the template images is specified. As a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of a human body detected from the image based on the posture or movement of the human body shown in the template image, information indicating the specified location or a partial image obtained by cutting out the location from the image is output. An image processing method is provided.

[0013] Also, according to one aspect of the present invention, a computer is caused to function as skeleton structure detection means for performing a process of detecting key points of a human body included in an image, similarity calculation means for calculating a similarity between the posture or movement of a human body detected from the image and the posture or movement of a human body shown in a template image registered in advance based on the detected key points, specification means for specifying a location within the image in which a human body satisfying the first similarity condition appears with respect to the posture or movement of the human body shown in any of the template images, although the similarity to the posture or movement of the human body shown in any of the template images is less than the first threshold value, output means for outputting, as a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of a human body detected from the image based on the posture or movement of the human body shown in the template image, information indicating the specified location or a partial image obtained by cutting out the location from the image. A program is provided to cause the computer to function as such.

Advantages of the Invention

[0014] According to one aspect of the present invention, there are provided an image processing apparatus, an image processing method, and a program that solve the problem of workability in the operation of registering, as a template image, an image including a human body in a posture or movement that is not determined to be the same as or of the same type as the posture or movement shown in a registered template image but is similar.

Brief Description of the Drawings

[0015] The above-mentioned object, as well as other objects, features and advantages, will become further apparent from the following public embodiments and the accompanying drawings below.

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, similar components are denoted by similar reference numerals, and the description thereof will be omitted as appropriate.

[0018] <First Embodiment> FIG. 1 is a functional block diagram showing an overview of an image processing apparatus 10 according to the first embodiment. As shown in FIG. 1, the image processing apparatus 10 includes a skeleton structure detection unit 11, a similarity calculation unit 12, a specification unit 13, and an output unit 14.

[0019] The skeleton structure detection unit 11 performs a process of detecting key points of a human body included in an image. The similarity calculation unit 12 calculates a similarity between the posture or movement of the human body detected from the image based on the detected key points and the posture or movement of the human body shown in a template image registered in advance. The specification unit 13 specifies a location in an image in which a human body that satisfies a first similarity condition with the posture or movement shown in any one of the template images is captured, although the similarity with the posture or movement shown in any of the template images is less than a first threshold value. The output unit 14 outputs information indicating the specified location or a partial image obtained by cutting out the location from the image as a candidate for a template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body shown in the template image.

[0020] According to this image processing apparatus 10, it is possible to solve the problem of workability in the work of registering, as a template image, an image including a human body having a posture or movement similar to, but not the same as, the posture or movement shown in a registered template image.

[0021] <Second Embodiment> "Overview" The image processing apparatus 10 calculates the similarity between the posture or movement of a human body included in an image (hereinafter simply referred to as "image") that is the source of a template image and the posture or movement of a human body shown in a previously registered template image. After that, although the similarity with the posture or movement of a human body shown in any of the template images is less than a first threshold value, it identifies a location in an image in which a human body that satisfies a first similarity condition with the posture or movement of a human body shown in any of the template images appears. Then, the image processing apparatus 10 outputs information indicating the identified location or a partial image obtained by cutting out the identified location from the image as a candidate for a template image to be additionally registered for the determination apparatus. Incidentally, the determination apparatus performs detection processing or the like using the registered template images, and when the similarity is equal to or greater than the first threshold value, determines that the posture or movement of the human body detected from the image and the posture or movement of the human body shown in the template image are the same or of the same type.

[0022] According to such an image processing apparatus 10, among the set of human bodies detected from the image, it is not determined that the posture or movement of the human body shown in any of the template images is the same or of the same type, but it can identify a location in the image in which a similar human body appears and output information regarding the identified location. This will be described in more detail with reference to FIG. 2.

[0023] In the second embodiment, as shown in FIG. 2, the set of human bodies detected from the image is classified into (1) a set of human bodies determined to have the same or the same type of posture or movement as the posture or movement of the human body shown in any of the template images, (2) a set of human bodies that are not determined to have the same or the same type of posture or movement as the posture or movement of the human body shown in any of the template images but have similar postures or movements, and (3) a set of other human bodies. The set of (3) other human bodies is a set of human bodies that are not determined to have the same or the same type of posture or movement as the posture or movement of the human body shown in any of the template images and do not have similar postures or movements. In the present embodiment, it identifies a location in the image in which a human body included in the set of human bodies that are not determined to have the same or the same type of posture or movement as the posture or movement of the human body shown in any of the template images but have similar postures or movements appears, and outputs information regarding the identified location. This will be described in detail below.

[0024] "Hardware Configuration" Next, an example of the hardware configuration of the image processing apparatus 10 will be described. Each functional unit of the image processing apparatus 10 is realized by an arbitrary combination of hardware and software centered around a CPU (Central Processing Unit), a memory, a program loaded into the memory, a storage unit such as a hard disk that stores the program (in addition to a program stored in advance at the stage of shipping the apparatus, a program downloaded from a storage medium such as a CD (Compact Disc) or a server on the Internet can also be stored), and a network connection interface. And it is understood by those skilled in the art that there are various modifications to the realization method and apparatus.

[0025] FIG. 3 is a block diagram illustrating the hardware configuration of the image processing apparatus 10. As shown in FIG. 3, the image processing apparatus 10 includes a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The image processing apparatus 10 may not have the peripheral circuit 4A. Note that the image processing apparatus 10 may be configured by a plurality of physically and / or logically divided apparatuses. In this case, each of the plurality of apparatuses can include the above-described hardware configuration.

[0026] Bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to transmit and receive data with each other. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and an interface for outputting information to an output device, an external device, an external server, etc. The input device is, for example, a keyboard, a mouse, a microphone, a physical button, a touch panel, etc. The output device is, for example, a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform operations based on their operation results.

[0027] "Functional Configuration" FIG. 1 is a functional block diagram showing an overview of the image processing apparatus 10 according to the second embodiment. As shown in FIG. 1, the image processing apparatus 10 includes a skeleton structure detection unit 11, a similarity calculation unit 12, a specification unit 13, and an output unit 14.

[0028] The skeleton structure detection unit 11 performs a process of detecting key points of a human body included in an image.

[0029] The "image" is an image that is the source of the template image. The template image is an image that is pre-registered in the technology disclosed in Patent Document 1 described above and includes a human body in a desired posture or a desired movement (a posture or movement that the user wants to detect). The image may be a moving image composed of a plurality of frame images or a still image composed of one image.

[0030] The skeletal structure detection unit 11 detects N (N is an integer of 2 or more) key points of the human body included in the image. When a moving image is the processing target, the skeletal structure detection unit 11 performs a process of detecting key points for each frame image. The process by the skeletal structure detection unit 11 is realized using the technology disclosed in Patent Document 1. Although details are omitted, in the technology disclosed in Patent Document 1, the detection of the skeletal structure is performed using a skeletal estimation technology such as OpenPose disclosed in Non-Patent Document 1. The skeletal structure detected by the technology is composed of "key points" which are characteristic points such as joints, and "bones (bone links)" indicating links between the key points.

[0031] FIG. 4 shows the skeletal structure of the human body model 300 detected by the skeletal structure detection unit 11, and FIGS. 5 to 7 show detection examples of the skeletal structure. The skeletal structure detection unit 11 uses a skeletal estimation technology such as OpenPose to detect the skeletal structure of the human body model (2D skeletal model) 300 as shown in FIG. 4 from a 2D image. The human body model 300 is a 2D model composed of key points such as joints of a person and bones connecting each key point.

[0032] The skeletal structure detection unit 11, for example, extracts feature points that can be key points from the image, and refers to the information obtained by machine learning of the key point images to detect N key points of the human body. The N key points to be detected are predetermined. The number of key points to be detected (that is, the number of N) and which part of the human body is to be the key point to be detected vary, and any variation can be adopted.

[0033] Hereinafter, as shown in FIG. 4, it is assumed that the head A1, neck A2, right shoulder A31, left shoulder A32, right elbow A41, left elbow A42, right hand A51, left hand A52, right hip A61, left hip A62, right knee A71, left knee A72, right foot A81, and left foot A82 are defined as N key points (N = 14) to be detected. In the human body model 300 shown in FIG. 3, as the bones of a person connecting these key points, bone B1 connecting the head A1 and the neck A2, bone B21 and bone B22 respectively connecting the neck A2 with the right shoulder A31 and the left shoulder A32, bone B31 and bone B32 respectively connecting the right shoulder A31 and the left shoulder A32 with the right elbow A41 and the left elbow A42, bone B41 and bone B42 respectively connecting the right elbow A41 and the left elbow A42 with the right hand A51 and the left hand A52, bone B51 and bone B52 respectively connecting the neck A2 with the right hip A61 and the left hip A62, bone B61 and bone B62 respectively connecting the right hip A61 and the left hip A62 with the right knee A71 and the left knee A72, and bone B71 and bone B72 respectively connecting the right knee A71 and the left knee A72 with the right foot A81 and the left foot A82 are further defined.

[0034] FIG. 5 is an example of detecting a person in an upright state. In FIG. 5, an upright person is imaged from the front, and bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 seen from the front are detected without overlapping, and the bones B61 and B71 of the right foot are slightly bent more than the bones B62 and B72 of the left foot.

[0035] FIG. 6 is an example of detecting a person in a crouched state. In FIG. 6, a crouched person is imaged from the right side, and bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 seen from the right side are detected respectively, and the bones B61 and B71 of the right foot and the bones B62 and B72 of the left foot are greatly bent and overlapping.

[0036] FIG. 7 is an example of detecting a person in a lying state. In FIG. 7, a person lying down is imaged from the front left diagonal. Bones B1, B51 and B52, B61 and B62, B71 and B72 are detected respectively from the front left diagonal. The bones B61 and B71 of the right foot and the bones B62 and B72 of the left foot are bent and overlapped.

[0037] Returning to FIG. 1, the similarity calculation unit 12 calculates the similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in the pre-registered template image based on the key points detected by the skeleton structure detection unit 11.

[0038] There are various ways to calculate the similarity of the above-mentioned human body posture or movement, and any technology can be adopted. For example, the technology disclosed in Patent Document 1 may be adopted. Also, the same method as that of a determination device that calculates the similarity between the posture or movement of the human body shown in the template image and the posture or movement of the human body detected from the image, and detects a human body with a similarity equal to or higher than the first threshold as a human body with the same posture or movement as the human body shown in the template image, or the same type of posture or movement may be adopted. Hereinafter, an example will be described, but it is not limited thereto.

[0039] As an example, the similarity calculation unit 12 may calculate the similarity of the postures of two human bodies by calculating the feature amount of the skeleton structure indicated by the detected key points and calculating the similarity between the feature amount of the skeleton structure of the human body detected from the image and the feature amount of the skeleton structure of the human body shown in the template image.

[0040] The feature amount of the skeletal structure indicates the features of a person's skeleton and serves as an element for classifying the state (posture and movement) of a person based on the person's skeleton. Usually, this feature amount includes a plurality of parameters. The feature amount may be the overall feature amount of the skeletal structure, the feature amount of a part of the skeletal structure, or may include a plurality of feature amounts such as those of each part of the skeletal structure. The method for calculating the feature amount may be any method such as machine learning or normalization, and the minimum value or maximum value may be obtained as normalization. As an example, the feature amount is the feature amount obtained by machine learning the skeletal structure, the size on the image from the head to the feet of the skeletal structure, the relative positional relationship of a plurality of key points in the vertical direction of the skeletal region including the skeletal structure on the image, the relative positional relationship of a plurality of key points in the horizontal direction of the skeletal region, and the like. The size of the skeletal structure is the height, area, etc. in the vertical direction of the skeletal region including the skeletal structure on the image. The vertical direction (height direction or longitudinal direction) is the vertical direction (Y-axis direction) in the image, for example, the direction perpendicular to the ground (reference plane). Also, the horizontal direction (lateral direction) is the left-right direction (X-axis direction) in the image, for example, the direction parallel to the ground.

[0041] Note that in order to perform the classification desired by the user, it is preferable to use a feature amount that is robust to the determination process. For example, when the user desires a determination that is independent of the orientation and body type of a person, a feature amount that is robust to the orientation and body type of a person may be used. By learning the skeletons of people facing in various directions in the same posture or the skeletons of people with various body types in the same posture, or by extracting only the features in the vertical direction of the skeleton, a feature amount that is independent of the orientation and body type of a person can be obtained. An example of the process for calculating the feature amount of the skeletal structure is disclosed in Patent Document 1.

[0042] FIG. 8 shows an example of the feature amount of each of the plurality of key points obtained by the similarity calculation unit 12. The set of the feature amounts of the plurality of key points becomes the feature amount of the skeletal structure. Note that the feature amount of the key point illustrated here is merely an example and is not limited thereto.

[0043] In this example, the feature amount of the key point indicates the relative positional relationship of a plurality of key points in the vertical direction of the skeleton region including the skeleton structure on the image. Since the key point A2 of the neck is used as the reference point, the feature amount of the key point A2 is 0.0, and the feature amounts of the key point A31 of the right shoulder and the key point A32 of the left shoulder at the same height as the neck are also 0.0. The feature amount of the key point A1 of the head higher than the neck is -0.2. The feature amounts of the key point A51 of the right hand and the key point A52 of the left hand lower than the neck are 0.4, and the feature amounts of the key point A81 of the right foot and the key point A82 of the left foot are 0.9. When the person raises the left hand from this state, since the left hand becomes higher than the reference point as shown in FIG. 9, the feature amount of the key point A52 of the left hand becomes -0.4. On the other hand, since normalization is performed using only the coordinates of the Y axis, as shown in FIG. 10, the feature amount does not change even if the width of the skeleton structure changes compared to FIG. 8. That is, the feature amount (normalized value) of this example indicates the feature in the height direction (Y direction) of the skeleton structure (key point) and is not affected by the change in the horizontal direction (X direction) of the skeleton structure.

[0044] There are various ways to calculate the similarity of postures indicated by such feature amounts. For example, after calculating the similarity of feature amounts for each key point, the similarity of the posture may be calculated based on the similarity of the feature amounts of a plurality of key points. For example, the average value, maximum value, minimum value, mode value, median value, weighted average value, weighted sum, etc. of the similarity of the feature amounts of a plurality of key points may be calculated as the similarity of the posture. When calculating the weighted average value or weighted sum, the weight of each key point may be set by the user or may be predetermined.

[0045] Also, movement is represented as a time change of a plurality of postures. Therefore, the similarity calculation unit 12, for example, calculates the similarity of the posture by the above method for each combination of a plurality of corresponding frame images, and then calculates the statistical value (average value, maximum value, minimum value, mode value, median value, weighted average value, weighted sum, etc.) of the similarity of the posture calculated for each combination of a plurality of frame images as the similarity of the movement.

[0046] Returning to FIG. 1, the specific part 13 identifies, as candidates for template images to be additionally registered for the determination device, locations within an image in which a human body appears that satisfies the first similarity condition with the posture or movement of the human body shown in any of the template images, although the similarity with the posture or movement of the human body shown in any of the template images is less than the first threshold value.

[0047] First, a process of identifying a human body (a human body belonging to the set of (2) and (3) in FIG. 2) whose similarity with the posture or movement of the human body shown in any of the template images is less than the first threshold value will be described.

[0048] The specific part 13 compares the similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in each of the plurality of template images with the first threshold value. Then, based on the result of the comparison, the specific part 13 identifies a human body whose similarity with the posture or movement of the human body shown in any of the template images is less than the first threshold value.

[0049] Note that the determination device determines the posture or movement of the human body detected from the image based on the posture or movement of the human body shown in the template image. Specifically, when the similarity is greater than or equal to the first threshold value, the determination device determines that the posture or movement of the human body detected from the image is the same as or of the same type as the posture or movement of the human body shown in the template image. That is, by the above process performed by the specific part 13, locations within the image in which a human body appears that is not determined by the determination device to be the same as or of the same type as the posture or movement of the human body shown in any of the template images among the set of human bodies detected from the image are identified.

[0050] Next, a process of identifying a location within an image in which a human body (a human body belonging to the set of (2) in FIG. 2) that satisfies the first similarity condition with the posture or movement of the human body shown in any of the template images appears will be described.

[0051] After identifying a human body belonging to the set of (2) and (3) in Fig. 2 from the human bodies detected in the image, the specific part 13 determines whether, for each identified human body, the posture or movement of the human body shown in any of the template images satisfies the first similarity condition. Then, based on the result of the determination, the specific part 13 identifies a human body (a human body belonging to the set of (2) in Fig. 2) whose posture or movement of the human body shown in any of the template images satisfies the first similarity condition, and also identifies a location within the image where the identified human body appears. Incidentally, a human body that does not satisfy the first similarity condition belongs to the set of (3) in Fig. 2.

[0052] The first similarity condition is · "The similarity degree with the posture or movement of the human body shown in the template image is equal to or greater than the second threshold value and less than the first threshold value", · "The similarity degree with the posture or movement of the human body shown in the template image calculated based on some of the plurality of key points (N key points) detected from each human body is equal to or greater than the third threshold value", · "The similarity degree with the posture or movement of the human body shown in the template image calculated in consideration of the weighting values assigned to each of the plurality of key points detected from each human body is equal to or greater than the fourth threshold value", and · "Including a plurality of frame images each showing a human body posture whose similarity degree with the posture of the human body shown in each of a predetermined ratio or more of the plurality of frame images included in the template image (which is a moving image) is equal to or greater than the fifth threshold value", includes at least one of the above.

[0053] When including a plurality of the above-exemplified conditions, the first similarity condition can be the content obtained by connecting the plurality of conditions with a logical operator such as "or". Hereinafter, each of the above-exemplified conditions will be described.

[0054] "The similarity degree with the posture or movement of the human body shown in the template image is equal to or greater than the second threshold value and less than the first threshold value" The "similarity" under this condition is a value calculated by the same method as the calculation method by the similarity calculation unit 12 described above. And the second threshold value is a value smaller than the first threshold value.

[0055] By appropriately setting the second threshold value, it is not determined that the body posture or movement shown by any template image is the same or of the same type, but a human body with a similar posture or movement (a human body belonging to the set of (2) in FIG. 2) can be detected.

[0056] "The similarity between the body posture or movement shown by the template image calculated based on some of the plurality of key points (N key points) detected from each human body is equal to or greater than the third threshold value" The "similarity" under this condition is a value calculated based on some of the plurality of key points (N key points) of the detection target. Except for using only the feature amounts of some of the plurality of key points (N key points), the same method as the calculation method by the similarity calculation unit 12 described above can be adopted to calculate the similarity under this condition.

[0057] Which key points to use is a matter of design, but for example, it may be specified by the user. The user can specify the key points of the body part to be emphasized (e.g., the upper body) and exclude the key points of the body part not to be emphasized (e.g., the lower body) from the specification.

[0058] By appropriately setting the third threshold value, it is not determined that the body posture or movement shown by any template image is the same or of the same type, but a human body with a similar posture or movement in which a part of the body is the same (a human body belonging to the set of (2) in FIG. 2) can be detected.

[0059] "The similarity between the body posture or movement shown by the template image calculated in consideration of the weighting values assigned to each of the plurality of key points detected from each human body is equal to or greater than the fourth threshold value" The "similarity" under this condition is a value calculated by assigning weights to a plurality of keypoints (N keypoints) to be detected. For example, after calculating the similarity of feature amounts for each keypoint by adopting the same method as the calculation method by the similarity calculation unit 12 described above, using the above weight values, the weighted average value or weighted sum of the similarity of feature amounts of the plurality of keypoints is calculated as the similarity of the postures. The weight of each keypoint may be set by the user or may be predetermined.

[0060] By appropriately setting the fourth threshold value, it is not determined that the body postures or movements shown by any of the template images are the same or of the same type, but when weight is placed on a part of the body, a human body with the same or similar posture or movement (a human body belonging to the set of (2) in FIG. 2) can be detected.

[0061] "Including a plurality of frame images each showing a human body posture whose similarity to the human body postures shown by each of a predetermined ratio or more of the frame images included in the template image which is a moving image is equal to or higher than a fifth threshold value" This condition is used when the image and the template image are moving images, and the movement of the human body is shown by the temporal change of the human body postures shown by each of the plurality of template images included in the moving image.

[0062] For example, although the template image is composed of M frame images, a plurality of frame images each including a human body posture that is similar to a predetermined level or more (similarity is equal to or higher than the fifth threshold value) to the human body postures shown by each of a predetermined ratio or more (e.g., 70% or more) of the M frame images satisfy this condition. As a method of calculating the similarity of postures for each combination of a plurality of corresponding frame images, the same method as the calculation method by the similarity calculation unit 12 described above can be adopted.

[0063] By appropriately setting the fifth threshold value and the predetermined ratio, it is determined that the movement of the human body shown in any template image is not the same or of the same type. However, it is possible to detect a human body with a movement that is the same as or similar to the movement of the human body in a partial time period of the template image (moving image) (a human body belonging to the set of (2) in FIG. 2).

[0064] In the case where the image is a still image, the "location specified by the specifying unit 13" is a partial area within one still image. In this case, for each still image, the above location is indicated by coordinates in the coordinate system set for the still image, for example. On the other hand, in the case where the image is a moving image, the "location specified by the specifying unit 13" is a partial area within each of some frame images among the plurality of frame images constituting the moving image. In this case, for each moving image, the above location is indicated by information indicating some frame images among the plurality of frame images (frame identification information, elapsed time from the start, etc.) and coordinates in the coordinate system set for the frame image.

[0065] The output unit 14 outputs, as a candidate for a template image to be additionally registered in the determination device, information indicating the location specified by the specifying unit 13, or a partial image obtained by cutting out the location specified by the specifying unit 13 from the image. When the output unit 14 outputs a partial image, the image processing apparatus 10 can have a processing unit that cuts out the location specified by the specifying unit 13 from the image to generate a partial image. Then, the output unit 14 can output the partial image generated by the processing unit.

[0066] The above-mentioned "location specified by the specifying unit 13", that is, the similarity with the posture or movement of the human body shown in any template image is less than the first threshold value. However, a location within an image in which a human body that satisfies the first similarity condition with the posture or movement of the human body shown in any template image appears becomes a candidate for the template image. The user can view the above information or the above partial image, etc., and select, as a template image, a location including a human body with a desired posture or a desired movement from among them.

[0067] Fig. 11 schematically shows an example of the information output by the output unit 14. In the example shown in Fig. 11, the human body identification information for identifying a plurality of detected human bodies from each other, the attribute information of each human body, and the similar sample images are displayed in association with each other. And, as an example of the attribute information, information indicating a location within the image (information indicating the location where the above-described human body appears), and the shooting date and time of the image are displayed. The attribute information may further include information indicating the installation location (shooting position) of the camera that captured the image (e.g., the rear inside of the No. 102 bus, the entrance of XX Park, etc.), and the attribute information of the person calculated by image analysis (e.g., gender, age group, body type, etc.).

[0068] In the column of the similar sample image, information (such as the file name of the image) indicating the template image that satisfies the first similarity condition with each human body is entered. In this way, the output unit 14 can further output information indicating the template image that satisfies the first similarity condition with the human body appearing in the location specified by the specifying unit 13.

[0069] Next, an example of the processing flow of the image processing apparatus 10 will be described using the flowchart of Fig. 12.

[0070] When the image processing apparatus 10 performs a process of detecting the key points of the human body included in the image (S10), based on the detected key points, it calculates the similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in the template image registered in advance (S11).

[0071] Next, the image processing apparatus 10 specifies, as a candidate for the template image to be additionally registered for the determination device, a location within the image where a human body that satisfies the first similarity condition with the posture or movement of the human body shown in any of the template images, but the similarity with the posture or movement of the human body shown in any of the template images is less than the first threshold value (S12).

[0072] Specifically, the image processing apparatus 10 compares the similarity between the posture or movement of a human body detected from an image and the posture or movement of a human body shown in each of a plurality of template images with a first threshold value. Then, based on the result of the comparison, the image processing apparatus 10 identifies a human body (a human body belonging to the set of (2) and (3) in FIG. 2) whose similarity with the posture or movement of a human body shown in any of the template images is less than the first threshold value. Thereafter, for each identified human body, the image processing apparatus 10 determines whether the posture or movement of a human body shown in any of the template images satisfies a first similarity condition. Then, based on the result of the determination, the image processing apparatus 10 identifies a human body (a human body belonging to the set of (2) in FIG. 2) whose posture or movement of a human body shown in any of the template images satisfies the first similarity condition, and also identifies a location within the image in which the identified human body appears.

[0073] Incidentally, the determination apparatus performs detection processing and the like using the registered template images. When the similarity is greater than or equal to the first threshold value, it is determined that the posture or movement of the human body detected from the image is the same as or of the same type as the posture or movement of the human body shown in the template image.

[0074] Then, the image processing apparatus 10 outputs information indicating the location specified in S12, or a partial image obtained by cutting out the location specified in S12 from the image (S13).

[0075] "Function and Effect" According to the image processing apparatus 10 of the second embodiment, the same function and effect as those of the first embodiment are achieved. Further, according to the image processing apparatus 10 of the second embodiment, information regarding a location within an image in which a human body that is not determined by the determination apparatus to have the same or the same type of posture or movement as the posture or movement of a human body shown in any of the template images but is similar appears can be output.

[0076] A more detailed description will be given with reference to FIG. 2. In the second embodiment, as shown in FIG. 2, the set of human bodies detected from the image is classified into: (1) a set of human bodies determined by the determination device to have the same or the same type of posture or movement of the human body shown in any of the template images; (2) a set of human bodies with similar postures or movements that are not determined to have the same or the same type of posture or movement as that shown in any of the template images; and (3) a set of other human bodies. The set of (3) other human bodies is a set of human bodies that are not determined to have the same or the same type of posture or movement as that shown in any of the template images and have dissimilar postures or movements. According to the image processing apparatus 10 of the second embodiment, a location in the image in which a human body included in the set of human bodies with similar postures or movements that are not determined to have the same or the same type of posture or movement as that shown in any of the template images appears is specified, and information regarding the specified location is output. The user can browse the specified location and select, as a template image, a location including a human body with a desired posture or movement from among them. As a result, the problem of workability in registering, as a template image, an image including a human body with a posture or movement that is not determined to be the same or of the same type as that shown in the registered template image but is similar is solved.

[0077] As described above, embodiments of the present invention have been described with reference to the drawings, but these are examples of the present invention, and various configurations other than the above can also be adopted.

[0078] Also, in the plurality of flowcharts used in the above description, a plurality of steps (processes) are described in order, but the execution order of the steps executed in each embodiment is not limited to the described order. In each embodiment, the order of the illustrated steps can be changed within a range that does not interfere with the content. Also, the above-described embodiments can be combined within a range where the contents do not conflict.

[0079] Some or all of the above embodiments can also be described as follows in the appended claims, but are not limited thereto. 1. A skeletal structure detection means for performing a process of detecting key points of a human body included in an image, a similarity calculation means for calculating a similarity between the posture or movement of the human body detected from the image and the posture or movement of the human body shown in a template image registered in advance based on the detected key points, a specifying means for specifying a location in the image where a human body that satisfies a first similarity condition with the posture or movement of the human body shown in any of the template images appears, although the similarity with the posture or movement of the human body shown in any of the template images is less than a first threshold value, an output means for outputting information indicating the specified location or a partial image obtained by cutting out the location from the image as a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body shown in the template image, and an image processing apparatus having the above. 2. The image processing apparatus according to claim 1, wherein the specifying means determines whether the human body detected from the image satisfies the first similarity condition based on the detected key points. 3. The image processing apparatus according to claim 2, wherein the first similarity condition includes that the similarity is equal to or greater than a second threshold value and less than the first threshold value. 4. The image processing apparatus according to claim 2 or 3, wherein the first similarity condition includes that the similarity with the posture or movement of the human body shown in the template image calculated based on some of the plurality of key points detected from each human body is equal to or greater than a third threshold value. 5. The image processing apparatus according to any one of claims 2 to 4, wherein the first similarity condition includes that the similarity with the posture or movement of the human body shown in the template image calculated in consideration of the weighting values assigned to each of the plurality of key points detected from each human body is equal to or greater than a fourth threshold value. 6. The image and the template image are moving images, and the movement of the human body is shown by the temporal change of the posture of the human body shown in each of the plurality of template images included in the moving image, The first similarity condition is that each of a plurality of frame images included in the template image and having a similarity degree of a predetermined ratio or more with the posture of the human body shown by each of the frame images includes a plurality of frame images showing each human body having a posture with a similarity degree of the fifth threshold value or more. The image processing apparatus according to any one of 2 to 5. 7. The output means further outputs information indicating the human body appearing in the specified location and the template image satisfying the first similarity condition. The image processing apparatus according to any one of 1 to 6. 8. A computer performs a process of detecting key points of a human body included in an image, calculates a similarity degree between the posture or movement of the human body detected from the image and the posture or movement of the human body shown by a template image registered in advance based on the detected key points, although the similarity degree with the posture or movement of the human body shown by any of the template images is less than the first threshold value, a location in the image where a human body satisfying the first similarity condition with the posture or movement of the human body shown by any of the template images appears is specified, As a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of the human body detected from the image based on the posture or movement of the human body shown by the template image, information indicating the specified location or a partial image obtained by cutting out the location from the image is output. Image processing method. 9. A computer is a skeleton structure detection means for performing a process of detecting key points of a human body included in an image, is a similarity calculation means for calculating a similarity degree between the posture or movement of the human body detected from the image and the posture or movement of the human body shown by a template image registered in advance based on the detected key points, is a specifying means for specifying a location in the image where a human body satisfying the first similarity condition with the posture or movement of the human body shown by any of the template images appears although the similarity degree with the posture or movement of the human body shown by any of the template images is less than the first threshold value, As a candidate for the template image to be additionally registered in a determination device that determines the posture or movement of a human body detected from the image based on the posture or movement of the human body indicated by the template image, information indicating the specified location, or an output means that outputs a partial image obtained by cutting out the location from the image, A program that functions as.

Explanation of Signs

[0080] 10 Image processing device 11 Skeleton structure detection unit 12 Similarity calculation unit 13 Specifying unit 14 Output unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. A skeletal structure detection means for performing a process of detecting key points of a human body included in an image; A similarity calculation means for calculating, as a first similarity, a similarity between a feature amount of the key points detected from a first image and a feature amount of the key points detected from a template image registered in advance, between a posture or movement of the human body detected from the first image and a posture or movement of the human body shown in the template image; A specifying means for specifying a location in the first image in which a human body that satisfies a first similarity condition with a posture or movement of the human body shown in any of the template images is captured, although the first similarity with the posture or movement of the human body shown in any of the template images is less than a first threshold; An output means for outputting, to an output device, information indicating the location in the first image or a partial image obtained by cutting out the location from the first image, as a candidate for the template image to be additionally registered in a determination device that detects a human body in which a second similarity between a feature amount of the key points of the human body detected from a second image and a feature amount of the key points of the human body detected from the template image is equal to or greater than the first threshold; comprising: The first similarity condition is: the first similarity is equal to or greater than a second threshold and less than the first threshold; a third similarity, which is a similarity between a feature amount of a first key point detected from the first image and a feature amount of the first key point detected from the template image, is equal to or greater than a third threshold; a fourth similarity calculated from the feature amounts of each of the plurality of key points detected from the first image, the feature amounts of each of the plurality of key points detected from the template image, and weight values of each of the plurality of predetermined key points is equal to or greater than a fourth threshold; and a moving image including a plurality of frame images from each of which a feature amount of a key point, for which a fifth similarity, which is a similarity with the feature amounts of the plurality of key points extracted from a predetermined ratio or more of the plurality of frame images included in the template image which is a moving image, is equal to or greater than a fifth threshold, is extracted; wherein at least one of the above is satisfied; the second threshold is smaller than the first threshold; the first key point is a predetermined part of the plurality of key points detected by the skeletal structure detection means; The fourth similarity is a weighted average value or weighted sum of the similarities between the feature amounts of the keypoints detected from the first image calculated for each of the keypoints and the feature amounts of the keypoints detected from the template image, and the weights of the similarities of each of the plurality of keypoints in the calculation of the weighted average value or weighted sum are the weighting values of each of the plurality of keypoints determined in advance. An image processing apparatus.

2. The output means The image processing apparatus according to claim 1, wherein the output means causes the output device to output a template image in which a human body posture or movement captured at the location in the first image and a human body with a posture or movement satisfying the first similarity condition are captured.

3. A computer performs a process of detecting keypoints of a human body included in an image, calculates the similarity between the feature amount of the keypoint detected from the first image and the feature amount of the keypoint detected from a pre-registered template image as a first similarity between the posture or movement of the human body detected from the first image and the posture or movement of the human body shown in the template image, although the first similarity with the posture or movement of the human body shown in any of the template images is less than a first threshold value, a location in the first image where a human body satisfying the first similarity condition with the posture or movement of the human body shown in any of the template images is captured is specified, as a candidate for the template image to be additionally registered in a determination device that detects a human body in which a second similarity between the feature amount of the keypoint of the human body detected from the second image and the feature amount of the keypoint of the human body detected from the template image is equal to or greater than the first threshold value, information indicating the location in the first image or a partial image obtained by cutting out the location from the first image is output to an output device, The first similarity condition is that the first similarity is equal to or greater than a second threshold value and less than the first threshold value, that a third similarity, which is the similarity between the feature amount of the first keypoint detected from the first image and the feature amount of the first keypoint detected from the template image, is equal to or greater than a third threshold value. The fourth similarity calculated from the feature amounts of each of the plurality of key points detected from the first image, the feature amounts of each of the plurality of key points detected from the template image, and the weighting values of each of the plurality of predetermined key points is equal to or greater than a fourth threshold value, and a moving image including a plurality of frame images from each of which a feature amount of a plurality of key points extracted from a predetermined ratio or more of the plurality of frame images included in the template image which is a moving image is a fifth similarity equal to or greater than a fifth threshold value, is at least one of them, the second threshold value is smaller than the first threshold value, the first key point is a predetermined part of the plurality of key points detected by the skeleton structure detecting means, the fourth similarity is a weighted average value or a weighted sum of the similarity between the feature amount of the key point detected from the first image and the feature amount of the key point detected from the template image calculated for each key point, and the weight of the similarity of each of the plurality of key points in the calculation of the weighted average value or the weighted sum is the weighting value of each of the plurality of predetermined key points, An image processing method.

4. A computer, skeleton structure detecting means for performing a process of detecting key points of a human body included in an image, similarity calculating means for calculating, as a first similarity, the similarity between the feature amount of the key point detected from the first image and the feature amount of the key point detected from a pre-registered template image, with respect to the posture or movement of the human body shown in the first image and the posture or movement of the human body shown in the template image, specifying means for specifying a location in the first image where a human body that satisfies a first similarity condition with the posture or movement of the human body shown in any of the template images appears, although the first similarity with the posture or movement of the human body shown in any of the template images is less than a first threshold value, As a candidate for the template image to be additionally registered in a determination device that detects a human body from among the human bodies detected from the second image, and for which the second similarity between the feature amount of the key point of the human body detected from the second image and the feature amount of the key point of the human body detected from the template image is equal to or greater than the first threshold value, output means for outputting to an output device information indicating the location in the first image or a partial image obtained by cutting out the location from the first image, functioning as, The first similarity condition is that, the first similarity is equal to or greater than a second threshold value and less than the first threshold value, a third similarity, which is the similarity between the feature amount of the first key point detected from the first image and the feature amount of the first key point detected from the template image, is equal to or greater than a third threshold value, a fourth similarity calculated from the feature amounts of each of the plurality of key points detected from the first image, the feature amounts of each of the plurality of key points detected from the template image, and weight values of each of the plurality of predetermined key points is equal to or greater than a fourth threshold value, and the template image is a moving image including a plurality of frame images from each of which a feature amount of a plurality of key points extracted from a predetermined ratio or more of the plurality of frame images included in the template image, which is a moving image, and for which a fifth similarity, which is the similarity with the feature amount of the key point, is equal to or greater than a fifth threshold value, is at least one of, the second threshold value is smaller than the first threshold value, the first key point is a predetermined part of the plurality of key points detected by the skeleton structure detection means, the fourth similarity is a weighted average value or a weighted sum of the similarities between the feature amount of the key point detected from the first image and the feature amount of the key point detected from the template image, calculated for each key point, and the weights of the similarities of each of the plurality of key points in the calculation of the weighted average value or the weighted sum are the weight values of each of the plurality of predetermined key points, program.

Images