Image processing apparatus, image processing method, and program

Abstract

This invention provides an image processing device (10) comprising: a skeletal structure detecting unit (11) that performs a processing of detecting the key points of human bodies included in an image; a calculation unit (12) that calculates the quality values of the detected key points for each human body; and an output unit (13) that outputs information indicating parts in which the human bodies for which the quality values are equal to or greater than a threshold value are seen, or that outputs a partial image obtained by cutting out those parts from the image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Technologies related to the present invention are disclosed in Patent Document 1 and Non-Patent Document 1. Patent Document 1 discloses a technique for calculating feature amounts of a plurality of key points of a human body included in an image, and searching for an image including a human body having a similar posture or a similar movement based on the calculated feature amounts, or classifying together those having similar postures or movements. Non-Patent Document 1 discloses a technique related to human skeleton estimation.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the technology disclosed in Patent Document 1 described above, by registering in advance, as a template image, an image including a human body in a desired posture or desired movement, it is possible to detect a human body in a desired posture or desired movement from among images to be processed. As a result of studying the technology disclosed in Patent Document 1, the inventor has newly found that the detection accuracy deteriorates unless an image of a certain quality is registered as a template image, and that there is room for improvement in the workability of the operation of preparing such a template image.

[0006] Since both Patent Document 1 and Non-Patent Document 1 described above do not disclose problems related to template images and solutions therefor, there is a problem that the above problems cannot be solved.

[0007] An example of the object of the present invention is to provide an image processing apparatus, an image processing method, and a program that solve the problem of workability in preparing a template image of a certain quality in view of the above-described problems.

Means for Solving the Problems

[0008] According to one aspect of the present invention, skeleton structure detection means for performing a process of detecting key points of a human body included in an image; calculation means for calculating a quality value of the detected key points for each human body; output means for outputting information indicating a location where a human body with a quality value equal to or greater than a threshold value appears, or a partial image obtained by cutting out the location from the image; An image processing apparatus having the above is provided.

[0009] Also, according to one aspect of the present invention, one or more computers perform a process of detecting key points of a human body included in an image, calculate a quality value of the detected key points for each human body, and output information indicating a location where a human body with a quality value equal to or greater than a threshold value appears, or a partial image obtained by cutting out the location from the image. An image processing method is provided.

[0010] Also, according to one aspect of the present invention, a computer is caused to function as skeleton structure detection means for performing a process of detecting key points of a human body included in an image, calculation means for calculating a quality value of the detected key points for each human body, output means for outputting information indicating a location where a human body with a quality value equal to or higher than a threshold value appears, or a partial image obtained by cutting out the location from the image, and a program is provided.

Advantages of the Invention

[0011] According to one aspect of the present invention, an image processing apparatus, an image processing method, and a program for solving the problem of workability of the work of preparing a template image of a certain quality can be obtained.

Brief Description of the Drawings

[0012] The above-described object, as well as other objects, features, and advantages, will become more apparent from the following public embodiments and the accompanying drawings described below.

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Best Mode for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0015] <First Embodiment> FIG. 1 is a functional block diagram showing an overview of an image processing apparatus 10 according to the first embodiment. As shown in FIG. 1, the image processing apparatus 10 includes a skeleton structure detection unit 11, a calculation unit 12, and an output unit 13. The skeleton structure detection unit 11 performs a process of detecting key points of a human body included in an image. The calculation unit 12 calculates a quality value of the detected key points for each human body. The output unit 13 outputs information indicating a location where a human body with a quality value equal to or higher than a threshold value appears, or a partial image obtained by cutting out the location from the image.

[0016] According to this image processing apparatus 10, it is possible to solve the problem of workability in the work of preparing a template image of a certain quality.

[0017] <Second Embodiment> "Overview" When the image processing apparatus 10 detects key points of a human body included in an image, based on the confidence level of the detection result of the key points, it calculates a quality value of the detected key points for each detected human body. Then, the image processing apparatus 10 outputs information indicating a location where a human body with the above quality value equal to or higher than a threshold value appears, or a partial image obtained by cutting out the location from the image.

[0018] The user can prepare a template image of a certain quality by selecting a template image from the location where a human body with the above quality value equal to or higher than a threshold value appears.

[0019] "Hardware Configuration" Next, an example of the hardware configuration of the image processing apparatus will be described. Each functional unit of the image processing apparatus is realized by an arbitrary combination of hardware and software centered around the CPU (Central Processing Unit), memory, program loaded into the memory, storage unit such as a hard disk storing the program (it can store not only programs pre-stored at the stage of shipping the apparatus but also programs downloaded from storage media such as CD (Compact Disc) or servers on the Internet), and network connection interface. And it is understood by those skilled in the art that there are various modifications to the realization method and apparatus.

[0020] FIG. 2 is a block diagram illustrating the hardware configuration of the image processing apparatus 10. As shown in FIG. 2, the image processing apparatus 10 includes a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The image processing apparatus 10 may not have the peripheral circuit 4A. Note that the image processing apparatus 10 may be composed of a plurality of physically and / or logically separated apparatuses. In this case, each of the plurality of apparatuses can have the above-described hardware configuration.

[0021] Bus 5A is a data transmission path for the processor 1A, the memory 2A, the peripheral circuit 4A, and the input / output interface 3A to transmit and receive data from each other. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and an interface for outputting information to an output device, an external device, an external server, etc. The input device is, for example, a keyboard, a mouse, a microphone, a physical button, a touch panel, etc. The output device is, for example, a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform operations based on their operation results.

[0022] "Functional Configuration" FIG. 1 is a functional block diagram showing an overview of the image processing apparatus 10 according to the second embodiment. As shown in FIG. 1, the image processing apparatus 10 includes a skeleton structure detection unit 11, a calculation unit 12, and an output unit 13.

[0023] The skeleton structure detection unit 11 performs a process of detecting key points of a human body included in an image.

[0024] The "image" is an image that is the source of the template image. The template image is an image that is pre-registered in the technology disclosed in Patent Document 1 described above, and is an image including a human body in a desired posture or a desired movement (a posture or movement that the user wants to detect). The image may be a moving image composed of a plurality of frame images, or may be a still image composed of one image.

[0025] The skeletal structure detection unit 11 detects N (N is an integer of 2 or more) key points of the human body included in the image. When the moving image is the processing target, the skeletal structure detection unit 11 performs the process of detecting key points for each frame image. The process by the skeletal structure detection unit 11 is realized using the technology disclosed in Patent Document 1. Although the details are omitted, in the technology disclosed in Patent Document 1, the detection of the skeletal structure is performed using a skeletal estimation technology such as OpenPose disclosed in Non-Patent Document 1. The skeletal structure detected by the technology is composed of "key points" which are characteristic points such as joints, and "bones (bone links)" indicating the links between the key points.

[0026] Figure 3 shows the skeletal structure of the human body model 300 detected by the skeletal structure detection unit 11, and Figures 4 and 5 show detection examples of the skeletal structure. The skeletal structure detection unit 11 uses a skeletal estimation technology such as OpenPose to detect the skeletal structure of the human body model (2D skeletal model) 300 as shown in Figure 3 from a 2D image. The human body model 300 is a 2D model composed of key points such as the joints of a person and bones connecting each key point.

[0027] The skeletal structure detection unit 11 extracts, for example, feature points that can be key points from the image, and refers to the information obtained by machine learning of the key point images to detect N key points of the human body. The N key points to be detected are determined in advance. The number of key points to be detected (that is, the number of N) and which part of the human body is to be the key point to be detected vary, and all variations can be adopted.

[0028] Hereinafter, as shown in FIG. 3, it is assumed that the head A1, neck A2, right shoulder A31, left shoulder A32, right elbow A41, left elbow A42, right hand A51, left hand A52, right hip A61, left hip A62, right knee A71, left knee A72, right foot A81, and left foot A82 are defined as N key points to be detected (N = 14). In the human body model 300 shown in FIG. 3, as the bones of a person connecting these key points, bone B1 connecting the head A1 and the neck A2, bone B21 and bone B22 respectively connecting the neck A2 with the right shoulder A31 and the left shoulder A32, bone B31 and bone B32 respectively connecting the right shoulder A31 and the left shoulder A32 with the right elbow A41 and the left elbow A42, bone B41 and bone B42 respectively connecting the right elbow A41 and the left elbow A42 with the right hand A51 and the left hand A52, bone B51 and bone B52 respectively connecting the neck A2 with the right hip A61 and the left hip A62, bone B61 and bone B62 respectively connecting the right hip A61 and the left hip A62 with the right knee A71 and the left knee A72, and bone B71 and bone B72 respectively connecting the right knee A71 and the left knee A72 with the right foot A81 and the left foot A82 are further defined.

[0029] FIG. 4 is an example of detecting a person in an upright state. In FIG. 4, an upright person is imaged from the front, and bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 seen from the front are detected without overlapping each other, and the bones B61 and B71 of the right foot are slightly bent more than the bones B62 and B72 of the left foot.

[0030] FIG. 5 is an example of detecting a person in a crouched state. In FIG. 5, a crouched person is imaged from the right side, and bone B1, bone B51 and bone B52, bone B61 and bone B62, bone B71 and bone B72 seen from the right side are detected respectively, and the bones B61 and B71 of the right foot and the bones B62 and B72 of the left foot are greatly bent and overlapping.

[0031] Returning to FIG. 1, the calculation unit 12 calculates the quality value of the detected keypoints for each person. Then, the calculation unit 12 identifies the location within the image in which a person whose quality value of the detected keypoints is equal to or greater than the threshold value appears. Hereinafter, these processes will be described in detail.

[0032] - Process of calculating the quality value of the detected keypoints - The calculation unit 12 calculates the quality value of the detected keypoints. The "quality value of the detected keypoints" is a value indicating how good the quality of the detected keypoints is, and can be calculated based on various data. In the present embodiment, the calculation unit 12 calculates the quality value based on the confidence level of the detection result of the keypoints. In the following embodiments, an example of calculating the above quality value based on data other than the confidence level of the detection result of the keypoints will be described. The method of calculating the confidence level is not particularly limited. For example, in a skeleton estimation technique such as OpenPose, the score output in association with each detected keypoint may be used as the confidence level of each keypoint.

[0033] The higher the confidence level of the detection result of the keypoints is, the higher the quality value calculated by the calculation unit 12 is. For example, the calculation unit 12 may calculate a statistical value (average value, maximum value, minimum value, median value, mode value, weighted average value, etc.) of the confidence levels of each of the N keypoints detected from each person as the quality value. When some of the N keypoints are not detected, the confidence level of the undetected keypoint may be a fixed value such as "0". This fixed value is set to be lower than the confidence level of the detected keypoints.

[0034] Note that when the image is a still image, the calculation unit 12 calculates the quality value for each person detected from the still image. On the other hand, when the image is a moving image, the calculation unit 12 calculates the quality value for each person detected from each of the plurality of frame images.

[0035] - Process of identifying the location within the image in which a person whose quality value of the detected keypoints is equal to or greater than the threshold value appears - Based on the calculation result of the process of calculating the above-described quality value, the calculation unit 12 identifies a location within the image in which a human body with a quality value of the detected keypoint being equal to or greater than the threshold value is captured. For each detected human body, the calculation unit 12 determines whether the quality value of the detected keypoint is equal to or greater than the threshold value. Then, based on the determination result, the calculation unit 12 identifies the location in which the human body with a quality value equal to or greater than the threshold value is captured.

[0036] When the image is a still image, the "location in which a human body with a quality value equal to or greater than the threshold value is captured" is a partial area within a single still image. In this case, for example, the location within the image in which a human body with a quality value of the detected keypoint being equal to or greater than the threshold value is captured is indicated by the coordinates in the coordinate system set for the image.

[0037] On the other hand, when the image is a moving image, the "location in which a human body with a quality value equal to or greater than the threshold value is captured" is a partial area within each of some of the frame images constituting the moving image. In this case, for example, the location within the image in which a human body with a quality value of the detected keypoint being equal to or greater than the threshold value is captured is indicated by information indicating some of the frame images among the plurality of frame images (frame identification information, elapsed time from the start, etc.) and the coordinates in the coordinate system set for the image.

[0038] Note that when the image is a moving image, it is preferable to identify "the location in which the same human body is continuously captured in each of a plurality of frame images and the condition that 'the quality value of the keypoint detected from the human body is equal to or greater than the threshold value' is satisfied".

[0039] Therefore, the calculation unit 12 may identify the human body of the same person captured across a plurality of frame images. The method for realizing the identification is not particularly limited. For example, a person tracking technique, a face recognition technique, or the like may be used to identify the same person captured across a plurality of frame images, and the human body detected at the position within each of the plurality of frame images in which the same person is captured may be identified as the human body of the same person. By this process, the calculation unit 12 can identify a plurality of frame images in which the human body of the same person is continuously captured.

[0040] Next, the condition of "the quality value of the key points detected from the human body is equal to or greater than the threshold value" will be described. This condition may require that all of the plurality of frame images satisfy this condition. That is, the calculation unit 12 may identify a plurality of frame images in which the human body of the same person continues to appear and the quality value of the key points detected from the human body in all of the frame images is equal to or greater than the threshold value.

[0041] Alternatively, the above condition may require that at least a part of the plurality of frame images satisfy the above condition. That is, the calculation unit 12 may identify a plurality of frame images in which the human body of the same person continues to appear and the quality value of the key points detected from the human body in at least a part of the frame images is equal to or greater than the threshold value. In this case, as a condition for the plurality of frame images, further, "the number of consecutive frame images in which a human body with a quality value less than the threshold value appears is Q or less" and the like may be added. By adding such an additional condition, it is possible to suppress the inconvenience that a portion where a human body with a low quality value appears continuously for a predetermined number of frames or more is specified as a candidate for the template image.

[0042] The output unit 13 outputs information indicating a location where a human body with a quality value equal to or greater than the threshold value (a human body whose quality value of the detected key points is above the threshold value) appears, or a partial image obtained by cutting out the location from the image. When the image is a moving image, the output unit 13 may output information indicating a location where the human body appears in each of the plurality of frame images in which the human body of the same person continues to appear and that satisfy the condition of "the quality value of the key points detected from the human body is equal to or greater than the threshold value", or a partial image obtained by cutting out the location from the image.

[0043] Note that when the output unit 13 outputs a partial image, the image processing apparatus 10 may include a processing unit that cuts out a location where a human body with a quality value equal to or greater than the threshold value from the image to generate a partial image. Then, the output unit 13 can output the partial image generated by the processing unit.

[0044] A location where a human body with a quality value equal to or higher than a threshold value is captured serves as a candidate for a template image. Based on the above information or the partial image, the user can view, etc., the location where a human body with a quality value equal to or higher than the threshold value is captured, and select, as a template image, a location including a human body in a desired posture or a desired movement.

[0045] FIG. 6 schematically shows an example of the information output by the output unit 13. In the example shown in FIG. 6, the human body identification information for identifying a plurality of detected human bodies from each other and the attribute information of each human body are displayed in association with each other. And, as an example of the attribute information, a quality value, information indicating a location within the image (information indicating the location where the above-mentioned human body is captured), and the shooting date and time of the image are displayed. The attribute information may further include information indicating the installation position (shooting position) of the camera that captured the image (e.g., the rear inside of the No. 102 bus, the entrance of XX Park, etc.) and the attribute information of the person calculated by image analysis (e.g., gender, age group, body type, etc.).

[0046] Next, an example of the processing flow of the image processing apparatus 10 will be described using the flowchart of FIG. 7.

[0047] When an image serving as the source of the template image is input to the image processing apparatus 10, the image processing apparatus 10 performs a process of detecting the key points of the human body included in the image (S10). Next, the image processing apparatus 10 calculates the quality value of the detected key points for each detected human body (S11). Next, the image processing apparatus 10 determines whether the quality value of the detected key points is equal to or higher than the threshold value for each detected human body (S12). Next, the image processing apparatus 10 specifies the location where a human body with a quality value equal to or higher than the threshold value is captured according to the determination result of S12 (S13). Then, the image processing apparatus 10 outputs information indicating the location where a human body with a quality value equal to or higher than the threshold value is captured, or a partial image obtained by cutting out the location from the image (S14).

[0048] "Function and effect" According to the image processing apparatus 10 of the second embodiment, the same operational effects as those of the first embodiment are achieved. Further, according to the image processing apparatus 10 of the second embodiment, it is possible to present to the user, as candidates for the template image, portions where a human body with a high confidence level of the detection result of the key points is captured. By selecting a template image from among the candidates for the template image presented in this way, the user can easily prepare a template image whose confidence level of the detection result of the key points satisfies a certain quality.

[0049] <Third Embodiment> The image processing apparatus 10 of the third embodiment differs from the first and second embodiments in the way of calculating the quality value.

[0050] The calculation unit 12 calculates a higher quality value for a human body with a relatively large number of detected key points than for a human body with a relatively small number of detected key points. For example, the calculation unit 12 may use the number of detected key points as the quality value. Alternatively, a weighting point may be set for each of the plurality of key points. A higher weighting point is set for a relatively important key point. Then, the calculation unit 12 may calculate, as the quality value, the sum of the weighting points of each of the detected key points.

[0051] Alternatively, the calculation unit 12 may calculate the quality value by combining the method described in the second embodiment and the method based on the number of the detected key points. For example, the calculation unit 12 calculates a first quality value by normalizing the quality value calculated by the method described in the second embodiment according to a predetermined rule, and calculates a second quality value by normalizing the quality value calculated by the method based on the number of the detected key points according to a predetermined rule. Then, the calculation unit 12 may calculate, as the quality value of the human body, a statistical value (average value, maximum value, minimum value, median value, mode value, weighted average value, etc.) of the first quality value and the second quality value.

[0052] Other configurations of the image processing apparatus 10 of the third embodiment are the same as those of the first and second embodiments.

[0053] According to the image processing apparatus 10 of the third embodiment, the same operational effects as those of the first and second embodiments are achieved. Further, according to the image processing apparatus 10 of the third embodiment, it is possible to present to the user, as candidates for the template image, portions in which a human body with many detected key points appears. The user can easily prepare a template image in which the number of detected key points satisfies a certain quality by selecting a template image from among the candidates for the template image presented in this way.

[0054] <Fourth Embodiment> In the image processing apparatus 10 of the fourth embodiment, the method of calculating the quality value is different from those of the first to third embodiments.

[0055] The calculation unit 12 calculates a quality value based on the degree of overlap with other human bodies. Note that the state in which "the human body of person A overlaps with the human body of person B" includes a state in which the human body of person A is partially or entirely hidden by the human body of person B, a state in which the human body of person A hides part or all of the human body of person B, and a state in which both occur. Hereinafter, the calculation method will be specifically described.

[0056] -First Method- The calculation unit 12 calculates the quality value of a human body that does not overlap with other human bodies to be higher than the quality value of a human body that overlaps with other human bodies. For example, a rule is created in advance and stored in the image processing apparatus 10 such that the quality value of a human body that does not overlap with other human bodies is X1 and the quality value of a human body that overlaps with other human bodies is X2. Note that X1 > X2. Then, the calculation unit 12 calculates the quality value of a human body that does not overlap with other human bodies as X1 and the quality value of a human body that overlaps with other human bodies as X2 based on the rule. In this case, the output unit 13 can output information indicating a portion in which a human body with a quality value of Y or more appears, or a partial image obtained by cutting out the portion from the image. Note that X1 > Y > X2.

[0057] Whether it overlaps with another human body may be specified based on the degree of overlap of the human body model 300 (see FIG. 3) detected by the skeleton structure detection unit 11, or may be specified based on the degree of overlap of the bodies shown in the image.

[0058] For example, when the distance within the image of predetermined key points (e.g., head A1) of two human bodies is equal to or less than a threshold value, it may be determined that the two human bodies overlap. In this case, the threshold value may be a variable value that changes according to the size within the image of the detected human body. The larger the size within the image of the detected human body, the larger the threshold value. Note that instead of the size within the image of the human body, the length of a predetermined bone (e.g., bone B1 connecting head A1 and neck A2), the size of the face within the image, etc. may be adopted.

[0059] In addition, when any bone of a certain human body intersects with any bone of another human body, it may be determined that the two human bodies overlap with each other.

[0060] -Second method- The calculation unit 12 calculates the quality value of a human body that does not overlap with other human bodies to be higher than the quality value of a human body that overlaps with other human bodies, and among the human bodies that overlap with other human bodies, calculates the quality value of the human body located in the front to be higher than the quality value of the human body located in the back.

[0061] That is, the calculation unit 12 calculates the quality value of a human body that does not overlap with other human bodies to be the highest, calculates the quality value of a human body that overlaps with other human bodies and is located in the front to be the next highest, and calculates the quality value of a human body that overlaps with other human bodies and is located in the back to be the lowest.

[0062] For example, a rule is created in advance and stored in the image processing apparatus 10, where the quality value of a human body that does not overlap with other human bodies is set as X1, the quality value X 21 of a human body that overlaps with other human bodies and is located in the front, and the quality value X 22 of a human body that overlaps with other human bodies and is located in the back. Note that X1 > X 21 > X 22That is, the calculation unit 12 calculates the quality value X1 of the human body that does not overlap with other human bodies based on the rule, and calculates the quality value X 21 of the human body that overlaps with other human bodies and is located in the front side, and calculates the quality value X 22 of the human body that overlaps with other human bodies and is located in the rear side. In this case, the output unit 13 can output information indicating a location where a human body with a quality value of Z or more appears, or a partial image obtained by cutting out the location from the image. Note that X1>X 21 >Z>X 22 , or X1>Z>X 21 >X 22 .

[0063] Whether it is in front of or behind other human bodies may be specified based on the occlusion or loss condition of the human body model 300 (see FIG. 3) detected by the skeletal structure detection unit 11, or may be specified based on the occlusion condition of the body shown in the image. For example, among two human bodies overlapping each other, if all N key points are detected for one body and only a part of the N key points are detected for the other body, it can be determined that the human body for which all N key points are detected is located in the front side and the other human body is located in the rear side.

[0064] Note that the calculation unit 12 may calculate the quality value by combining at least one of the methods described in the second and third embodiments and a method based on the degree of overlap with the other human body. For example, the calculation unit 12 performs at least one of a process of calculating a first quality value by normalizing the quality value calculated by the method described in the second embodiment with a predetermined rule, and a process of calculating a second quality value by normalizing the quality value calculated by the method described in the third embodiment with a predetermined rule. Further, the calculation unit 12 calculates a third quality value by normalizing the quality value calculated by the method based on the degree of overlap with the other human body with a predetermined rule. Then, the calculation unit 12 may calculate a statistical value (average value, maximum value, minimum value, median value, mode value, weighted average value, etc.) of at least one of the first and second quality values and the third quality value as the quality value of the human body.

[0065] The other configurations of the image processing apparatus 10 according to the fourth embodiment are the same as those of the first to third embodiments.

[0066] According to the image processing apparatus 10 of the fourth embodiment, the same operational effects as those of the first to third embodiments are realized. Further, according to the image processing apparatus 10 of the fourth embodiment, a location where a human body that does not overlap with other human bodies appears can be presented to the user as a candidate for the template image. Further, according to the image processing apparatus 10 of the fourth embodiment, in addition to the location where a human body that does not overlap with other human bodies appears, a location where a human body that overlaps with other human bodies but is located in the front appears can be presented to the user as a candidate for the template image. By selecting a template image from among the candidates for the template image presented in this way, the user can easily prepare a template image whose degree of overlap with other human bodies satisfies a certain quality.

[0067] <Fifth Embodiment> The image processing apparatus 10 according to the fifth embodiment differs from the first to fourth embodiments in the way of calculating the quality value.

[0068] First, the skeleton structure detection unit 11 detects a person region in the image and performs a process of detecting key points within the detected person region. That is, the skeleton structure detection unit 11 does not target all regions in the image for the process of detecting key points, but only the detected person region for the process of detecting key points. The details of the process of detecting a person region in the image are not particularly limited and may be realized using an object detection technique such as YOLO.

[0069] Then, the calculation unit 12 calculates a quality value based on the confidence level of the detection result of the person region. The method for calculating the confidence level of the detection result of the person region is not particularly limited. For example, in an object detection technique such as YOLO, the score (which may also be referred to as a confidence level, etc.) output in association with the detected object region may be used as the confidence level of each person region.

[0070] The calculation unit 12 calculates a higher quality value as the confidence level of the detection result of the human region is higher. For example, the calculation unit 12 may calculate the confidence level of the detection result of the human region as the quality value.

[0071] In addition, the calculation unit 12 may calculate the quality value by combining at least one of the methods described in the second to fourth embodiments with the method based on the confidence level of the detection result of the human region. For example, the calculation unit 12 performs a process of calculating a first quality value by normalizing the quality value calculated by the method described in the second embodiment according to a predetermined rule, a process of calculating a second quality value by normalizing the quality value calculated by the method described in the third embodiment according to a predetermined rule, and a process of calculating a third quality value by normalizing the quality value calculated by the method described in the fourth embodiment according to a predetermined rule. At least one of them. Further, the calculation unit 12 calculates a fourth quality value by normalizing the quality value calculated by the method based on the confidence level of the detection result of the human region according to a predetermined rule. Then, the calculation unit 12 may calculate a statistical value (average value, maximum value, minimum value, median value, mode value, weighted average value, etc.) of at least one of the first to third quality values and the fourth quality value as the quality value of the human body.

[0072] Other configurations of the image processing apparatus 10 according to the fifth embodiment are the same as those in the first to fourth embodiments.

[0073] According to the image processing apparatus 10 of the fifth embodiment, the same operational effects as those in the first to fourth embodiments are realized. Further, according to the image processing apparatus 10 of the fifth embodiment, a portion where a person appears with high confidence can be presented to the user as a candidate for the template image. The user can easily prepare a template image whose detection result of the human region satisfies a certain quality by selecting a template image from among the candidates for the template image presented in this way.

[0074] <Sixth Embodiment> The image processing apparatus 10 according to the sixth embodiment is different from the first to fifth embodiments in the way of calculating the quality value.

[0075] The calculation unit 12 calculates a quality value based on the size of the human body in the image. The calculation unit 12 calculates a quality value of a relatively large human body to be higher than that of a relatively small human body. The size of the human body in the image may be indicated by the size (area, etc.) of the person region shown in the fifth embodiment, may be indicated by the length of a predetermined bone (e.g., bone B1), may be indicated by the length between a predetermined two key points (e.g., key points A31 and A32), or may be indicated by other methods.

[0076] In addition, the calculation unit 12 may calculate a quality value by combining at least one of the methods described in the second to fifth embodiments and a method based on the size of the human body in the above image. For example, the calculation unit 12 performs a process of normalizing the quality value calculated by the method described in the second embodiment according to a predetermined rule to calculate a first quality value, a process of normalizing the quality value calculated by the method described in the third embodiment according to a predetermined rule to calculate a second quality value, a process of normalizing the quality value calculated by the method described in the fourth embodiment according to a predetermined rule to calculate a third quality value, and a process of normalizing the quality value calculated by the method described in the fifth embodiment according to a predetermined rule to calculate a fourth quality value, and performs at least one of these processes. Further, the calculation unit 12 normalizes the quality value calculated by the method based on the size of the human body in the above image according to a predetermined rule to calculate a fifth quality value. Then, the calculation unit 12 may calculate a statistical value (average value, maximum value, minimum value, median value, mode value, weighted average value, etc.) of at least one of the first to fourth quality values and the fifth quality value as the quality value of the human body.

[0077] Other configurations of the image processing apparatus 10 in the sixth embodiment are the same as those in the first to fifth embodiments.

[0078] According to the image processing apparatus 10 of the sixth embodiment, the same operational effects as those of the first to fifth embodiments are achieved. Further, according to the image processing apparatus 10 of the sixth embodiment, a location where a human body is captured to a certain extent can be presented to the user as a candidate for the template image. The user can easily prepare a template image whose human body size satisfies a certain quality by selecting a template image from among the presented template image candidates.

[0079] <Modification Example 1> In the case where a plurality of images obtained by simultaneously photographing the same person with a plurality of cameras are input to the image processing apparatus 10, and when the quality values of the key points detected from each of the human bodies of the same person detected from each of the plurality of images are equal to or higher than a threshold value, the output unit 13 may output information indicating a location where the human body with the highest quality value among the human bodies of the same person detected from each of the plurality of images is captured, or a partial image obtained by cutting out that location from the image. In this modification example, in addition to the information described in the second embodiment, the identification information of the image is included in the "information indicating the location where the human body with a quality value equal to or higher than the threshold value is captured".

[0080] <Modification Example 2> In the above embodiment, when the image is a moving image, the "location where the human body with a quality value equal to or higher than the threshold value is captured" is a partial area within each of some of the frame images constituting the moving image. Then, the output unit 13 outputs information indicating such a location or a partial image obtained by cutting out such a location from the image. This is a configuration assuming that a plurality of human bodies may be included in one frame image.

[0081] As a modification, when the image is a moving image, the part of the moving image in which a human body with a quality value equal to or higher than the threshold value appears may be a part of a plurality of frame images constituting the moving image. Then, the output unit 13 may output information indicating a part of such a plurality of frame images, or a partial image obtained by cutting out a part of the frame images from the image. Further, the frame image itself in which a human body with a quality value equal to or higher than the threshold value appears may be output as a candidate for the template image. This is a configuration assuming that only one human body with a quality value equal to or higher than the threshold value can be included in one frame image.

[0082] As described above, the embodiments of the present invention have been described with reference to the drawings, but these are examples of the present invention, and various configurations other than the above can also be adopted.

[0083] Also, in the plurality of flowcharts used in the above description, a plurality of steps (processes) are described in order, but the execution order of the steps executed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown can be changed within a range that does not substantially affect the content. Also, the above-described embodiments can be combined within a range where the contents do not conflict.

[0084] Some or all of the above embodiments may be described as follows in the following supplementary notes, but are not limited thereto. 1. A skeleton structure detection means for performing a process of detecting key points of a human body included in an image, For each human body, a calculation means for calculating a quality value of the detected key points, An output means for outputting information indicating a location where a human body with a quality value equal to or higher than the threshold value appears, or a partial image obtained by cutting out the location from the image, An image processing apparatus having the above. 2. The image processing apparatus according to 1, wherein the calculation means calculates the quality value based on the confidence level of the detection result of the key points. 3. The skeleton structure detection means detects a person area in the image and performs a process of detecting the key points within the detected person area, The image processing apparatus according to 1 or 2, wherein the calculation means calculates the quality value based on the confidence level of the detection result of the person area. 4. The image processing apparatus according to any one of 1 to 3, wherein the calculation means calculates the quality value based on the degree of overlap with other human bodies. 5. The image processing apparatus according to 4, wherein the calculation means calculates the quality value of a human body that does not overlap with other human bodies to be higher than the quality value of a human body that overlaps with other human bodies. 6. The image processing apparatus according to 5, wherein the calculation means calculates the quality value of a human body located on the front side among the human bodies that overlap with other human bodies to be higher than the quality value of a human body located on the rear side. 7. The image processing apparatus according to any one of 1 to 6, wherein the calculation means calculates the quality value of a human body with a relatively large number of detected keypoints to be higher than the quality value of a human body with a relatively small number of detected keypoints. 8. The image processing apparatus according to any one of 1 to 7, wherein the calculation means calculates the quality value based on the size of the human body on the image. 9. One or more computers perform a process of detecting keypoints of a human body included in an image, calculate the quality value of the detected keypoints for each human body, output information indicating a location where a human body with a quality value equal to or higher than a threshold value appears, or a partial image obtained by cutting out the location from the image. Image processing method. 10. A program that causes a computer to function as a skeleton structure detection means for performing a process of detecting keypoints of a human body included in an image, a calculation means for calculating the quality value of the detected keypoints for each human body, an output means for outputting information indicating a location where a human body with a quality value equal to or higher than a threshold value appears, or a partial image obtained by cutting out the location from the image. Program

Description of Reference Numerals

[0085] 10 Image processing apparatus 11 Skeleton structure detection unit 12 Calculation unit 13 Output unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. Skeleton structure detection means for performing a process of detecting key points of a human body included in an image, Calculation means for calculating a quality value of the detected key points for each human body, Output means for outputting information indicating a location where a human body with a quality value equal to or higher than a threshold value appears, or a partial image obtained by cutting out the location from the image, having, The calculation means calculates the quality value based on the degree of overlap with other human bodies. An image processing apparatus.

2. The calculation means calculates the quality value of a human body that does not overlap with other human bodies to be higher than the quality value of a human body that overlaps with other human bodies. The image processing apparatus according to claim 1.

3. The calculation means calculates the quality value of a human body located in the front among human bodies that overlap with other human bodies to be higher than the quality value of a human body located in the rear. The image processing apparatus according to claim 2.

4. The calculation means calculates the quality value based on the confidence level of the detection result of the key points. The image processing apparatus according to any one of claims 1 to 3.

5. The skeleton structure detection means detects a person area in the image and performs a process of detecting the key points within the detected person area, The calculation means calculates the quality value based on the confidence level of the detection result of the person area. The image processing apparatus according to any one of claims 1 to 4.

6. The calculation means calculates the quality value of a human body with a relatively large number of detected key points to be higher than the quality value of a human body with a relatively small number of detected key points. The image processing apparatus according to any one of claims 1 to 5.

7. The calculation means calculates the quality value based on the size of the human body on the image. The image processing apparatus according to any one of claims 1 to 6.

8. One or more computers, perform a process of detecting key points of a human body included in an image, calculate a quality value of the detected key points for each human body, output information indicating a location where a human body with a quality value equal to or higher than a threshold value appears, or a partial image obtained by cutting out the location from the image, In the process of calculating the quality value, the quality value is calculated based on the degree of overlap with other human bodies. An image processing method.

9. A computer, skeleton structure detection means for performing a process of detecting key points of a human body included in an image, calculation means for calculating a quality value of the detected key points for each human body, Information indicating a location where a human body with the quality value equal to or higher than the threshold value appears, or output means for outputting a partial image obtained by cutting out the location from the image, function as, The calculation means is a program for calculating the quality value based on the degree of overlap with other human bodies.

Images