Image processing system, apparatus, processing method, and program

Abstract

To achieve faster search processing in similar image search processing based on feature amounts of each of a plurality of key points of a human body included in an image.SOLUTION: An image processing system 10 includes: a target image acquisition unit 11 that acquires a target image; a skeleton structure detection unit 12 that performs processing to detect a key point of a human body included in the target image; a first collation unit 13 that extracts, on the basis of the detected key point, from among a plurality of reference images, a first reference image whose relationship with the target image satisfies a first extraction condition; and a second collation unit 14 that extracts, on the basis of the detected key point, from the first reference image, a second reference image whose relationship with the target image satisfies a second extraction condition.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing system, apparatus, processing method, and program.

Background Art

[0002] Technologies related to the present invention are disclosed in Patent Documents 1 to 3 and Non-Patent Document 1.

[0003] Patent Document 1 discloses a technique for calculating feature amounts of a plurality of key points of a human body included in an image, and searching for an image including a human body with a similar posture or a similar movement based on the calculated feature amounts, or classifying together those with similar postures or movements. Further, Non-Patent Document 1 discloses a technique related to human skeleton estimation.

[0004] Patent Document 2 discloses a technique for estimating skeleton data of a person included in an image, calculating the weight of a joint based on the reliability of the estimation result of each joint, and calculating the similarity between the estimated skeleton data and the skeleton data estimated from predetermined image data using the calculated weight of each joint.

[0005] Patent Document 3 discloses a technique for calculating the similarity of a human body posture using the feature amounts of a plurality of key points of a human body included in an image and the weight of each key point.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Non-Patent Documents

[0007] [Non-Patent Document 1] Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, P. 7291-7299 [Overview of the project] [Problems that the invention aims to solve]

[0008] In image search processes that use the feature quantities of multiple keypoints of the human body to find similar images, faster search processing is required. For example, when setting search conditions (e.g., similarity threshold, weight of each keypoint, etc.), the operator repeatedly performs the search while adjusting the conditions, and adjusts the conditions appropriately while referring to the search results each time. In situations where such search processing is repeated, if the search process takes a long time, the work efficiency will decrease.

[0009] Although Patent Document 1 and Non-Patent Document 1 disclose a search process for similar images based on the feature quantities of each of several key points of the human body contained in an image, they do not disclose the problem of speeding up the search process or a means to solve it.

[0010] Although Patent Documents 2 and 3 disclose a technique for calculating similarity using the weight of each keypoint, they do not disclose the problem of speeding up the search process or a means to solve it.

[0011] One example of the object of the present invention is to provide an image processing system, apparatus, processing method, and program that solve the problem of achieving faster search processing in the search process for similar images based on the feature quantities of each of multiple key points of the human body contained in an image, in view of the problems described above. [Means for solving the problem]

[0012] According to one aspect of the present invention, A means for acquiring a target image, A skeletal structure detection means that performs a process to detect key points of the human body included in the target image, Based on the detected keypoint, a first matching means extracts a first reference image from among a plurality of reference images whose relationship with the target image satisfies a first extraction condition, A second matching means extracts a second reference image from the first reference image based on the detected keypoint, the second reference image whose relationship with the target image satisfies the second extraction condition, An image processing system having the following characteristics is provided.

[0013] According to one aspect of the present invention, A means for acquiring a target image, A skeletal structure detection means that performs a process to detect key points of the human body included in the target image, Based on the detected keypoint, a first matching means extracts a first reference image from among a plurality of reference images whose relationship with the target image satisfies a first extraction condition, A second matching means extracts a second reference image from the first reference image based on the detected keypoint, the second reference image whose relationship with the target image satisfies the second extraction condition, A device having the following is provided.

[0014] According to one aspect of the present invention, One or more computers, Obtain the target image, The process is performed to detect key points of the human body contained in the aforementioned target image. Based on the detected keypoints, a first reference image is extracted from among multiple reference images whose relationship with the target image satisfies the first extraction condition. An image processing method is provided that extracts a second reference image from the first reference images, the relationship between which and the target image satisfies a second extraction condition, based on the detected key points.

[0015] According to one aspect of the present invention, a computer is caused to function as target image acquisition means for acquiring a target image, skeleton structure detection means for performing a process of detecting key points of a human body included in the target image, first matching means for extracting a first reference image from a plurality of reference images, the relationship between which and the target image satisfies a first extraction condition, based on the detected key points, second matching means for extracting a second reference image from the first reference images, the relationship between which and the target image satisfies a second extraction condition, based on the detected key points, A program is provided.

Advantages of the Invention

[0016] According to one aspect of the present invention, an image processing system, apparatus, processing method, and program are realized that solve the problem of realizing a faster search process in a search process for similar images based on the feature amounts of a plurality of key points of a human body included in an image.

Brief Description of the Drawings

[0017] The above-described object, and other objects, features, and advantages will become more apparent from the following public embodiments and the accompanying drawings below.

[0018] [Figure 1] It is a diagram showing an example of a functional block diagram of an image processing system. [Figure 2] It is a diagram showing a configuration example of an image processing system. [Figure 3] It is a diagram showing an example of a hardware configuration of an image processing system. [Figure 4]This figure shows an example of the skeletal structure of a human body model detected by an image processing system. [Figure 5] This figure shows an example of the skeletal structure of a human body model detected by an image processing system. [Figure 6] This figure shows an example of the skeletal structure of a human body model detected by an image processing system. [Figure 7] This figure shows an example of the skeletal structure of a human body model detected by an image processing system. [Figure 8] This figure shows an example of keypoint features calculated by an image processing system. [Figure 9] This figure shows an example of keypoint features calculated by an image processing system. [Figure 10] This figure shows an example of keypoint features calculated by an image processing system. [Figure 11] This is a schematic diagram illustrating an example of reference image information. [Figure 12] This is a sequence diagram showing an example of the processing flow of an image processing system. [Figure 13] This is a sequence diagram showing an example of the processing flow of an image processing system. [Figure 14] This figure shows an example of a functional block diagram for an image processing system. [Figure 15] This is a sequence diagram showing an example of the processing flow of an image processing system. [Figure 16] This is a sequence diagram showing an example of the processing flow of an image processing system. [Figure 17] This is a flowchart illustrating an example of the processing flow of an image processing system. [Figure 18] This figure shows an example of a settings screen provided by an image processing system. [Figure 19] This figure shows an example of a settings screen provided by an image processing system. [Figure 20] This figure shows an example of a settings screen provided by an image processing system. [Figure 21]This figure shows an example of a settings screen provided by an image processing system. [Figure 22] This figure shows an example of a settings screen provided by an image processing system. [Modes for carrying out the invention]

[0019] Embodiments of the present invention will be described below with reference to the drawings. In all drawings, similar components are denoted by the same reference numerals, and their descriptions are omitted as appropriate.

[0020] <First Embodiment> Figure 1 is a functional block diagram showing an overview of the image processing system 10 according to the first embodiment. The image processing system 10 includes a target image acquisition unit 11, a skeletal structure detection unit 12, a first matching unit 13, and a second matching unit 14.

[0021] The target image acquisition unit 11 acquires the target image. The skeletal structure detection unit 12 performs a process to detect key points of the human body contained in the target image. The first matching unit 13 extracts a first reference image from among multiple reference images based on the detected key points, whose relationship with the target image satisfies the first extraction condition. The second matching unit 14 extracts a second reference image from among the first reference images based on the detected key points, whose relationship with the target image satisfies the second extraction condition.

[0022] The image processing system 10 with this configuration solves the problem of achieving faster search processing in the search process for similar images based on the feature quantities of each of the multiple key points of the human body contained in the image.

[0023] <Second Embodiment> "overview" The image processing system 10 of this embodiment is a more concrete version of the image processing system 10 of the first embodiment. The image processing system 10 of this embodiment performs the process of searching for a desired reference image from among multiple reference images in two steps. That is, in the first step the images are narrowed down to a certain extent, and in the second step the desired reference image is searched for from the narrowed-down reference images.

[0024] As shown in Figure 2, the image processing system 10 of this embodiment includes a server 1 and a client terminal 2. The client terminal 2 is a personal computer, smartphone, tablet, smartwatch, mobile phone, television with internet connectivity, etc., but is not limited to these.

[0025] In this embodiment, Server 1 performs the first step described above. That is, Server 1 extracts a first reference image from among a plurality of reference images whose relationship with the target image satisfies the first extraction condition. Then, Client Terminal 2 performs the second step described above. That is, Client Terminal 2 extracts a second reference image from among the extracted first reference images (narrowed-down reference images) whose relationship with the target image satisfies the second extraction condition. The configuration of the image processing system 10 will be described in detail below.

[0026] "Hardware configuration" Next, an example of the hardware configuration of the image processing system 10 will be described. Each functional unit of the image processing system 10 is realized by any combination of hardware and software, centered around a CPU (Central Processing Unit) of any computer, memory, a program loaded into memory, a storage unit such as a hard disk that stores that program (which can store programs that are pre-installed at the time of shipment, as well as programs downloaded from recording media such as CDs (Compact Discs) or from servers on the Internet), and a network connection interface. It will be understood by those skilled in the art that there are various modifications to the implementation method and the device.

[0027] Figure 3 is a block diagram illustrating the hardware configuration of the image processing system 10. As shown in Figure 3, the image processing system 10 includes a processor 1A, memory 2A, input / output interface 3A, peripheral circuitry 4A, and bus 5A. Peripheral circuitry 4A includes various modules. The image processing system 10 does not necessarily have peripheral circuitry 4A. The image processing system 10 may also be composed of multiple physically and / or logically separated devices (server 1 and client terminal 2). In this case, each of the multiple devices may have the above hardware configuration.

[0028] Bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuits 4A, and input / output interface 3A to send and receive data to and from each other. Processor 1A is a processing unit such as a CPU or GPU (Graphics Processing Unit). Memory 2A is a memory such as RAM (Random Access Memory) or ROM (Read Only Memory). Input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. Input devices include, for example, keyboards, mice, microphones, physical buttons, touch panels, etc. Output devices include, for example, displays, speakers, printers, mailers, etc. Processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0029] "Functional Configuration" Next, the functional configuration of the image processing system 10 of this embodiment will be described in detail. Figure 1 shows an example of a functional block diagram of the image processing system 10. As shown in the figure, the image processing system 10 includes a target image acquisition unit 11, a skeletal structure detection unit 12, a first matching unit 13, and a second matching unit 14. Server 1 includes the skeletal structure detection unit 12 and the first matching unit 13. Client terminal 2 includes the target image acquisition unit 11 and the second matching unit 14.

[0030] Client terminal 2 can communicate with server 1 and perform various processes, for example, via pre-installed dedicated software or applications, or via programs (such as web pages) provided by server 1, thereby realizing the functions of the target image acquisition unit 11 and the second matching unit 14. The configuration of the functional units of the image processing system 10 will be described below.

[0031] The target image acquisition unit 11 acquires a target image. The target image is a still image that is the subject of processing performed by the skeletal structure detection unit 12, the first matching unit 13, and the second matching unit 14.

[0032] The target image acquisition unit 11 may, for example, receive user input specifying one still image stored in a predetermined accessible storage device, and acquire the specified still image as the target image. Alternatively, the target image acquisition unit 11 may acquire a frame image specified by the user from a video as the target image. The video may be one that was filmed in the past or a live video. For example, the target image acquisition unit 11 may receive user input while the video is playing and acquire the frame image that was displayed on the screen at the time the user input was received as the target image. Alternatively, the target image acquisition unit 11 may acquire multiple frame images from the video sequentially as target images at time intervals specified by the user. Note that the process of acquiring target images described here is just an example and is not limited to these.

[0033] As described above, in this embodiment, the client terminal 2 is equipped with a target image acquisition unit 11. The target image acquisition unit 11 of the client terminal 2 receives input specifying a target image as described above via its own input device (touch panel, physical button, keyboard, mouse, microphone, etc.). The target image acquisition unit 11 then stores the acquired target image in a storage device within the client terminal 2. The target image acquisition unit 11 also transmits the acquired target image to the server 1.

[0034] The skeletal structure detection unit 12 performs a process to detect key points of the human body contained in the target image. The skeletal structure detection unit 12 detects N (where N is an integer of 2 or more) key points of the human body contained in the target image. This process by the skeletal structure detection unit 12 is realized using the technology disclosed in Patent Document 1. Details are omitted, but the technology disclosed in Patent Document 1 uses skeletal estimation technology such as OpenPose disclosed in Non-Patent Document 1 to detect the skeletal structure. The skeletal structure detected by this technology consists of "key points," which are characteristic points such as joints, and "bones (bone links)," which indicate the links between key points.

[0035] Figure 4 shows the skeletal structure of the human body model 300 detected by the skeletal structure detection unit 12, and Figures 5 to 7 show examples of skeletal structure detection. The skeletal structure detection unit 12 uses skeletal estimation technology such as OpenPose to detect the skeletal structure of the human body model (2D skeletal model) 300 as shown in Figure 4 from a 2D image. The human body model 300 is a 2D model composed of key points such as the joints of a person and bones connecting each key point.

[0036] The skeletal structure detection unit 12, for example, extracts feature points that could serve as keypoints from an image, and detects N keypoints in the human body by referring to machine learning information obtained from the images of the keypoints. The N keypoints to be detected are predetermined. The number of keypoints to be detected (i.e., the number of N) and which parts of the human body are used as keypoints can vary, and any variation can be adopted.

[0037] For example, as shown in Figure 4, the head A1, neck A2, right shoulder A31, left shoulder A32, right elbow A41, left elbow A42, right hand A51, left hand A52, right hip A61, left hip A62, right knee A71, left knee A72, right foot A81, and left foot A82 are defined as the N key points (N=14) to be detected. Furthermore, in the human body model 300 shown in Figure 4, the following bones are defined as the bones of a person that connect these key points: bone B1 connecting head A1 and neck A2; bones B21 and B22 connecting neck A2 to the right shoulder A31 and left shoulder A32, respectively; bones B31 and B32 connecting right shoulder A31 and left shoulder A32 to right elbow A41 and left elbow A42, respectively; bones B41 and B42 connecting right elbow A41 and left elbow A42 to right hand A51 and left hand A52, respectively; bones B51 and B52 connecting neck A2 to right hip A61 and left hip A62, respectively; bones B61 and B62 connecting right hip A61 and left hip A62 to right knee A71 and left knee A72, respectively; and bones B71 and B72 connecting right knee A71 and left knee A72 to right foot A81 and left foot A82, respectively.

[0038] Figure 5 shows an example of detecting a person in an upright position. In Figure 5, an upright person is imaged from the front, and bones B1, B51 and B52, B61 and B62, and B71 and B72 are detected without overlapping when viewed from the front, and bones B61 and B71 of the right leg are slightly more bent than bones B62 and B72 of the left leg.

[0039] Figure 6 shows an example of detecting a person in a crouching position. In Figure 6, the crouching person is imaged from the right side, and bones B1, B51 and B52, B61 and B62, B71 and B72 are detected from the right side. Bones B61 and B71 of the right leg and bones B62 and B72 of the left leg are significantly bent and overlapping.

[0040] Figure 7 shows an example of detecting a person lying down. In Figure 7, the person lying down is imaged from the left front, and bones B1, B51 and B52, B61 and B62, B71 and B72 are detected from the left front, with bones B61 and B71 of the right leg and bones B62 and B72 of the left leg being bent and overlapping.

[0041] The first matching unit 13 extracts a first reference image from a plurality of pre-registered reference images based on the key points detected by the skeletal structure detection unit 12, the first reference image whose relationship with the target image satisfies the first extraction condition.

[0042] The first extraction criterion is that the "similarity of human body postures contained in the image" calculated by the "first calculation method" is "greater than or equal to the first reference value." In other words, the first matching unit 13 calculates the similarity between the human body postures contained in the target image and the human body postures contained in each of the reference images using the first calculation method. The first matching unit 13 then extracts the reference images whose calculated similarity is greater than or equal to the first reference value as the first reference image.

[0043] The second matching unit 14 extracts a second reference image from the first reference images extracted by the first matching unit 13, based on the key points detected by the skeletal structure detection unit 12, such that the relationship between the second reference image and the target image satisfies the second extraction condition. That is, the second matching unit 14 performs a comparison with the target image using the reference images (first reference images) narrowed down by the first matching unit 13 as the comparison target, and extracts the second reference image from the first reference images.

[0044] The second extraction criterion is that the "similarity of human body postures contained in the image" calculated by the "second calculation method" is "greater than or equal to the second standard value". In other words, the second matching unit 14 calculates the similarity between the human body postures contained in the target image and the human body postures contained in each of the first reference images using the second calculation method. The second matching unit 14 then extracts the first reference images whose calculated similarity is greater than or equal to the second standard value as the second reference images.

[0045] The first calculation method and the second calculation method may be different from each other or the same. For example, the first calculation method and the second calculation method may differ in at least one of the number of keypoints and the types of keypoints referenced when calculating the similarity of human body postures.

[0046] Furthermore, the first and second calculation methods may have different weight settings for each key point referenced when calculating the similarity of human postures. For example, the first calculation method may calculate the similarity of human postures by setting the same weight for all key points, while the second calculation method may calculate the similarity of human postures based on the weight set for each key point. Also, the first and second calculation methods may have different weights for each key point.

[0047] Furthermore, the first and second reference values ​​can be set separately and independently. Therefore, the first and second reference values ​​can be set to the same value or to different values.

[0048] Furthermore, the first and second extraction conditions may include other conditions that are different from each other. For example, either the first or second extraction condition may be: • A certain number or more of the keypoints used to calculate the similarity of human body postures have been detected, and • The specified keypoint (e.g., head) among the keypoints referenced when calculating the similarity of human body postures must be detected. It may include at least one of the following. The "predetermined number (minimum detection point)" and "predetermined key points (essential detection key points)" of the said condition may be predetermined or can be set by the user.

[0049] For example, the first extraction criterion may include the condition, or the second extraction criterion may include the condition.

[0050] In addition, both the first and second extraction conditions may include the condition in question. In that case, their contents may differ from each other.

[0051] For example, if both the first and second extraction conditions include "a predetermined number or more of the keypoints referenced when calculating the similarity of human body postures have been detected," then the predetermined number can be set separately and independently. In this case, the predetermined number for the first extraction condition and the predetermined number for the second extraction condition can be the same value, or they can be different values.

[0052] Furthermore, if both the first and second extraction conditions include "a predetermined key point among the key points referenced when calculating the similarity of human body postures has been detected," then the predetermined key point may be set separately and independently. In this case, the type and number of the predetermined key point in the first extraction condition and the predetermined key point in the second extraction condition may be the same or different.

[0053] Furthermore, the second extraction criterion may allow at least one of the above-mentioned items (number of keypoints referenced when calculating the similarity of human body posture, type of keypoint, weight of each keypoint, minimum detection point, and required detection keypoint) to be changed by user input. The first extraction criterion may have the above-mentioned items fixed.

[0054] Here, specific examples of the first and second extraction conditions will be described. Note that these examples are merely illustrative, and the first and second extraction conditions in this embodiment are not limited to these.

[0055] The first extraction criterion is that "the similarity of human body postures calculated based on all N key points is equal to or greater than the first criterion value." The similarity for the first extraction criterion is calculated using the same weight for all N key points.

[0056] The second extraction criterion is that "the similarity of human body postures calculated based on a portion of the N keypoints is equal to or greater than the second criterion value." This similarity for the second extraction criterion is calculated based on the weights assigned to each keypoint.

[0057] The second extraction criterion allows the user to change the number of keypoints referenced when calculating the similarity of human body postures, the types of keypoints, and the weight of each keypoint. On the other hand, the first extraction criterion has a fixed number of keypoints referenced when calculating the similarity of human body postures, the types of keypoints, and the weight of each keypoint.

[0058] Furthermore, the second extraction criterion allows the second criterion value to be changed by user input. The first extraction criterion may allow the first criterion value to be changed by user input, or it may be a fixed value.

[0059] Furthermore, the second extraction criterion is: • A certain number or more of the keypoints used to calculate the similarity of human body postures have been detected, and • The specified keypoints among the keypoints referenced when calculating the similarity of human body postures have been detected. It includes at least one of the following. The "predetermined number" and "predetermined key points" of the said condition may be predetermined or may be changed by user input. The first extraction condition does not include the said condition.

[0060] Here, we will explain an example of a process that calculates the similarity between the human body posture detected from the target image and the human body posture shown in a pre-registered reference image, based on the key points detected by the skeletal structure detection unit 12.

[0061] There are various methods for calculating the similarity of human body postures, and any technology can be employed. For example, the technology disclosed in Patent Document 1 may be used. An example is described below, but is not limited to this example.

[0062] For example, the similarity between two human poses can be calculated by determining the feature quantities of the skeletal structure indicated by the detected keypoints, and then calculating the similarity between the feature quantities of the human skeletal structure detected from the target image and the feature quantities of the human skeletal structure indicated from the reference image.

[0063] Skeletal structure features represent the characteristics of a person's skeleton and serve as elements for classifying a person's posture based on their skeletal structure. Typically, these features include multiple parameters. The features referenced in calculating similarity may be the overall features of the skeletal structure, features of a part of the skeletal structure, or multiple features representing different parts of the skeletal structure. The method for calculating the features can be any method, such as machine learning or normalization, and normalization may involve finding minimum or maximum values. As an example, features may include features obtained by machine learning of the skeletal structure, the size of the skeletal structure from head to toe in an image, the relative positional relationship of multiple keypoints in the vertical direction of the skeletal region containing the skeletal structure in the image, and the relative positional relationship of multiple keypoints in the horizontal direction of the said skeletal region. The size of the skeletal structure refers to the height or area in the vertical direction of the skeletal region containing the skeletal structure in the image. The vertical direction (height direction or vertical direction) is the vertical direction (Y-axis direction) in the image, for example, the direction perpendicular to the ground (reference plane). Furthermore, the left-right direction (horizontal direction) refers to the left-right direction (X-axis direction) in the image, for example, the direction parallel to the ground.

[0064] Furthermore, in order to perform the search desired by the user, it is preferable to use features that are robust to the search process. For example, if the user desires a search that is independent of the orientation or body shape of a person, features that are robust to the orientation and body shape of a person may be used. Features that are independent of the orientation and body shape of a person can be obtained by learning the skeletons of people facing in various directions while maintaining the same posture, or by extracting features only in the vertical direction of the skeleton. An example of a process for calculating features of a skeletal structure is disclosed in Patent Document 1.

[0065] Figure 8 shows examples of feature quantities for each of several keypoints. The set of feature quantities for multiple keypoints constitutes the feature quantities of the skeletal structure. Note that the keypoint feature quantities exemplified here are merely examples and are not limited to these.

[0066] In this example, the keypoint features represent the relative positions of multiple keypoints in the vertical direction of the skeletal region containing the skeletal structure in the image. Since keypoint A2 on the neck is used as the reference point, the feature value for keypoint A2 is 0.0, and the feature values ​​for keypoints A31 on the right shoulder and A32 on the left shoulder, which are at the same height as the neck, are also 0.0. The feature value for keypoint A1 on the head, which is higher than the neck, is -0.2. The feature values ​​for keypoints A51 on the right hand and A52 on the left hand, which are lower than the neck, are 0.4, and the feature values ​​for keypoints A81 on the right foot and A82 on the left foot are 0.9. If the person raises their left hand from this state, the left hand will be higher than the reference point as shown in Figure 9, so the feature value for keypoint A52 on the left hand will be -0.4. On the other hand, since normalization is performed using only the Y-axis coordinate, the feature values ​​do not change even if the width of the skeletal structure changes compared to Figure 8, as shown in Figure 10. In other words, the feature quantities (normalized values) in this example represent the features of the skeletal structure (keypoints) in the height direction (Y direction) and are not affected by changes in the lateral direction (X direction) of the skeletal structure.

[0067] There are various ways to calculate the similarity of postures represented by such features. For example, the similarity of the features can be calculated for each keypoint, and then the similarity of postures can be calculated based on the similarity of the features of multiple keypoints. For example, the mean, maximum, minimum, mode, median, weighted mean, weighted sum, etc., of the similarity of the features of multiple keypoints can be calculated as the similarity of postures. When calculating the weighted mean or weighted sum, the weights of each keypoint can be set by the user or predetermined.

[0068] Here, Figure 11 illustrates the reference image information that is pre-registered in the image processing system 10. In this embodiment, the reference image and reference image information are registered in the server 1. The reference image information shown in Figure 11 is such that the reference image identification information, data name, and feature quantities are linked to each other.

[0069] Reference image identification information is information that identifies multiple reference images from one another.

[0070] The data name is information attached to each reference image. Multiple reference images can be given the same data name. Conversely, a single reference image can be given multiple data names. The data name can be based on the content of the image (e.g., the posture of the person, how the object is depicted). The illustrated "Wheelchair, Overhead View" is attached to a reference image that includes a person in a wheelchair and is taken from above, looking down at that person. In addition, a reference image that includes a person holding a mobile phone in their right hand and talking on the phone may be given the data name "Mobile Phone, Right Hand, Overhead View". For example, a reference image that includes a person in a wheelchair holding a mobile phone in their right hand and talking on the phone may be given the data names "Wheelchair, Overhead View" and "Mobile Phone, Right Hand, Overhead View".

[0071] The features are the pose characteristics of the human body contained in each reference image (e.g., a set of features for each keypoint).

[0072] Furthermore, in addition to user input specifying the target image, client terminal 2 may also accept user input specifying the data name. Client terminal 2 may then send the user input specifying the data name in addition to the specified target image to server 1. In this case, the first matching unit 13 may extract reference images associated with the specified data name from the reference images, and then extract a first reference image that satisfies the first extraction condition from the extracted reference images. With this configuration, the reference images to be searched can be narrowed down by data name, and faster search processing can be achieved.

[0073] The content of the "user input specifying data names" described above can be configured in various ways. For example, client terminal 2 may accept input that directly specifies one or more data names as the "user input specifying data names." Alternatively, server 11 may create groups by grouping multiple data names that have commonalities and manage each group by associating a label name with it. For example, a group of data names such as "mobile phone, right hand, overhead view" and "mobile phone, left hand, overhead view" may be associated with the label name "mobile phone usage." Then, client terminal 2 may accept input that selects a label name as the "user input specifying data names." In this case, the data names associated with the group of selected label names will be specified.

[0074] Here, using the sequence diagram in Figure 12, we will explain an example of the processing flow of the image processing system 10, which consists of server 1 and client terminal 2.

[0075] First, client terminal 2 receives user input specifying the target image (S10). Next, client terminal 2 sends the specified target image to server 1 (S11).

[0076] Server 1 performs a process to detect key points of the human body contained in the target image, and then, based on the detected key points, extracts a first reference image from among multiple reference images whose relationship with the target image satisfies the first extraction condition (S12). Next, Server 1 sends the first reference image, information on the key points of the human body detected from each of the first reference images (e.g., feature quantities, etc.) (see Figure 11), and information on the key points of the human body detected from the target image (e.g., feature quantities, etc.) to the client terminal 2 (S13).

[0077] Based on the information regarding key points of the human body detected from each of the received first reference images and the information regarding key points of the human body detected from the target image, the client terminal 2 extracts a second reference image from the received first reference images whose relationship with the target image specified in S10 satisfies the second extraction condition (S14). Then, the client terminal 2 displays the extracted second reference image (S15). This display is achieved by displaying on a screen, projecting an image using a projection device, etc.

[0078] Furthermore, client terminal 2 can store the data (images and information) received in S13 in its own storage device and repeatedly perform the processes in S14 and S15 using that data.

[0079] For example, a user may input to client terminal 2 to change the second extraction criteria. Then, client terminal 2 may extract a second reference image from the received first reference image whose relationship with the target image specified in S10 satisfies the changed second extraction criteria (S14), and display the extracted second reference image (S15). This process will be described in detail in the fifth embodiment.

[0080] In addition, multiple second extraction conditions may be set in advance. The client terminal 2 then extracts second reference images from the received first reference images that satisfy each of the multiple second extraction conditions in relation to the target image specified in S10 (S14), and may display the second reference images extracted based on each of the multiple second extraction conditions separately (S15).

[0081] As described above, when performing extraction based on the second extraction criterion multiple times, if both the extraction based on the first extraction criterion and the extraction based on the second extraction criterion are performed each time, the processing load on the computer increases and the processing speed slows down. As in this example, by dividing the process into a first extraction process (S12) and a second extraction process (S14), and configuring the system to allow multiple second extraction processes to correspond to one first extraction process, the processing load on the computer is reduced and the processing speed is increased.

[0082] <Effects and Effects> According to the image processing system 10 of this embodiment, the extraction process (search process) of similar images based on the feature quantities of each of the multiple key points of the human body contained in the image can be performed in two steps. That is, in the first step, the reference images to be searched are narrowed down, and in the second step, images similar to the target image can be searched from among the narrowed-down reference images. By performing the search process for similar images based on the feature quantities of each of the multiple key points of the human body contained in the image in two steps in this way, a faster search process can be achieved.

[0083] For example, if the process is divided into two steps, the result of the first step can be stored, allowing the second step to be performed multiple times using that result. In other words, the second step can be performed multiple times in response to one first step. In contrast, if the extraction process is not divided into two steps, the entire extraction process must be executed each time. Compared to such comparative examples, the image processing system 10 of this embodiment reduces the processing load on the computer and increases the processing speed.

[0084] Furthermore, in this embodiment, in the first extraction condition of the first step, the number of key points referenced when calculating the similarity of human body postures, the type of key points, and the weight of each key point are fixed, while in the second extraction condition of the second step, these items can be changed by user input. Incidentally, as a technique for speeding up searches, there is a technique that stores data in a database while clustering it and narrows the search to clusters similar to the query during a search. However, when searching while changing the search conditions each time, the similarity between data changes depending on the search conditions, so the above technique cannot be used and the search becomes slow. To address this problem, by fixing the first extraction condition of the first step and making the second extraction condition of the second step variable, the first extraction condition (the process of narrowing down from a large amount of data) can be performed quickly, and the search conditions (second extraction condition) can also be changed in the second step, so that the desired search can be performed quickly.

[0085] <Third Embodiment> The image processing system 10 of this embodiment also has a server 1 and a client terminal 2, as shown in Figure 2.

[0086] In the second embodiment, the process of extracting similar images (search process) based on the feature quantities of each of the multiple keypoints of the human body contained in the image was divided into two steps, with Server 1 performing the first step and Client Terminal 2 performing the second step. In contrast, in this embodiment, Server 1 performs both the first and second steps. This will be explained in detail below.

[0087] Figure 1 shows an example of a functional block diagram of the image processing system 10. As shown in the figure, the image processing system 10 includes a target image acquisition unit 11, a skeletal structure detection unit 12, a first matching unit 13, and a second matching unit 14. Server 1 comprises the skeletal structure detection unit 12, the first matching unit 13, and the second matching unit 14. Client terminal 2 comprises the target image acquisition unit 11. The configuration of each functional unit is as described in the second embodiment.

[0088] Here, using the sequence diagram in Figure 13, we will explain an example of the processing flow of an image processing system 10, which consists of a server 1 and a client terminal 2.

[0089] First, client terminal 2 receives user input specifying the target image (S20). Next, client terminal 2 sends the specified target image to server 1 (S21).

[0090] Server 1 performs a process to detect key points of the human body contained in the target image, and then, based on the detected key points, extracts a first reference image from among multiple reference images whose relationship with the target image satisfies the first extraction condition (S22). Next, based on the key point detection result in S22, Server 1 extracts a second reference image from the first reference image extracted in S22 whose relationship with the target image received in S21 satisfies the second extraction condition (S23). Then, Server 1 transmits the extracted second reference image to client terminal 2 (S24).

[0091] Subsequently, client terminal 2 displays the received second reference image (S25). This display is achieved by displaying it on a screen, projecting an image using a projection device, etc.

[0092] Furthermore, Server 1 stores the data (images and information) obtained in the S22 process in its own storage device and can repeatedly perform the S23 and S24 processes using this data. When Client Terminal 2 receives a new second reference image (S24), it can display the newly received second reference image.

[0093] For example, a user may input to client terminal 2 to change the second extraction conditions. Client terminal 2 may then send the changed second extraction conditions to server 1. Server 1 may then extract a second reference image from the first reference images extracted in S22 that satisfies the changed second extraction conditions in relation to the target image received in S21 (S23), and send the extracted second reference image to client terminal 2 (S24). This process will be described in detail in the fifth embodiment.

[0094] In addition, multiple second extraction conditions may be set in advance. Then, from the first reference images extracted in S22, server 1 may extract second reference images whose relationship with the target image specified in S20 satisfies each of the multiple second extraction conditions (S23), and transmit the second reference images extracted based on each of the multiple second extraction conditions to client terminal 2 in a manner that allows them to be identified by each other (S24).

[0095] As described above, when performing extraction based on the second extraction criterion multiple times, if both the extraction based on the first extraction criterion and the extraction based on the second extraction criterion are performed each time, the processing load on the computer increases and the processing speed slows down. As in this example, by dividing the process into a first extraction process (S22) and a second extraction process (S23), and configuring the system to perform the second extraction process multiple times in response to one first extraction process, the processing load on the computer is reduced and the processing speed is increased.

[0096] The other configurations of the image processing system 10 in this embodiment are the same as those of the image processing system 10 in the first and second embodiments.

[0097] The image processing system 10 of this embodiment achieves the same effects and advantages as the image processing system 10 of the first and second embodiments. Furthermore, the image processing system 10 of this embodiment reduces the processing load on the client terminal 2.

[0098] <Fourth Embodiment> The image processing system 10 of this embodiment is composed of a single physical and / or logical device. An example of a functional block diagram of the image processing system 10 of this embodiment is shown in Figure 1. In this embodiment, a single physical and / or logical device comprises a target image acquisition unit 11, a skeletal structure detection unit 12, a first matching unit 13, and a second matching unit 14, and performs the processing described in the first to third embodiments.

[0099] The other configurations of the image processing system 10 in this embodiment are the same as those of the image processing system 10 in the first to third embodiments. Even when the image processing system 10 in this embodiment is configured as described above, the same effects and advantages as those of the image processing system 10 in the first to third embodiments are achieved.

[0100] <Fifth Embodiment> The image processing system 10 of this embodiment has a function to change the second extraction conditions. This will be described in detail below.

[0101] Figure 14 shows an example of a functional block diagram of the image processing system 10. As shown in the figure, the image processing system 10 includes a target image acquisition unit 11, a skeletal structure detection unit 12, a first matching unit 13, a second matching unit 14, a display control unit 15, and a change acceptance unit 16.

[0102] The display control unit 15 causes the second reference image extracted by the second matching unit 14 to be displayed on the display device. For example, if the image processing system 10 is composed of a server 1 and a client terminal 2, as in the second and third embodiments, the display control unit 15 causes the second reference image to be displayed on the display device (display, projection device, etc.) of the client terminal 2. Also, if the image processing system 10 is composed of a single device physically and / or logically, as in the fourth embodiment, the display control unit 15 causes the second reference image to be displayed on the display device (display, projection device, etc.) of that device.

[0103] The change reception unit 16 accepts input to change the second extraction conditions. The change reception unit 16 may accept input to change, for example, at least one of the second criterion value defined in the second extraction conditions, the number of key points referenced when calculating the similarity of human body postures, the types of key points referenced when calculating the similarity of human body postures, the weight of each key point referenced when calculating the similarity of human body postures, the minimum detection point, and the required detection key points.

[0104] The minimum number of detection points is the predetermined number in the condition "a predetermined number or more of the key points referenced when calculating the similarity of human body postures are detected," which can be included in the second extraction condition described in the second embodiment.

[0105] The essential detection keypoint is a predetermined keypoint in the condition "a predetermined keypoint among the keypoints referenced when calculating the similarity of human body postures has been detected," which can be included in the second extraction condition described in the second embodiment.

[0106] When the image processing system 10 is composed of a server 1 and a client terminal 2, as in the second and third embodiments, the change reception unit 16 can receive input to change the second extraction conditions via the input device (touch panel, physical buttons, keyboard, mouse, microphone, etc.) of the client terminal 2. Also, when the image processing system 10 is composed of a single physical and / or logical device, as in the fourth embodiment, the change reception unit 16 can receive input to change the second extraction conditions via the input device (touch panel, physical buttons, keyboard, mouse, microphone, etc.) of that device.

[0107] Furthermore, in response to the input received by the change reception unit 16 to change the second extraction conditions, the second matching unit 14 newly extracts a second reference image from the first reference image whose relationship with the target image satisfies the changed second extraction conditions. Then, the display control unit 15 changes the content to be displayed on the display device from the second reference image that satisfies the second extraction conditions before the change to the second reference image that satisfies the changed second extraction conditions.

[0108] Next, an example of the processing flow of an image processing system 10, which consists of a server 1 and a client terminal 2, will be explained using the sequence diagram in Figure 15. In this example, the server 1 has a first matching unit 13, and the client terminal 2 has a second matching unit 14.

[0109] First, client terminal 2 receives user input specifying the target image (S30). Next, client terminal 2 sends the specified target image to server 1 (S31).

[0110] Server 1 processes the key points of the human body contained in the target image, and then, based on the detected key points, extracts a first reference image from among multiple reference images whose relationship with the target image satisfies the first extraction condition (S32). Next, Server 1 sends the first reference image, information on the key points of the human body detected from each of the first reference images (e.g., feature quantities, etc.) (see Figure 11), and information on the key points of the human body detected from the target image (e.g., feature quantities, etc.) to the client terminal 2 (S33).

[0111] Client terminal 2 stores the data (images and information) received in S33 in its own storage device, and based on this data, extracts a second reference image from the first reference images received in S33 that satisfies the second extraction condition in relation to the target image specified in S30 (S34). Then, client terminal 2 displays the extracted second reference image (S35). This display is achieved by displaying on a screen, projecting an image using a projection device, etc.

[0112] Subsequently, the user makes an input to change the second extraction criteria while referring to the search results (second reference image) displayed on client terminal 2. Client terminal 2 accepts the input to change the second extraction criteria (S36). Then, in response to the acceptance of the input, client terminal 2 extracts a new second reference image from the first reference images received in S33 that satisfies the changed second extraction criteria in relation to the target image specified in S30 (S37). Client terminal 2 performs the extraction process in S37 based on the data received in S33 and stored in its own storage device. Next, client terminal 2 changes the content displayed on the display device from the second reference image that satisfies the second extraction criteria before the change to the second reference image that satisfies the changed second extraction criteria (S38).

[0113] Client terminal 2 can repeatedly perform the processes S36 through S38.

[0114] Next, another example of the processing flow of the image processing system 10, which consists of a server 1 and a client terminal 2, will be explained using the sequence diagram in Figure 16. In this example, the server 1 has a first matching unit 13 and a second matching unit 14.

[0115] First, client terminal 2 receives user input specifying the target image (S40). Next, client terminal 2 sends the specified target image to server 1 (S41).

[0116] Server 1 performs a process to detect key points of the human body contained in the target image, and then, based on the detected key points, extracts a first reference image from among multiple reference images whose relationship with the target image satisfies the first extraction condition (S42). Server 1 then stores the data (images and information) obtained in the process of S42 in the storage device of its own device.

[0117] Next, based on the keypoint detection result in S42, Server 1 extracts a second reference image from the first reference image extracted in S42 that satisfies the second extraction condition in relation to the target image received in S41 (S43). Then, Server 1 transmits the extracted second reference image to Client Terminal 2 (S44).

[0118] Client terminal 2 displays the received second reference image (S45). This display is achieved by displaying it on a screen, projecting an image using a projection device, etc.

[0119] Subsequently, the user makes input to modify the second extraction criteria while referring to the search results (second reference image) displayed on client terminal 2. Client terminal 2 accepts the input to modify the second extraction criteria (S46). Then, client terminal 2 sends the modified second extraction criteria to server 1 (S47).

[0120] Next, Server 1 extracts a second reference image from the first reference image extracted in S42, the second reference image whose relationship with the target image specified in S40 satisfies the modified second extraction conditions (S48). Server 1 performs the extraction process in S48 based on the data obtained in the process of S42 and stored in the storage device of its own device. Next, Server 1 transmits the second reference image that satisfies the modified second extraction conditions to Client Terminal 2 (S49).

[0121] Then, client terminal 2 changes the content displayed on the display device from the second reference image that satisfies the second extraction conditions before the change to the second reference image that satisfies the second extraction conditions after the change (S50).

[0122] Server 1 and client terminal 2 can repeatedly perform the processes S46 to S50.

[0123] Next, using the flowchart in Figure 17, another example of the processing flow of an image processing system 10, which is physically and / or logically composed of a single device, will be explained.

[0124] First, the image processing system 10 receives user input specifying the target image (S60). Next, the image processing system 10 performs a process to detect key points of the human body contained in the target image, and then, based on the detected key points, extracts a first reference image from among multiple reference images whose relationship with the target image satisfies the first extraction condition (S61). Then, the image processing system 10 stores the data (images and information) obtained in the process of S61 in the storage device of its own device.

[0125] Next, based on the keypoint detection result in S61, the image processing system 10 extracts a second reference image from the first reference image extracted in S61, such that the relationship between the second reference image and the target image specified in S60 satisfies the second extraction condition (S62). Then, the image processing system 10 displays the extracted second reference image (S63). This display is achieved by displaying it on a screen, projecting an image using a projection device, etc.

[0126] Subsequently, the user makes input to change the second extraction conditions while referring to the search results (second reference image) displayed on the image processing system 10. The image processing system 10 accepts the input to change the second extraction conditions (S64).

[0127] Next, the image processing system 10 extracts a second reference image from the first reference image extracted in S61, the second reference image whose relationship with the target image specified in S60 satisfies the modified second extraction conditions (S65). The image processing system 10 performs the extraction process in S65 based on the data obtained in the process of S61 and stored in the storage device of its own device. Next, the image processing system 10 changes the content to be displayed on the display device from the second reference image that satisfies the modified second extraction conditions to the second reference image that satisfies the modified second extraction conditions (S66).

[0128] The image processing system 10 can repeatedly perform the processes S64 to S66.

[0129] The other configurations of the image processing system 10 in this embodiment are the same as those of the image processing system 10 in the first to fourth embodiments. Even with the image processing system 10 configured as in this embodiment, the same effects and advantages as those of the image processing system 10 in the first to fourth embodiments are achieved.

[0130] Furthermore, according to the image processing system 10 of this embodiment, the search process can be sped up when repeatedly performing the search process and changing the second extraction conditions while checking the search results.

[0131] <Sixth Embodiment> The image processing system 10 of this embodiment accepts changes to the second extraction conditions via a distinctive UI (user interface) screen. This will be described in detail below.

[0132] Figure 14 shows an example of a functional block diagram of the image processing system 10. As shown in the figure, the image processing system 10 includes a target image acquisition unit 11, a skeletal structure detection unit 12, a first matching unit 13, a second matching unit 14, a display control unit 15, and a change acceptance unit 16.

[0133] The change reception unit 16 accepts input to change the second extraction conditions via a distinctive settings screen (UI screen). When the image processing system 10 is composed of a server 1 and a client terminal 2, as in the second and third embodiments, the client terminal 2 displays the settings screen. When the image processing system 10 is composed of a single device, physically and / or logically, as in the fourth embodiment, that device displays the settings screen.

[0134] Figure 18 shows an example of a settings screen. In the UI screen shown, the leftmost area allows you to select "Still Image," "Capture Image," "Live," and "Settings." Selecting "Settings" in this area displays the settings screen shown in the figure.

[0135] In the illustrated settings screen, a video is displayed in region M. This video may be a live image currently being captured by any camera, or it may be a video that was previously captured and saved.

[0136] The "Rotation Angle" is a UI component used to rotate the image in area M. For example, 0 degrees, 90 degrees, 180 degrees, and 270 degrees are selectable, and the image displayed in area M rotates by the selected angle. For example, if "90 degrees" is selected in the illustrated state, the image displayed in area M will rotate 90 degrees clockwise.

[0137] The "detection threshold" is the first reference value for the first extraction condition.

[0138] The "label name" is as described in the second embodiment. The user can select the label name via the UI component.

[0139] The "border color," "initial selection," "select all check items," and "display unused poses" will be explained later.

[0140] Below the area where the above items are displayed, a UI component is shown to accept input for changing the second extraction condition. Depending on the selected label name, the current settings corresponding to one or more data names associated with that label name group are displayed. The user can change these settings to their desired content. For example, as shown in the illustration, if the user selects "wheelchair" as the label name, the current settings corresponding to "wheelchair: overhead view," which is a data name associated with that label name group, are displayed. Also, although not shown in the illustration, if the user selects "mobile phone use" as the label name, for example, the current settings corresponding to each data name associated with that label name group, such as "mobile phone, right hand, overhead view," "mobile phone, left hand, overhead view," etc., are displayed. In other words, for each data name such as "mobile phone, right hand, overhead view," "mobile phone, left hand, overhead view," etc., a human body model, second threshold, minimum detection point, etc., as shown in the illustration, are displayed.

[0141] Region R displays a human body model composed of N keypoints. The keypoints referenced and those not referenced are clearly indicated when calculating the similarity of human body postures. In the example shown, keypoint K1, indicated by a white circle, is referenced when calculating the similarity of human body postures, while keypoint K2, indicated by a black circle, is not referenced.

[0142] The user can select one of N keypoints and change its weight. In the example shown, the keypoint enclosed by the mark Q has been selected by the user. The name of this keypoint is "joint3". Depending on the selection of a keypoint, the name of the selected keypoint and a UI component to change its weight will be displayed on the screen, as shown in the illustration. In the example shown, the weight of joint3 is "0.0". This means that this keypoint will not be referenced when calculating the similarity of human body postures.

[0143] The user can change the weight of the selected keypoint, for example, by manipulating the slider bar shown in the diagram or by directly entering a numerical value. For example, the weight of joint3 can be changed from "0" to "a number other than 0". In response to this change, joint3 switches from being a keypoint that is not referenced when calculating the similarity of human body posture to a keypoint that is referenced. Accordingly, the display of joint3 in region R changes from a black circle to a white circle.

[0144] Additionally, you can select a keypoint (keypoint K1, indicated by a white circle) to reference when calculating the similarity of human body postures, and set its weight to "0". Depending on this change, the keypoint will switch from being referenced to not being referenced when calculating the similarity of human body postures. Accordingly, the display of that keypoint in region R will change from a white circle to a black circle.

[0145] Additionally, you can select keypoints (keypoint K1, indicated by a white circle) to reference when calculating the similarity of human body postures, and change the weight of these keypoints to a range other than "0".

[0146] "ID19: Wheelchair - Overhead View" is the "data name" described in the second embodiment. In this embodiment, a second extraction condition is set for each data name. By referring to the display of data names such as "ID19: Wheelchair - Overhead View," the user can recognize which data name corresponds to which second extraction condition is displayed and currently being set.

[0147] The "second threshold" is the second reference value for the second extraction criterion.

[0148] The "minimum detection point" is as described in the fifth embodiment. In this example, the second extraction condition includes "a predetermined number or more of the key points referenced when calculating the similarity of human postures are detected." In the illustrated example, six key points (key point K1 indicated by white circles) are the "key points referenced when calculating the similarity of human postures," and the minimum detection point is "2." In this case, the condition for satisfying the second extraction condition is that two or more of these six key points are detected.

[0149] The change reception unit 16 includes a human body model (a human body model displayed in region R) composed of such multiple key points, and can receive input to change the second extraction conditions via a setting screen that accepts input to select the key points to be set on the human body model and input to change the weight of the selected key points (key points surrounded by mark Q).

[0150] Furthermore, the change reception unit 16 can receive input to change the second extraction condition via a settings screen that highlights the selected key points (highlighted with Mark Q) in the human body model described above.

[0151] Furthermore, the change reception unit 16 can receive input to change the second extraction conditions via a setting screen that displays, in the above-mentioned human body model, key points (key point K1 indicated by a white circle) and other key points (key point K2 indicated by a black circle) in different ways.

[0152] Note that pressing the "Save Settings" button in the upper left corner of the screen in Figure 18 will save the settings at that time. The settings to be saved are the second extraction criteria, but the first reference value of the first extraction criteria may also be saved in this operation.

[0153] When the "Analyze" button in the upper left corner of the screen is pressed, the target image acquisition unit 11 acquires the frame image that was displayed in region M at that time as the target image. Subsequently, the skeletal structure detection unit 12, the first matching unit 13, and the second matching unit 14 perform the processing described in the first to fifth embodiments on the target image. Then, the display control unit 15 displays the second reference image extracted by the second matching unit 14, as shown in Figure 19. The 10 images displayed in the "Matching Results" column in the figure are the second reference images extracted by the second matching unit 14.

[0154] Furthermore, as shown in the figure, the display control unit 15 can switch the image displayed in area M from the initial moving image to the specified target image (still image) in response to the selection of a target image (pressing the "Analyze" button in the upper left corner of the screen in Figure 18). The display control unit 15 can then superimpose a frame W onto the target image. The frame W is displayed so as to surround the person in the target image who satisfies the second extraction condition corresponding to the data name associated with the selected group of label names. The "frame line color" that can be set on this screen is the color of this frame W.

[0155] The display control unit 15 may further superimpose the key points of the human body detected within the target image onto the target image. This superimposition is realized based on the detection results by the skeletal structure detection unit 12. In this superimposition, all key points may be displayed in the same display manner or in different display manners. For example, the key points of the right half of the body and the key points of the left half of the body may be displayed in different display manners, or the key points of the upper body and the key points of the lower body may be displayed in different display manners. In addition, key points that are referenced when calculating the similarity of the human body posture may be highlighted. Furthermore, when one key point is selected in region R, the selected key point may be highlighted in the human body model superimposed on the target image.

[0156] The user can input a change to the second extraction condition while referring to the matching results. For example, suppose the user changes the minimum detection point to "3" from the state shown in Figure 19. Then, in response to this change in the second extraction condition, the second matching unit 14 extracts a new second reference image from the first reference image whose relationship with the target image satisfies the changed second extraction condition. The display control unit 15 then changes the content displayed in the matching results column from the second reference image that satisfies the second extraction condition before the change to the second reference image that satisfies the second extraction condition after the change, as shown in Figure 20. Figures 19 and 20 show how changing the minimum detection point from 2 to 3 changes the number of extracted second reference images from 10 to 6.

[0157] As shown in Figure 21, checking "Show unused postures" will also display a second set of extraction criteria (saved in the settings) corresponding to data names associated with groups of label names other than the selected label name. In Figure 21, "Wheelchair" is selected for "Label Name," but because "Show unused postures" is checked, a second set of extraction criteria corresponding to data names not associated with the selected label name group, such as "Mobile phone, right hand," is also displayed.

[0158] As shown in Figure 22, the user can specify whether or not to reference each of the multiple second extraction conditions corresponding to each of the multiple data names in the second reference image extraction process by the second matching unit 14. In Figure 22, checkboxes (checkboxes next to each region R) are displayed corresponding to each of the multiple second extraction conditions corresponding to each of the multiple data names. If these checkboxes are operated individually and "Request all checked items" is checked, all of the second extraction conditions with checked checkboxes will be referenced in the second reference image extraction process by the second matching unit 14. The second matching unit 14 will then extract a second reference image that satisfies all of the second extraction conditions with checked checkboxes. If "Request all checked items" is not checked, the second matching unit 14 will extract a second reference image that satisfies at least one of the second extraction conditions with checked checkboxes.

[0159] Here, we will briefly explain the process that is executed when the "Still Image," "Capture," or "Live" option is selected in the leftmost area of ​​the UI screen shown in Figures 18 to 22.

[0160] When "Still Image" is selected, a screen is displayed for selecting a processing image from the images stored in the memory device. Once an image is selected as the processing image, the skeletal structure detection unit 12, the first matching unit 13, and the second matching unit 14 perform the processing described in the first to fifth embodiments on that processing image. The first matching unit 13 and the second matching unit 14 extract the first and second reference images based on the settings of the first and second extraction conditions at that time. The extracted second reference image is then displayed on the screen as the matching result.

[0161] When "Imaging" is selected, a screen is displayed for selecting a processing image from a live image currently being captured by any camera, or from previously captured video footage. On this screen, the live image or previously captured video footage is played back. The user then performs a shooting operation at any time during playback. The frame image displayed at that time is then selected as the processing image. Once an image is selected as the processing image, the skeletal structure detection unit 12, the first matching unit 13, and the second matching unit 14 perform the processing on that processing image as described in the first to fifth embodiments. The first matching unit 13 and the second matching unit 14 extract the first and second reference images based on the settings of the first and second extraction conditions at that time. The extracted second reference image is then displayed on the screen as the matching result.

[0162] When "Live" is selected, a screen is displayed for selecting a processing image from a live image currently being captured by any camera, or from previously captured video footage. On this screen, the live image or previously captured video footage is played back. The user then inputs a time interval for selecting the processing images. Multiple frame images are then selected as processing images at the specified time interval. The skeletal structure detection unit 12, the first matching unit 13, and the second matching unit 14 sequentially perform the processing described in the first to fifth embodiments for each of the selected processing images. The first matching unit 13 and the second matching unit 14 extract the first and second reference images based on the settings of the first and second extraction conditions at that time. The extracted second reference image is then displayed on the screen as the matching result.

[0163] Regardless of whether "Still Image," "Image Capture," or "Live" is selected, the user must select at least one label name. For example, checkboxes corresponding to multiple label names are displayed on the screen. The user selects at least one label name by checking the checkbox for the desired label name. The image processing system 10 then performs an extraction process using a second extraction condition (which is set and saved) corresponding to the data name associated with the selected group of label names, and displays the extracted second reference image on the screen as the matching result.

[0164] Here, we will explain the "initial selection" in the settings screen (see Figures 18 to 22) mentioned above. When a label name is selected on the settings screen and the settings are saved with the "initial selection" checkbox checked, that label name will be selected by default in the UI components for selecting the label name on the "still image," "imaging," and "Live" screens mentioned above. For example, as shown in Figure 21, when "wheelchair" is selected as the label name on the settings screen and the settings are saved with the "initial selection" checkbox checked, the label name "wheelchair" will be selected by default in the UI components for selecting the label name on the "still image," "imaging," and "Live" screens mentioned above.

[0165] The other configurations of the image processing system 10 in this embodiment are the same as those of the image processing system 10 in the first to fifth embodiments. Even with the image processing system 10 configured as in this embodiment, the same effects and advantages as those of the image processing system 10 in the first to fifth embodiments are achieved.

[0166] Furthermore, according to the image processing system 10 of this embodiment, input for changing the second extraction conditions can be made via the characteristic settings screen described above. By making input for changing the second extraction conditions via the characteristic settings screen described above, the user can efficiently and more accurately set the desired second extraction conditions.

[0167] The embodiments of the present invention have been described above with reference to the drawings, but these are illustrative examples of the present invention, and various other configurations can be adopted. The configurations of the embodiments described above may be combined with each other, or some configurations may be replaced with other configurations. Furthermore, the configurations of the embodiments described above may be modified in various ways without departing from the spirit of the invention. In addition, the configurations and processes disclosed in each of the embodiments and modifications described above may be combined with each other.

[0168] Furthermore, while the flowcharts used in the above description show multiple steps (processes) in sequence, the execution order of the steps performed in each embodiment is not limited to the order in which they are described. In each embodiment, the order of the illustrated steps can be changed to the extent that it does not impede the content. Also, the above embodiments can be combined to the extent that their contents do not conflict.

[0169] Some or all of the above embodiments may also be described as follows, but are not limited to the following: 1. A means for acquiring a target image, A skeletal structure detection means that performs a process to detect key points of the human body included in the target image, Based on the detected keypoint, a first matching means extracts a first reference image from among a plurality of reference images whose relationship with the target image satisfies a first extraction condition, A second matching means extracts a second reference image from the first reference image based on the detected keypoint, the second reference image whose relationship with the target image satisfies the second extraction condition, An image processing system having the following features. 2. The first extraction condition is that the similarity of human body postures included in the images calculated by the first calculation method is equal to or greater than the first reference value. The image processing system according to claim 1, wherein the second extraction condition is that the similarity of human body postures in the images calculated by the second calculation method is equal to or greater than the second reference value. 3. The first calculation method and the second calculation method are image processing systems in which at least one of the number of keypoints and the types of keypoints referenced when calculating the similarity of human body postures is different from each other. 4. The second extraction criterion is: A predetermined number or more of the keypoints referenced when calculating the similarity of human body postures have been detected, and A predetermined key point among the key points referenced when calculating the similarity of human body postures has been detected. The image processing system according to claim 3, comprising at least one of the following. 5. The first calculation method and the second calculation method are image processing systems according to any one of 2 to 4, wherein the weight settings for each of the keypoints referenced when calculating the similarity of human body postures are different from each other. 6. In the first calculation method described above, the similarity of human body postures is calculated by giving the same weight to all of the key points. The second calculation method described above involves an image processing system as described in 5, which calculates the similarity of human body postures based on weights set for each key point. 7. Display control means for displaying the second reference image on a display device, A change acceptance means for accepting input to change the second extraction condition, It further possesses, In response to receiving input to change the second extraction condition, The second matching means extracts a new second reference image from the first reference image that satisfies the modified second extraction condition regarding the relationship with the target image. The image processing system according to any one of 1 to 6, wherein the display control means changes the content to be displayed on the display device from a second reference image that satisfies the second extraction conditions before the change to a second reference image that satisfies the second extraction conditions after the change. 8. The aforementioned change acceptance means is: The image processing system according to 7, which includes a human body model composed of a plurality of keypoints, and accepts input to change the second extraction condition via a setting screen that accepts input to select a keypoint to be set on the human body model and input to change the weight of the selected keypoint. 9. The aforementioned change acceptance means is: The image processing system according to 8, which accepts input to change the second extraction condition via the settings screen that highlights the selected key points in the human body model. 10. The aforementioned change acceptance means is: The image processing system according to 8, which accepts input to change the second extraction condition via the setting screen that displays, in the human body model, the key points whose set weight is greater than a threshold and the other key points in different ways. 11. Having a server and client terminals, The server has the first matching means and transmits the extracted first reference image to the client terminal. The image processing system according to claim 1, wherein the client terminal has the second matching means, and extracts the second reference image from the first reference image received from the server. 12. A means for acquiring a target image, A skeletal structure detection means that performs a process to detect key points of the human body included in the target image, Based on the detected keypoint, a first matching means extracts a first reference image from among a plurality of reference images whose relationship with the target image satisfies a first extraction condition, A second matching means extracts a second reference image from the first reference image based on the detected keypoint, the second reference image whose relationship with the target image satisfies the second extraction condition, A device having. 13. One or more computers, Obtain the target image, The process is performed to detect key points of the human body contained in the aforementioned target image. Based on the detected keypoints, a first reference image is extracted from among multiple reference images whose relationship with the target image satisfies the first extraction condition. An image processing method for extracting a second reference image from the first reference image based on the detected keypoints, wherein the relationship between the first reference image and the target image satisfies the second extraction condition. 14. Computers, A means for acquiring a target image, A skeletal structure detection means that performs a process to detect key points of the human body included in the aforementioned target image. A first matching means extracts a first reference image from among a plurality of reference images based on the detected key points, the first reference image whose relationship with the target image satisfies a first extraction condition. A second matching means for extracting a second reference image from the first reference image based on the detected key points, the second reference image whose relationship with the target image satisfies the second extraction condition. A program that makes it function as such. [Explanation of Symbols]

[0170] 1 server 2 Client terminals 10 Image Processing Systems 11 Target image acquisition unit 12 Skeletal structure detection unit 13 First matching unit 14 Second verification unit 15 Display Control Unit 16 Change Request Department 1A Processor 2A Memory 3A input / output I / F 4A Peripheral Circuits 5A bus

Claims

1. A means for acquiring a target image, A skeletal structure detection means that performs a process to detect key points of the human body included in the target image, Based on the detected keypoint, a first matching means extracts a first reference image from among a plurality of reference images whose relationship with the target image satisfies a first extraction condition, A second matching means extracts a second reference image from the first reference image based on the detected keypoint, the second reference image whose relationship with the target image satisfies the second extraction condition, It has, The first extraction condition is that the similarity of human body postures included in the image calculated by the first calculation method is equal to or greater than the first standard value. The second extraction condition is that the similarity of human body postures included in the images calculated by the second calculation method is equal to or greater than the second standard value. The first calculation method and the second calculation method are image processing systems in which the weight settings for each of the keypoints referenced when calculating the similarity of human body postures are different from those of the other.

2. The image processing system according to claim 1, wherein the first calculation method and the second calculation method are at least one of the number of keypoints and the type of keypoints that are referenced when calculating the similarity of human body postures.

3. The second extraction condition is, A predetermined number or more of the keypoints referenced when calculating the similarity of human body postures have been detected, and A predetermined key point among the key points referenced when calculating the similarity of human body postures has been detected. The image processing system according to claim 1, further comprising at least one of the following.

4. In the first calculation method described above, the similarity of human body postures is calculated by giving the same weight to all of the key points. The image processing system according to claim 1, wherein the second calculation method calculates the similarity of human body postures based on weights set for each key point.

5. A display control means for displaying the second reference image on a display device, A change acceptance means for receiving input to change the second extraction condition, It further possesses, In response to receiving input to change the second extraction condition, The second matching means extracts a new second reference image from the first reference image that satisfies the modified second extraction condition in relation to the target image, The image processing system according to any one of claims 1 to 4, wherein the display control means changes the content to be displayed on the display device from the second reference image that satisfies the second extraction conditions before the change to the second reference image that satisfies the second extraction conditions after the change.

6. The aforementioned change acceptance means is: The image processing system according to claim 5, which includes a human body model composed of a plurality of key points, and which accepts input to select a key point to be set on the human body model, and accepts input to change the second extraction condition via a setting screen that accepts input to change the weight of the selected key point.

7. The aforementioned change acceptance means is: The image processing system according to claim 6, wherein the system accepts input to change the second extraction conditions via the settings screen that highlights the selected key points in the human body model.

8. The aforementioned change acceptance means is: The image processing system according to claim 6, wherein the system accepts input to change the second extraction condition via a setting screen that displays, in the human body model, the key points whose set weight is greater than a threshold and the other key points in different ways.

9. It has a server and client terminals. The server has the first matching means and transmits the extracted first reference image to the client terminal. The image processing system according to claim 1, wherein the client terminal has the second matching means, and extracts the second reference image from the first reference image received from the server.

10. A means for acquiring a target image, A skeletal structure detection means that performs a process to detect key points of the human body included in the target image, Based on the detected keypoint, a first matching means extracts a first reference image from among a plurality of reference images whose relationship with the target image satisfies a first extraction condition, A second matching means extracts a second reference image from the first reference image based on the detected keypoint, the second reference image whose relationship with the target image satisfies the second extraction condition, It has, The first extraction condition is that the similarity of human body postures included in the image calculated by the first calculation method is equal to or greater than the first standard value. The second extraction condition is that the similarity of human body postures included in the images calculated by the second calculation method is equal to or greater than the second standard value. The first calculation method and the second calculation method are devices in which the weight settings for each of the keypoints referenced when calculating the similarity of human body postures are different from those of the other.

11. One or more computers, Obtain the target image, The process is performed to detect key points of the human body contained in the aforementioned target image. Based on the detected keypoints, a first reference image is extracted from among multiple reference images whose relationship with the target image satisfies the first extraction condition. Based on the detected keypoints, a second reference image is extracted from the first reference image whose relationship with the target image satisfies the second extraction condition. The first extraction condition is that the similarity of human body postures included in the image calculated by the first calculation method is equal to or greater than the first standard value. The second extraction condition is that the similarity of human body postures included in the images calculated by the second calculation method is equal to or greater than the second standard value. The first calculation method and the second calculation method are image processing methods in which the weight settings for each of the keypoints referenced when calculating the similarity of human body postures are different from those of the other.

12. Computers, A means for acquiring a target image, A skeletal structure detection means that performs a process to detect key points of the human body included in the aforementioned target image. A first matching means extracts a first reference image from among a plurality of reference images based on the detected key points, the first reference image whose relationship with the target image satisfies the first extraction condition. A second matching means for extracting a second reference image from the first reference image based on the detected key points, the second reference image whose relationship with the target image satisfies the second extraction condition. To make it function as, The first extraction condition is that the similarity of human body postures included in the image calculated by the first calculation method is equal to or greater than the first standard value. The second extraction condition is that the similarity of human body postures included in the images calculated by the second calculation method is equal to or greater than the second standard value. The first calculation method and the second calculation method are programs in which the weight settings for each of the keypoints referenced when calculating the similarity of human body postures are different from each other.

Images