Information processing device, information processing method, and program

Abstract

This information processing device (100) comprises a posture acquisition unit (113), a retrieval unit (117), and a display control unit (119). The posture acquisition unit (113) acquires first posture information that shows the posture of a person shown in each of a plurality of reference images that are associated with a prescribed posture, and second posture information that shows the posture of an imaged subject shown in a query image. The retrieval unit (117) retrieves, on the basis of the similarity between the first and the second posture information, a reference image from among the plurality of reference images that shows a person whose posture or action resembles the imaged subject shown in the query image. The display control unit (119) causes a display unit to display the first posture information and / or the second posture information in a display mode that corresponds to the index that is used for retrieving a reference image.

Description

Information processing device, information processing method, and recording medium

[0001] The present invention relates to an information processing device, an information processing method, and a recording medium.

[0002] For example, the image search device described in Patent Document 1 includes a posture estimation unit, a feature extraction unit, a query generation unit, and an image search unit.

[0003] The posture estimation unit described in the document recognizes posture information of a search target, which is composed of multiple feature points, from an input image. The feature extraction unit described in the document extracts features from the posture information and the input image. The query generation unit described in the document generates a search query from an image database that stores feature amounts associated with input images and posture information specified by a user. The image search unit described in the document searches the image database for images containing similar postures according to the search query.

[0004] For example, the image processing device described in Patent Document 2 includes an image acquisition unit, a skeletal structure detection unit, a query evaluation unit, a selection unit, a feature calculation unit, and a search unit.

[0005] The image acquisition unit described in the same document acquires candidate query images. The skeletal structure detection unit described in the same document detects the two-dimensional skeletal structures of people included in the candidate query images. The query evaluation unit described in the same document calculates evaluation values ​​for the candidate query images based on the detection results of the two-dimensional skeletal structures. The selection unit described in the same document selects a query image from the candidate query images based on the evaluation values. The feature calculation unit described in the same document calculates feature values ​​for the two-dimensional skeletal structure detected from the query image. The search unit described in the same document searches for analysis target images that include a person with a posture similar to that of the person included in the query image, based on the similarity of the calculated feature values.

[0006] Patent Literature 3 describes a technology for calculating feature amounts for each of a plurality of key points of a human body included in an image, searching for images containing human bodies with similar postures or movements based on the calculated feature amounts, and classifying images with similar postures or movements together. Non-Patent Literature 1 describes a technology related to human skeleton estimation.

[0007] JP 2019-0901138 A International Publication No. 2021 / 250808 International Publication No. 2021 / 084677

[0008] Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields];, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, P. 7291-7299

[0009] Patent Documents 1 and 2 describe techniques for estimating posture or behavior based on an image. However, Patent Documents 1 and 2 do not know whether the posture has been estimated correctly, making it difficult to improve the accuracy of estimating the posture of a subject shown in an image.

[0010] Note that neither Patent Document 3 nor Non-Patent Document 1 discloses a technique for improving the accuracy of detecting a person in a predetermined posture from an image of the person.

[0011] In view of the above-mentioned problems, one example of the object of the present invention is to provide an information processing device, an information processing method, and a recording medium that solve the problem of improving the accuracy of estimating the posture of a person being photographed shown in an image.

[0012] According to one aspect of the present invention, there is provided an information processing device comprising: a posture acquisition means for acquiring first posture information indicating the posture of a person shown in each of a plurality of reference images associated with a predetermined posture, and second posture information indicating the posture of a person shown in a query image; a search means for searching, from among the plurality of reference images, for a reference image indicating a person whose posture or behavior is similar to that of the person shown in the query image, based on the similarity between the first posture information and the second posture information; and a display control means for displaying at least one of the first posture information and the second posture information on a display means in a display mode corresponding to an index used to search for the reference images.

[0013] According to one aspect of the present invention, an information processing method is provided in which a computer acquires first posture information indicating the posture of a person shown in each of a plurality of reference images associated with a predetermined posture, and second posture information indicating the posture of a person shown in a query image, searches from the plurality of reference images for a reference image showing a person whose posture or behavior is similar to that of the person shown in the query image based on the similarity between the first posture information and the second posture information, and displays at least one of the first posture information and the second posture information on a display means in a display mode corresponding to an index used to search for the reference image.

[0014] According to one aspect of the present invention, a recording medium having recorded thereon a program for causing a computer to acquire first posture information indicating the posture of a person shown in each of a plurality of reference images associated with a predetermined posture, and second posture information indicating the posture of a person shown in a query image, search for a reference image from among the plurality of reference images that indicates a person whose posture or behavior is similar to that of the person shown in the query image based on the similarity between the first posture information and the second posture information, and display at least one of the first posture information and the second posture information on a display means in a display manner corresponding to an index used to search for the reference image, is provided.

[0015] According to one aspect of the present invention, it is possible to provide an information processing device, an information processing method, and a recording medium that solve the problem of improving the accuracy of estimating the posture of a person being photographed shown in an image.

[0016] 1 is a diagram illustrating an overview of an information processing device according to an embodiment; FIG. 2 is a diagram illustrating an overview of an information processing system according to an embodiment; FIG. 3 is a flowchart illustrating an overview of information processing according to an embodiment; FIG. 4 is a diagram illustrating an example of a detailed functional configuration of an information processing system according to an embodiment; FIG. 5 is a diagram illustrating an example of a configuration of reference information including a reference image associated with a calling posture; FIG. 6 is a diagram illustrating an example of a configuration of weight information indicating a weight associated with a calling posture; FIG. 7 is a diagram illustrating an example of a functional configuration of a similarity acquisition unit according to an embodiment; FIG. 8 is a diagram illustrating an example of a physical configuration of an information processing device according to an embodiment; FIG. 9 is a flowchart illustrating an example of a posture estimation process according to an embodiment; FIG. 10 is a diagram illustrating an example of a method of thinning out a portion from a plurality of frame images; FIG. 11 is a flowchart illustrating an example of detailed similarity acquisition process according to an embodiment; FIG. 12 is a flowchart illustrating an example of estimation support process according to an embodiment; FIG. 13 is a diagram illustrating an example of a result display screen that displays search results; FIG. 14 is a diagram illustrating an example of a display of a query image on the result display screen; FIG. 15 is a diagram illustrating another example of display of a query image on the result display screen.

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0018] 1 is a diagram showing an overview of an information processing device 100 according to an embodiment. The information processing device 100 includes a posture acquisition unit 113, a search unit 117, and a display control unit 119.

[0019] The posture acquisition unit 113 acquires first posture information indicating the posture of a person shown in each of a plurality of reference images associated with a predetermined posture, and second posture information indicating the posture of a person shown in a query image.

[0020] The search unit 117 searches the plurality of reference images for a reference image showing a person whose posture or behavior is similar to that of the subject shown in the query image, based on the similarity between the first posture information and the second posture information.

[0021] The display control unit 119 causes the display unit to display at least one of the first attitude information and the second attitude information in a display mode according to the index used to search for the reference image.

[0022] According to this information processing device 100, it is possible to provide an information processing device that solves the problem of improving the accuracy of estimating the posture of a person to be photographed shown in an image.

[0023] 2 is a diagram showing an overview of an information processing system S1 according to an embodiment. The information processing system S1 includes an image capturing unit 101 for capturing a query image, and an information processing device 100.

[0024] According to this information processing system S1, it is possible to provide an information processing system that solves the problem of improving the accuracy of estimating the posture of a person to be photographed shown in an image.

[0025] FIG. 3 is a flowchart showing an overview of information processing according to an embodiment.

[0026] The posture acquisition unit 113 acquires first posture information indicating the posture of a person shown in each of a plurality of reference images associated with a predetermined posture, and second posture information indicating the posture of a person shown in a query image (step S103).

[0027] The search unit 117 searches for a reference image showing a person whose posture or behavior is similar to that of the subject shown in the query image from among multiple reference images, based on the similarity between the first posture information and the second posture information (step S201).

[0028] The display control unit 119 causes the display unit to display at least one of the first orientation information and the second orientation information in a display mode corresponding to the index used to search for the reference image (step S203).

[0029] According to this information processing, it is possible to provide an information processing method that solves the problem of improving the accuracy of estimating the posture of a person to be photographed shown in an image.

[0030] (Details) A detailed example of the information processing system S1 according to an embodiment will be described below.

[0031] 4 is a diagram showing a detailed example of the functional configuration of an information processing system S1 according to an embodiment. The information processing system S1 includes an imaging unit 101, an information processing device 100, and an analysis device 102. The imaging unit 101, the information processing device 100, and the analysis device 102 are connected via a network N that may be wired, wireless, or a combination of these, and can transmit and receive information to and from each other.

[0032] The photographing unit 101 photographs a person (subject) performing a predetermined action. The photographing unit 101 is, for example, a camera installed in a branch of a financial institution such as a bank, and photographs an operator operating an automated teller machine (ATM).

[0033] The photographing unit 101 is not limited to a camera for photographing an ATM operator, but may be a camera for photographing the inside of a store such as a bank, or may be a camera installed in various stores other than financial institutions. Furthermore, there may be multiple photographing units 101.

[0034] The image capturing unit 101 captures an image of a predetermined image capturing area and transmits image information representing a moving image to the information processing device 100 .

[0035] Specifically, the image capturing unit 101 captures images multiple times in succession at a predetermined frame rate. The image capturing unit 101 generates frame information including frame images for each capture. The image capturing unit 101 transmits the frame information including each of the frame images in time series to the information processing device 100 via the network N.

[0036] The analysis device 102 is a device that analyzes images. The analysis device 102 acquires image information generated by the imaging unit 101 via the network N. In this embodiment, an example will be described in which the analysis device 102 acquires image information from the imaging unit 101 via the information processing device 100, but the analysis device 102 may also acquire image information directly from the imaging unit 101.

[0037] The analysis device 102 is a device that analyzes the image included in the acquired image information.

[0038] Specifically, the analysis device 102 has one or more analysis functions that perform processing (analysis processing) to analyze an image. The analysis functions that the analysis device 102 has include one or more of: (1) an object detection function, (2) a face analysis function, (3) a human figure analysis function, (4) a posture analysis function, (5) a behavior analysis function, (6) an appearance attribute analysis function, (7) a gradient feature analysis function, (8) a color feature analysis function, and (9) a movement line analysis function.

[0039] (1) The object detection function detects people and objects from an image. The object detection function can also determine the positions of people and objects within an image. YOLO (You Only Look Once) is an example of a model that can be applied to object detection processing. The object detection function detects, for example, an operator, a mobile phone (including a smartphone), a wheelchair, etc. Furthermore, for example, the object detection function determines the positions of detected people and objects.

[0040] (2) The face analysis function detects human faces from images, extracts the features of the detected faces (facial feature values), and classifies the detected faces (classification). The face analysis function can also determine the position of the face within the image. The face analysis function can also determine the identity of people detected from different images based on the similarity between the facial feature values ​​of people detected from different images.

[0041] (3) The human morphology analysis function extracts the physical characteristics of people included in an image (for example, values ​​indicating overall characteristics such as whether they are fat or thin, their height, and their clothing), and classifies (classifies) people included in an image. The human morphology analysis function can also identify the position of a person within an image. The human morphology analysis function can also determine the identity of people included in different images based on the physical characteristics of the people included in different images.

[0042] (4) The posture analysis function generates posture information indicating the posture of a person. The posture information includes, for example, a posture estimation model of a person. The posture estimation model is a model connecting the joints of a person estimated from an image. The posture estimation model is composed of multiple model elements corresponding to joint elements corresponding to joints, trunk elements corresponding to the torso, bone elements corresponding to bones connecting the joints, etc. The posture analysis function, for example, detects the joint points of a person from an image and connects the joint points to create a posture estimation model.

[0043] The posture analysis function then uses information from the posture estimation model to estimate a person's posture, extract features of the estimated posture (posture features), and classify (classify) people included in the images. The posture analysis function can also determine the identity of people included in different images based on the posture features of people included in different images.

[0044] For example, the posture analysis function creates posture estimation models such as a talking posture and a wheelchair posture, and extracts posture features for those postures. The talking posture is the posture of talking on a mobile phone. The wheelchair posture is the posture of a person using a wheelchair.

[0045] The posture analysis function can apply, for example, the techniques disclosed in Patent Document 3 and Non-Patent Document 1. (5) The behavior analysis process can estimate a person's movements using information from a posture estimation model, changes in posture, etc., extract features of the person's movements (movement features), and classify (classify) people included in an image. The behavior analysis process can also estimate a person's height and identify the person's position in an image using information from a stick figure model. The behavior analysis process can estimate, for example, behaviors such as changes or transitions in posture and movements (changes or transitions in position) from an image and extract movement features of the behavior.

[0046] (6) The appearance attribute analysis function can recognize appearance attributes associated with a person. The appearance attribute analysis function extracts features (appearance attribute features) related to the recognized appearance attributes and classifies (classifies) people included in the image. Appearance attributes are attributes related to appearance, and include, for example, one or more of clothing color, shoe color, hairstyle, and whether or not a hat, tie, or glasses is worn.

[0047] (7) The gradient feature analysis function extracts gradient feature quantities (gradient feature quantities) in an image. For gradient feature detection processing, techniques such as SIFT, SURF, RIFF, ORB, BRISK, CARD, and HOG can be applied.

[0048] (8) The color feature analysis function can detect objects from an image, extract color features of the detected objects, and classify the detected objects. The color features are, for example, color histograms. The color feature analysis function can detect, for example, people and objects included in the image.

[0049] (9) The flow line analysis function can determine the flow line (trajectory of movement) of a person included in a video, for example, using the result of the identity determination in any of the above-described analysis functions (2) to (6). In more detail, for example, by connecting people determined to be the same in different images in a time series, the flow line of that person can be determined. Note that, in cases where video images captured by multiple image capture units 101 capturing different shooting areas are acquired, the flow line analysis function can also determine the flow line spanning multiple videos captured in different shooting areas.

[0050] Image features include, for example, the results of object detection by the object detection function, facial features, human body features, posture features, movement features, appearance attribute features, gradient features, color features, and movement lines.

[0051] Each of the analysis functions (1) to (9) may appropriately use the results of analysis performed by other analysis functions. The information processing device 100 may include an analysis unit having the functions of the analysis device 102.

[0052] An information processing device 100 according to an embodiment is a device that estimates the posture of a person included in a frame image. As shown in Fig. 4 , the information processing device 100 functionally includes an image acquisition unit 111, a storage unit 112, a posture acquisition unit 113, a similarity acquisition unit 114, an estimation unit 115, an input unit 116, a search unit 117, a display unit 118, and a display control unit 119.

[0053] The image acquisition unit 111 acquires image information indicating a moving image from the imaging unit 101. That is, the image acquisition unit 111 acquires a plurality of frame images in time series obtained by capturing images a plurality of times consecutively in time.

[0054] More specifically, the image acquisition unit 111 acquires frame information including each of a plurality of frame images in time series from the imaging unit 101. The image acquisition unit 111 stores the acquired frame information.

[0055] The storage unit 112 is a storage unit for storing various types of information, and stores, in advance, for example, reference information 112a indicating a reference image, weight information 112b indicating a weight, and the like.

[0056] The reference image is an image of a person associated with a predetermined posture. The reference image is an image that is referenced to estimate the posture of a person included in a query image, and is appropriately selected and set in the storage unit 112. The predetermined posture is, for example, a talking posture, a wheelchair posture, etc. Note that the talking posture may be further subdivided and set into a right-hand talking posture in which a mobile phone is held in the right hand and a right-hand talking posture in which a mobile phone is held in the left hand and a talking posture, etc.

[0057] 5 is a diagram showing an example of the configuration of reference information 112a including reference images associated with talking postures. The reference information 112a shown in FIG. 5 includes, for example, positive examples and negative examples.

[0058] A positive example is a reference image of a person in a talking pose. For example, a positive example shows a person standing and talking on a mobile phone with their right or left hand.

[0059] Negative examples are reference images of people who are not in a phone call pose. Images of people who are not in a phone call pose but are in a pose similar to a phone call pose may be selected as negative examples. For example, a negative example may show a person standing upright without holding a mobile phone.

[0060] 5 shows an example of reference information 112a including M positive examples and N negative examples associated with calling postures, where M and N are both integers equal to or greater than 1.

[0061] The reference information 112a may include any number of reference images as long as it includes at least one reference image. The reference information 112a may also include only positive examples.

[0062] The weight is a value indicating the degree of importance given to each model element for determining the similarity between posture estimation models in a given posture. Weight information 112b includes the weight of each model element for each given posture.

[0063] FIG. 6 is a diagram showing an example of the configuration of weight information 112b indicating weights associated with different postures during a call. The weight information 112b shown in FIG. 6 associates element IDs and weights in different postures during a call. The element IDs are information for identifying model elements. The element IDs are, for example, numbers appropriately assigned to bone elements and joint elements corresponding to the trunk element, the upper and lower parts of the left and right arms, the thighs and lower legs, etc. Weights are determined for each model element of a given posture. While FIG. 6 shows an example in which the weights are integers equal to or greater than 0, the method for setting weights may be changed as appropriate.

[0064] For example, in a talking posture, the weight set for the arms is greater than the weight set for the legs because the mobile phone is held while talking, and in a talking posture when talking with the right hand, the weight set for the right hand is greater than the weight set for the left hand.

[0065] The posture acquisition unit 113 acquires, from the storage unit 112, a plurality of reference images associated with a predetermined posture, such as a talking posture, and acquires first posture information based on the acquired plurality of reference images.

[0066] The first pose information is information indicating the pose of a person shown in each of a plurality of reference images associated with a predetermined pose. The first pose information includes, for example, a first model that is a pose estimation model for the person shown in the reference image.

[0067] Furthermore, the orientation acquisition unit 113 acquires time-series frame images from the image acquisition unit 111 and acquires a query image by thinning out some of the time-series frame images. Then, the orientation acquisition unit 113 acquires second orientation information based on the acquired query image.

[0068] The second pose information is information indicating the pose of the subject shown in the query image. The second pose information includes, for example, a second model that is a pose estimation model for the subject shown in the query image.

[0069] In detail, for example, the posture acquisition unit 113 transmits each of the acquired reference image and query image to the analysis device 102 via the network N. When the reference image is transmitted to the analysis device 102, the posture acquisition unit 113 acquires first posture information including a first model related to the person shown in the reference image from the analysis device 102. When the query image is transmitted to the analysis device 102, the posture acquisition unit 113 acquires second posture information including a second model related to the person shown in the query image from the analysis device 102.

[0070] The similarity obtaining unit 114 obtains the similarity in the posture between the subject shown in the query image and the person shown in the reference image for each combination of a time-series query image and a plurality of reference images corresponding to a predetermined posture.

[0071] The similarity is a value that indicates the degree of similarity between pose estimation models in a given pose.

[0072] For example, the similarity acquisition unit 114 acquires a first model of a person shown in each of a plurality of reference images corresponding to a predetermined posture from the posture acquisition unit 113. The similarity acquisition unit 114 also acquires a second model of a person shown in each of the time-series query images from the posture acquisition unit 113. For each combination of the first model and the second model, the similarity acquisition unit 114 calculates the similarity using the first model and the second model.

[0073] The similarity includes an overall similarity and an element similarity.

[0074] The overall similarity is a value indicating the overall degree of similarity between the first model and the second model in a given posture, that is, the overall similarity between the first model and the second model.

[0075] The element similarity is the similarity for each corresponding model element between the first model and the second model in a predetermined posture, that is, the similarity for each corresponding model element between the first model and the second model.

[0076] The similarity may include at least one of the overall similarity and the element similarity.

[0077] 7 is a diagram showing an example of the functional configuration of the similarity acquisition unit 114 according to this embodiment. The similarity acquisition unit 114 includes an overall calculation unit 114a and an element calculation unit 114b.

[0078] The overall calculation unit 114a calculates the overall similarity between the first model and the second model. Specifically, the overall calculation unit 114a calculates the overall similarity using weights corresponding to the predetermined postures included in the weight information 112b and the element similarities calculated by the element calculation unit 114b.

[0079] For example, when overall calculation unit 114a obtains the similarity of each model element from element calculation unit 114b, it calculates the product of each similarity of the model element and the weight of the corresponding model element, and sums up the products obtained for each model element constituting the posture estimation model. The value obtained as a result of this summation is the overall similarity.

[0080] The element calculation unit 114b calculates element similarity, which is the degree of similarity for each corresponding model element between the first model and the second model. For example, the element calculation unit 114b calculates element similarity for each corresponding model element between the first model and the second model based on the size, length, inclination, etc.

[0081] The estimation unit 115 estimates the posture of the person being photographed shown in each of the multiple query images based on multiple query images obtained by taking multiple photographs while the person is performing a specified action and a reference image showing a person associated with a specified posture.

[0082] For example, the estimation unit 115 estimates the posture of the person being photographed shown in each of the time-series query images based on the similarity (for example, overall similarity) obtained by the similarity acquisition unit 114 .

[0083] In addition, the estimation unit 115 may estimate the posture of the person being photographed shown in at least one thinned frame image from the time series of frame images based on the at least one thinned frame image and a reference image.

[0084] In this case, the posture acquisition unit 113 acquires at least one of the thinned frame images from the image acquisition unit 111 and acquires a second model of the person shown in the frame image. The similarity acquisition unit 114 calculates an overall similarity based on the second model of the person shown in the frame image and the first models of the person shown in each of the multiple reference images. Then, the estimation unit 115 estimates the posture of the person shown in the at least one thinned frame image based on the overall similarity calculated by the similarity acquisition unit 114.

[0085] There are various methods for the estimation unit 115 to estimate the posture of the person being photographed based on the similarity. As examples, posture estimation methods 1 to 5 will be described below.

[0086] (Posture Estimation Method 1) For example, the estimation unit 115 may estimate the posture of the person shown in the query image or the frame image based on the reference image with the highest similarity among the positive examples and negative examples. In this case, for example, if the reference image with the highest similarity is a positive example, the estimation unit 115 estimates that the posture of the person is a predetermined posture corresponding to the reference image. If the reference image with the highest similarity is a negative example, the estimation unit 115 estimates that the posture of the person is not the predetermined posture corresponding to the reference image.

[0087] (Posture Estimation Method 2) For example, the estimation unit 115 may estimate the posture of the photographed person shown in the query image or the frame image based on the average positive example value and the average negative example value. The average positive example value is the average value of the similarity between the query image or the frame image and multiple positive examples associated with a predetermined posture. The average negative example value is the average value of the similarity between the query image or the frame image and multiple negative examples associated with a predetermined posture.

[0088] In this case, for example, if the positive example average value is greater than the negative example average value, the estimation unit 115 estimates that the posture of the photographed person is the predetermined posture corresponding to the reference image. If the positive example average value is equal to or less than the negative example average value, the estimation unit 115 estimates that the posture of the photographed person is not the predetermined posture corresponding to the reference image.

[0089] (Posture Estimation Method 3) Furthermore, for example, the estimation unit 115 may perform image matching between a query image or a frame image and a reference image, and estimate the posture of the person shown in the query image or the frame image based on the similarity between the reference image that matches in the image matching and the query image or the frame image. In this case, the estimation unit 115 may estimate the posture of the person shown in the query image or the frame image based on the average positive example value and the average negative example value among the reference images that match in the image matching.

[0090] In detail, for example, when the average value of positive examples among the reference images matched by image matching is greater than the average value of negative examples, the estimation unit 115 estimates that the posture of the photographed person is a predetermined posture corresponding to the reference images. When the average value of positive examples among the reference images matched by image matching is equal to or less than the average value of negative examples, the estimation unit 115 estimates that the posture of the photographed person is not a predetermined posture corresponding to the reference images.

[0091] Note that various known techniques may be applied to image matching. For example, the estimation unit 115 calculates the similarity between images based on feature amounts between the subject shown in the query image or frame image and the person shown in the reference image. The estimation unit 115 compares the similarity between the images with a threshold to determine whether the query image or frame image and the reference image match. For example, the estimation unit 115 determines that the images match (are similar) when the similarity between the images is equal to or greater than the threshold, and determines that the images do not match (are dissimilar) when the similarity between the images is less than the threshold.

[0092] (Posture Estimation Method 4) The estimation unit 115 may estimate the posture of the person being photographed using only image matching, without using similarity.

[0093] For example, the estimation unit 115 may estimate that the subject shown in the query image or the frame image is in a predetermined posture corresponding to the reference image when the query image or the frame image matches at least one positive example in the image matching. Alternatively, for example, the estimation unit 115 may estimate that the subject shown in the query image or the frame image is not in a predetermined posture corresponding to the reference image when the query image or the frame image matches at least one negative example in the image matching.

[0094] Furthermore, for example, the estimation unit 115 may estimate the posture of the subject based on the number of matches between positive examples and negative examples that match in image matching. In this case, for example, if the number of matches with positive examples is greater than the number of matches with negative examples, the estimation unit 115 estimates that the subject shown in the query image or frame image is in a predetermined posture corresponding to the reference image. Also, for example, if the number of matches with positive examples is equal to or less than the number of matches with negative examples, the estimation unit 115 estimates that the subject shown in the query image or frame image is not in a predetermined posture corresponding to the reference image.

[0095] In image matching, if a query image or a frame image does not match either a positive example or a negative example, the estimation unit 115 may determine that the query image or the frame image is different from either a positive example or a negative example, or may determine that the query image or the frame image is a negative example.

[0096] (Pose estimation method 5) The estimation unit 115 may determine whether a query image or the frame image conforms to (is similar to) a positive example or a negative example by using a learning model that has been trained through machine learning using a reference image. This learning model is a learning model that has been trained through machine learning for determining whether the subject conforms to a positive example or a negative example. In this case, the estimation unit 115 inputs image information including a query image or the frame image that shows the subject into the learning model, and obtains a determination result as to whether the query image or the frame image conforms to (is similar to) a positive example or a negative example.

[0097] The input data to the learning model during learning includes image information of a person. Furthermore, learning may be performed using supervised learning that includes labels (correct answers) that indicate whether the input data matches a positive example or a negative example.

[0098] The input unit 116 is a keyboard, a mouse, a touch panel, or the like that accepts input from the user.

[0099] The search unit 117 acquires, for example, a query image appropriately specified by a user. The search unit 117 searches, from among multiple reference images, for a reference image showing a person whose posture or behavior is similar to that of the subject shown in the query image, based on the similarity between the first posture information and the second posture information. The search unit 117 outputs, as a search result, for example, a reference image whose similarity is equal to or greater than a predetermined reference value.

[0100] The query image specified by the user may be any of the frame images, or may be an image other than the image captured by the image capturing unit 101. The reference value used for the search may be a value related to the overall similarity or a value related to the element similarity.

[0101] When the reference value is determined for the element similarity, the search unit 117 outputs, for example, a reference image corresponding to a first model having an element similarity equal to or greater than the reference value as a search result.

[0102] The display unit 118 is a display or the like that displays various types of information. The display control unit 119 controls the display unit 118 and causes the display unit 118 to display various types of information. For example, when the estimation unit 115 detects a person to be photographed in a predetermined posture, the display control unit 119 causes the display unit 118 to display a query image or a frame image in which the person to be photographed is marked. The mark is, for example, a rectangular frame that surrounds the person to be photographed.

[0103] Furthermore, the display control unit 119 causes the display unit 118 to display at least one of the first attitude information and the second attitude information in a display mode according to the index used to search for the reference image.

[0104] The indices used to search for reference images are, for example, one or more of similarity, weight, etc. However, the indices used to search for reference images are not limited to these. The display mode is, for example, the thickness, color, and type of line representing trunk elements, the thickness, color, and type of line representing bone elements, the size, color, and type of line of shapes (e.g., circles, squares, etc.) representing joint elements, etc. The line type is, for example, a solid line, a dotted line, a dashed line, a one-dot chain line, a two-dot chain line, etc.

[0105] In detail, the display control unit 119 displays at least one of the first orientation information corresponding to the reference image included in the search results of the search unit 117 and the second orientation information corresponding to the user-specified query image acquired by the search unit 117 on the display unit 118 in a display mode according to the index.

[0106] If the search results output from the search unit 117 are greater than a predetermined maximum number of images to be displayed, the display control unit 119 may select the maximum number of reference images to be displayed, starting with the reference image corresponding to the first model with the greatest similarity.

[0107] When the index includes the similarity, the display control unit 119 may cause the display unit 118 to display at least one of the first attitude information and the second attitude information in a display mode according to the similarity.

[0108] When displaying the second attitude information, the display control unit 119 may associate the second attitude information with a specific reference image from among multiple reference images, and display the second attitude information on the display unit 118 in a display mode that corresponds to the similarity between the second attitude information and the first attitude information corresponding to the specific reference image.

[0109] In this case, the first attitude information and the second attitude information may include the first model and the second model, respectively. When displaying the second model, the display control unit 119 may associate the second model with a specific reference image among the multiple reference images and cause the display unit 118 to display the second model in a display mode according to the degree of similarity between the second model and the first model corresponding to the specific reference image.

[0110] The first model corresponding to a particular reference image is one of the first models having the greatest overall similarity with the second model, the first model having the greatest element similarity, the first model specified by the user, etc.

[0111] When displaying the second model, the display control unit 119 may calculate the average element similarity for each model element between the second model and each of the first models output as search results for that model, and display the second model on the display unit 118 in a display mode corresponding to the average value.

[0112] When the similarity includes element similarity, the display control unit 119 may cause the display unit 118 to display only model elements that constitute at least one of the first model and the second model and whose element similarity is equal to or greater than a first reference value or equal to or less than a second reference value, or whose weight is equal to or greater than a predetermined threshold value. Each of the first reference value and the second reference value is a value that is set as appropriate.

[0113] When the index includes a weight, the display control unit 119 may cause the display unit 118 to display a plurality of model elements that make up at least one of the first model and the second model in a display mode according to the weight.

[0114] The display control unit 119 may cause the display unit 118 to display at least one of the trunk elements and bone elements, which are model elements, as a line with an arrow.

[0115] (Physical configuration of information processing system S1) The information processing system S1 is physically composed of an imaging unit 101, an information processing device 100, and an analysis device 102, which are connected via a network N. The imaging unit 101, the information processing device 100, and the analysis device 102 are each composed of a physically different single device. The imaging unit 101 is, for example, a camera.

[0116] The information processing device 100 and the analysis device 102 may be physically configured as a single device, in which case the information processing device 100 and the analysis device 102 are connected using an internal bus 1010 (described later) instead of the network N. Furthermore, one or both of the information processing device 100 and the analysis device 102 may be physically configured as multiple devices connected via an appropriate communication line such as the network N.

[0117] 8 is a diagram showing an example of the physical configuration of an information processing device 100 according to this embodiment. The information processing device 100 is, for example, a general-purpose computer. The information processing device 100 includes, for example, a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070.

[0118] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0119] The processor 1020 is implemented as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0120] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0121] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules for realizing each function of the information processing device 100. The processor 1020 reads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module.

[0122] The network interface 1050 is an interface for connecting the information processing device 100 to the network N.

[0123] The input interface 1060 is an interface for a user to input information, and is configured with one or more of, for example, a keyboard, a mouse, a touch panel, and the like.

[0124] The output interface 1070 is an interface for presenting information to the user, and is configured, for example, by a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like.

[0125] The analysis device 102 is, for example, a general-purpose computer, and has a physical configuration similar to that of the information processing device 100 (see FIG. 8).

[0126] Storage device 1040 of analysis device 102 stores program modules for realizing each function of analysis device 102. Processor 1020 of analysis device 102 reads each of these program modules into memory 1030 and executes them, thereby realizing each function corresponding to the program module. Network interface 1050 of analysis device 102 is an interface for connecting analysis device 102 to network N. Except for these points, analysis device 102 may be physically configured in the same way as information processing device 100.

[0127] (Operation of Information Processing System S1) The information processing system S1 according to this embodiment executes processing for estimating the posture of a subject included in a query image. The processing executed by the information processing system S1 includes a posture estimation process and an estimation support process.

[0128] The posture estimation process is a process for estimating the posture of the subject included in the query image using a reference image associated with a predetermined posture. The estimation support process is a process for supporting the estimation of the posture of the subject.

[0129] 9 is a flowchart showing an example of the posture estimation process according to this embodiment. The posture estimation process is executed, for example, while the information processing system S1 is in operation.

[0130] The image acquisition unit 111 acquires a plurality of frame images in time series (step S101), and stores the acquired frame images.

[0131] More specifically, for example, the image acquisition unit 111 sequentially acquires a plurality of frame images in time series from time T1 to just before time T2, where time T2-time T1=time interval ΔT.

[0132] The image acquisition unit 111 thins out some of the frame images acquired in step S101 to acquire a query image (step S102).

[0133] In more detail, for example, the image acquisition unit 111 thins out some of the multiple frame images according to a predetermined rule. For example, FIG. 10 is a diagram illustrating an example of a method for thinning out some of the multiple frame images. As illustrated in FIG. 10, the image acquisition unit 111 thins out frame images acquired during a predetermined time interval ΔT (excluding both ends). As a result, the image acquisition unit 111 acquires query images in a time series at a predetermined constant time interval ΔT. Note that the method for thinning out some of the multiple frame images to acquire query images is not limited to this. For example, the time interval ΔT does not need to be constant and may be changed depending on the operation mode (a mode for tracking a subject, a mode for detecting the subject's posture). Alternatively, the query images may be a plurality of unthinned frame images.

[0134] The orientation acquisition unit 113 acquires first orientation information based on a plurality of reference images associated with a predetermined orientation, and second orientation information based on the query image acquired in step S102 (step S103).

[0135] In detail, for example, posture acquisition unit 113 acquires a plurality of reference images corresponding to a predetermined posture from storage unit 112. When the predetermined posture is a call posture and reference information 112a shown in FIG. 5 is stored in storage unit 112, posture acquisition unit 113 acquires reference image 1 to reference image 7. Posture acquisition unit 113 transmits the acquired reference image 1 to reference image 7 to analysis device 102. In response to this, analysis device 102 generates first posture information including a first model of a person represented by each of reference image 1 to reference image 7 and transmits the first posture information to information processing device 100. Posture acquisition unit 113 acquires the first posture information from analysis device 102.

[0136] The posture acquisition unit 113 acquires the query image acquired in step S102 from the image acquisition unit 111. The posture acquisition unit 113 transmits the acquired query image to the analysis device 102. In response to this, the analysis device 102 generates second posture information including a second model of the subject indicated by the query image and transmits the second posture information to the information processing device 100. The posture acquisition unit 113 acquires the second posture information from the analysis device 102.

[0137] The similarity obtaining unit 114 obtains the similarity between each of the first models included in the first orientation information and the second orientation information obtained in step S102 and the second model (step S104).

[0138] FIG. 11 is a flowchart showing a detailed example of the similarity obtaining process (step S104) according to this embodiment.

[0139] The element calculation unit 114b repeats steps S104b to S104c for each of the first models included in the first orientation information acquired in step S103 (step S104a).

[0140] The element calculation unit 114b calculates element similarity, which is the similarity for each corresponding model element between the first model and the second model (step S104b).

[0141] The overall calculation unit 114a acquires the weight information 112b stored in the memory unit 112, and calculates the overall similarity between the first model and the second model based on the element similarity and weight for each model element calculated in step S104b (step S104c).

[0142] For example, the overall calculation unit 114a calculates the sum of the products of the element similarities and weights of the corresponding model elements, and sets this sum as the overall similarity.

[0143] Overall calculation unit 114a repeats steps S104b to S104c for each of the first models included in the first posture information acquired in step S103, and as a result, calculates the overall similarity between each of the first models and the second model. After executing steps S104b to S104c for each of the first models included in the first posture information acquired in step S103, overall calculation unit 114a ends loop A (step S104a) and returns to the posture estimation process.

[0144] Referring again to Fig. 9, the estimation unit 115 estimates the posture of the person being photographed shown in the query image based on the query image acquired in step S102 and a plurality of reference images (step S105).

[0145] For example, the estimation unit 115 estimates the posture of the person shown in the query image acquired in step S102 based on the overall similarity between the query image and each of reference images 1 to 7. Note that in step S105, the estimation unit 115 may use any of the posture estimation methods 1 to 5 described above, or may use a method other than posture estimation methods 1 to 5, in order to estimate the posture of the person.

[0146] The estimation unit 115 determines whether or not a predetermined posture has been detected (step S106).

[0147] In detail, for example, the estimation unit 115 determines that the predetermined posture has been detected when the subject shown in the query image is estimated to be in the predetermined posture in step S105. The estimation unit 115 determines that the predetermined posture has not been detected when the subject shown in the query image is estimated to be not in the predetermined posture in step S105.

[0148] If it is determined that the predetermined posture has not been detected (step S106; No), the image acquisition unit 111 executes step S101 again.

[0149] If it is determined that the predetermined posture has been detected (step S106; No), the display control unit 119 causes the display unit 118 to display that the predetermined posture has been detected (step S107). After that, the image acquisition unit 111 executes step S101 again.

[0150] In step S107, the display control unit 119 causes a query image showing the subject in a predetermined posture to be displayed on the display unit 118. The query image displayed here may be an image in which the subject is marked, as described above.

[0151] The user can know that a person to be photographed in a predetermined posture has been detected by looking at the display unit 118. For example, if the person is in a talking posture while operating an ATM, there is a possibility that the person may be a victim of a bank transfer fraud or a suspicious person, and the user can take measures such as notifying a security guard near the ATM to check.

[0152] By repeatedly performing such posture estimation processing, the posture of the subject can be estimated for each of the time-series query images.

[0153] Here, in order to accurately estimate the posture of the subject, it is important to set an appropriate reference image. However, multiple reference images are often set, and it is often unclear which reference image is being emphasized for posture estimation. Therefore, even if the posture is estimated incorrectly, it is difficult to estimate the reference image that caused the error.

[0154] The estimation support process is a process of searching for a reference image showing a person whose posture or behavior is similar to that of the subject shown in the query image in order to support estimation of the posture of the subject. When an incorrect posture is estimated, the estimation support process is useful for estimating the reference image that caused the incorrect posture estimation. The estimation support process will be described below with reference to the drawings.

[0155] FIG. 12 is a flowchart showing an example of the estimation support process according to this embodiment.

[0156] The estimation support process starts in response to, for example, a user giving an instruction to start a search by specifying a query image, posture, maximum number of images to be displayed, a first reference value indicating the lower limit of overall similarity, a second reference value indicating the lower limit of element similarity, etc.

[0157] The specified query image is, for example, a query image whose posture has been erroneously estimated in the posture estimation process. The specified posture is the posture to be searched for. The following description will be given using an example that targets a positive example of a talking posture (see FIG. 5). The first and second reference values ​​are set to search for a reference image that is likely to affect the posture estimation result. The second reference value indicates the lower limit of the maximum element similarity among the element similarities related to the second model corresponding to the reference image.

[0158] The same process may be performed for negative examples. Furthermore, either or both of the maximum number of images to be displayed and the reference value may not be set. In this case, if the maximum number of images to be displayed is not set, the search results may be displayed in descending order of similarity, for example.

[0159] See Fig. 12. The posture acquisition unit 113, the similarity acquisition unit 114, and the estimation unit 115, for example, follow instructions from the search unit 117 and execute processes similar to steps S103 to S105 of the posture estimation process.

[0160] In detail, for example, the posture acquisition unit 113 acquires first posture information based on a plurality of reference images associated with the specified posture, which is a call posture, and second posture information based on a specified query image (step S103).

[0161] The similarity obtaining unit 114 obtains the similarity between each of the first models included in the first orientation information and the second orientation information obtained in step S103 and the second model (step S104).

[0162] The estimation unit 115 estimates the posture of the subject shown in the specified query image based on the query image and a plurality of reference images (step S105).

[0163] The search unit 117 searches a plurality of reference images for a reference image showing a person whose posture or behavior is similar to that of the subject shown in the query image (step S201).

[0164] In more detail, for example, the search unit 117 searches for a reference image whose overall similarity is equal to or greater than a first reference value and whose maximum element similarity is equal to or greater than a second reference value. The search unit 117 outputs the search result.

[0165] The display control unit 119 determines whether or not there is a reference image that satisfies the reference value (step S202) based on the search result output from the search unit 117. If there is no reference image that satisfies the reference value (step S202; No), the display control unit 119 ends the estimation assistance process.

[0166] If there is a reference image that satisfies the reference value (step S202; Yes), the display control unit 119 causes the display unit 118 to display the search results and the like (step S203).

[0167] In detail, for example, the display control unit 119 displays at least one of the first posture information (e.g., the first model) and the second posture information (e.g., the second model) on the display unit 119 in a display mode according to an index used to search for reference images. Furthermore, if the number of reference images that satisfy the criterion value exceeds the maximum number to be displayed, the display control unit 119 selects the maximum number of reference images to be displayed in descending order of similarity (e.g., overall similarity) and displays them as search results.

[0168] FIG. 13 is a diagram showing an example of a result display screen SC that displays search results.

[0169] 13, the set value of the first reference value is 0.98, and the set value of the second reference value is 0.5. Furthermore, the "talking posture" is selected as the posture. The result of the estimation regarding the posture of the subject shown in the query image is a positive estimation ("the posture is a talking posture").

[0170] 13 shows an example in which the maximum number of images to display is "3." The result display screen SC includes a query image Q, which is the input image, and reference images A to C, which are search results. Reference images A to C are any of the positive examples (reference image 1 to reference image M) shown in FIG.

[0171] Below each of the query image Q and reference images A to C, the overall similarity, element similarity (maximum value), and element similarity (average value) are displayed.

[0172] In detail, the overall similarity, element similarity (maximum value), and element similarity (average value) with the associated reference image are displayed below the query image Q. In the example shown in Fig. 13, the query image Q is associated with reference image A, which has the highest overall similarity among reference images A to C. In other words, reference image A corresponds to the above-mentioned "specific reference image." Because the query image Q is associated with reference image A, the similarities associated with the query image Q are the same values ​​as the similarities associated with the reference image A.

[0173] 13 shows an example in which the overall similarity between a query image Q and each of reference images A to C is represented by the thickness of the rectangular frame line surrounding the image. In the example shown in Fig. 13, the query image Q and each of reference images A to C are surrounded by three rectangular frame lines of different thicknesses depending on the magnitude of the overall similarity. In this way, by changing the display mode of the first orientation information (e.g., the first model) and the second orientation information (e.g., the second model) depending on the magnitude of the overall similarity, it becomes easier to intuitively grasp the magnitude of the overall similarity.

[0174] 14 is a diagram showing a display example of a query image Q. FIG. 14 shows an example in which a second model is displayed superimposed on the query image Q.

[0175] In the second model shown in FIG. 14 , circles represent joint elements. Lines with arrows connecting the joint elements represent trunk elements and bone elements. The arrows indicate directions away from the base point, which is the center of the upper chest (near the base of the neck). The two lines extending downward from the base point represent trunk elements, and the other lines represent bone elements. The base point may be set as appropriate.

[0176] In the second model shown in FIG. 14 , the trunk elements and bone elements are represented by lines of three different thicknesses and two different line types (solid lines and dotted lines). The thickness corresponds to the weight. The line type corresponds to the element similarity. In detail, the second model shown in FIG. 14 represents the weight in three levels, with the line thickness increasing with the associated weight of the model element. Furthermore, the second model shown in FIG. 14 represents the element similarity between each of the model elements constituting the first model corresponding to the associated reference image A and the second model corresponding to each model element in two levels. A large element similarity is represented by a solid line, and a small element similarity is represented by a dotted line.

[0177] Although not shown, it is preferable that the first model is displayed superimposed on each of the reference images A to C, similar to the query image Q. Also, it is preferable that the model elements of the first model are displayed in different display modes according to the weights and element similarities, similar to the second model.

[0178] In this way, by changing the display mode of the model elements according to the weight and element similarity, it becomes easier to intuitively grasp the magnitude of the weight and element similarity.

[0179] Fig. 15 is a diagram showing another display example of the query image Q. Fig. 15 shows an example in which only the model elements (bone elements, trunk elements) represented by solid lines in Fig. 14 and their associated joint elements, i.e., model elements with high element similarity and their associated joint elements, are displayed on the display unit 118. The query images Q shown in Figs. 14 and 15 are the same except that model elements with low element similarity and their associated joint elements are not displayed.

[0180] 15 has been described as an example in which a query image Q including only model elements with high element similarity and their associated joint elements is displayed on the display unit 118. That is, an example has been described in which model elements with low element similarity and their associated joint elements are not included in the display of the query image Q. However, the method of displaying model elements with low element similarity and their associated joint elements in the display of the query image Q is not limited to this. For example, model elements with low element similarity and their associated joint elements may be displayed thin or small, may be displayed semi-transparently, or may be displayed with a shape (e.g., an X) that is different from the other model elements and their associated joint elements.

[0181] (Operations and Effects) As described above, according to this embodiment, the information processing device 100 includes the attitude acquisition unit 113 , the search unit 117 , and the display control unit 119 .

[0182] The posture acquisition unit 113 acquires first posture information indicating the posture of a person shown in each of a plurality of reference images associated with a predetermined posture, and second posture information indicating the posture of a person shown in a query image. The search unit 117 searches the plurality of reference images for a reference image indicating a person whose posture or behavior is similar to that of the person shown in the query image, based on the similarity between the first posture information and the second posture information. The display control unit 119 displays at least one of the first posture information and the second posture information on the display unit 118 in a display mode corresponding to an index used to search for a reference image.

[0183] As a result, at least one of the first posture information and the second posture information is displayed in a display mode corresponding to the indicator, so that the user can intuitively grasp the indicator used to search for reference images and estimate inappropriate reference images for estimating the posture of the subject. This allows the user to take measures such as deleting inappropriate reference images. Therefore, it is possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0184] According to this embodiment, the indicator includes a similarity. The display control unit 119 causes the display unit 118 to display at least one of the first attitude information and the second attitude information in a display mode according to the similarity.

[0185] As a result, at least one of the first posture information and the second posture information is displayed in a display mode according to the degree of similarity, so that the user can intuitively grasp the degree of similarity and estimate an inappropriate reference image for estimating the posture of the subject. Therefore, the user can take measures such as deleting the inappropriate reference image. Therefore, it is possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0186] According to this embodiment, when displaying the second orientation information, the display control unit 119 associates the second orientation information with a specific reference image from among multiple reference images, and displays the second orientation information on the display unit 118 in a display mode that corresponds to the similarity between the second orientation information and the first orientation information corresponding to the specific reference image.

[0187] As a result, the second posture information is displayed in a display mode according to the similarity, so that the user can intuitively grasp the similarity regarding the second posture information and estimate an inappropriate reference image for estimating the posture of the photographed person. Therefore, the user can take measures such as deleting the inappropriate reference image. Therefore, it is possible to improve the accuracy of estimating the posture of the photographed person shown in the image.

[0188] According to this embodiment, the first pose information includes a first model, which is a pose estimation model for a person shown in the reference image. The second pose information includes a second model, which is a pose estimation model shown for a subject included in the query image. The pose estimation model is composed of a plurality of model elements.

[0189] As a result, at least one of the first model and the second model is displayed in a display mode corresponding to the indicator, so that the user can intuitively grasp the indicator and the posture estimation model used to search for the reference image, and can estimate an inappropriate reference image for estimating the posture of the subject. This allows the user to take measures such as deleting the inappropriate reference image. Therefore, it is possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0190] According to this embodiment, the index includes a weight indicating the degree of importance attached to each model element for determining the similarity between posture estimation models in a given posture. Display control unit 119 causes display unit 118 to display the multiple model elements constituting at least one of the first model and the second model in a display mode according to the weight.

[0191] As a result, at least one of the first model and the second model is displayed in a display mode according to the weight, so that the user can intuitively understand the posture estimation model and weight used to search for reference images and estimate an inappropriate reference image for estimating the posture of the subject. This allows the user to take measures such as deleting the inappropriate reference image. Therefore, it is possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0192] According to this embodiment, the similarity includes at least one of an overall similarity, which is the overall similarity between the first model and the second model, and an element similarity, which is the similarity for each corresponding model element between the first model and the second model.

[0193] As a result, at least one of the first model and the second model is displayed in a display mode corresponding to at least one of the overall similarity and the element similarity. Therefore, the user can intuitively grasp at least one of the first model and the second model used to search for a reference image and at least one of the overall similarity and the element similarity, and can estimate an inappropriate reference image for estimating the posture of the subject. Therefore, the user can take measures such as deleting the inappropriate reference image. Therefore, it is possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0194] According to this embodiment, the index includes a similarity. When displaying a second model, the display control unit 119 associates the second model with a specific reference image from among the multiple reference images, and causes the display unit 118 to display the second model in a display mode according to the similarity between the second model and the first model corresponding to the specific reference image. The first model corresponding to the specific reference image is the first model with the largest overall similarity, the first model with the largest element similarity, or a first model designated by the user.

[0195] As a result, the second model is displayed in a display mode according to the similarity, so the user can intuitively grasp the similarity between the first model and the second model corresponding to a specific reference image and estimate an inappropriate reference image for estimating the posture of the subject. This allows the user to take measures such as deleting the inappropriate reference image. Therefore, it is possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0196] According to this embodiment, when displaying the second model, the display control unit 119 calculates an average value of the element similarity for each model element, and displays the second model on the display unit 118 in a display mode according to the average value.

[0197] As a result, the second model is displayed in a display mode according to the similarity, so the user can intuitively grasp the similarity between the first model and the second model corresponding to a specific reference image and estimate an inappropriate reference image for estimating the posture of the subject. This allows the user to take measures such as deleting the inappropriate reference image. Therefore, it is possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0198] According to this embodiment, when the similarity includes element similarity, the display control unit 119 causes the display unit 118 to display only model elements that constitute at least one of the first model and the second model, whose element similarity is greater than or equal to a first reference value, less than or equal to a second reference value, or whose weight is greater than or equal to a predetermined threshold value.

[0199] This allows only model elements whose element similarity is equal to or greater than the first reference value or equal to or less than the second reference value to be displayed on the display unit 118. This allows the user to intuitively grasp parts that are estimated to be similar overall but are not similar, or parts that are estimated to be dissimilar overall but are similar.

[0200] Furthermore, only model elements whose weights are equal to or greater than a predetermined threshold are displayed on the display unit 118, so that the user can intuitively grasp the parts to which large weights have been set.

[0201] Therefore, the user can take measures such as deleting inappropriate reference images, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0202] According to this embodiment, the information processing device 100 further includes a similarity obtaining unit 114 that obtains the similarity using the first model and the second model.

[0203] This allows the user to intuitively grasp the index for at least one of the first model and the second model used to calculate the similarity by associating it with at least one of the first model and the second model. This allows the user to take measures such as deleting inappropriate reference images. This makes it possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0204] According to this embodiment, the similarity obtaining unit 114 includes an overall calculation unit 114a that obtains the overall similarity between the first model and the second model.

[0205] This allows the user to intuitively grasp the overall similarity of at least one of the first model and the second model used to calculate the overall similarity in association with at least one of these models. This allows the user to take measures such as deleting inappropriate reference images. This makes it possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0206] According to this embodiment, similarity acquisition unit 114 further includes an element calculation unit that calculates element similarity, which is the similarity for each corresponding model element between the first model and the second model. Overall calculation unit 114 a calculates the overall similarity using the element similarity and weights that indicate the degree to which each model element is emphasized in order to calculate the similarity between posture estimation models in a predetermined posture.

[0207] This allows the user to intuitively grasp the overall similarity and element similarity for at least one of the first model and the second model used to calculate the overall similarity and element similarity in association with at least one of these models. This allows the user to take measures such as deleting inappropriate reference images. This makes it possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0208] According to this embodiment, the model elements include joint elements corresponding to the multiple joints, and trunk elements and bone elements corresponding to the trunk and skeleton connecting the multiple joints, respectively. The display control unit 119 displays at least one of the trunk elements and bone elements on the display unit 118 as a line with an arrow.

[0209] Generally, when lines representing bone elements are close to each other on the image displayed on the display unit 118, it may be difficult to understand the correspondence between the lines representing the bone elements and the skeleton of the subject if the lines do not have arrows. By adding arrows to the lines, it is possible to easily associate the lines representing the bone elements with the skeleton of the subject. This makes it easier to intuitively understand. Therefore, it is possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0210] Although the embodiments and modifications of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.

[0211] In addition, although the flowcharts used in the above description show multiple steps (processes) in a sequential order, the order of steps executed in each embodiment is not limited to the order shown. In each embodiment, the order of steps shown in the drawings can be changed as long as it does not cause any problems in terms of content. Furthermore, the above-described embodiments and variations can be combined as long as the content is not contradictory.

[0212] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0213] 1. An information processing device comprising: a posture acquisition means for acquiring first posture information indicating the posture of a person shown in each of a plurality of reference images associated with a predetermined posture, and second posture information indicating the posture of a person shown in a query image; a search means for searching, from the plurality of reference images, for a reference image indicating a person whose posture or behavior is similar to that of the person shown in the query image based on the similarity between the first posture information and the second posture information; and a display control means for displaying at least one of the first posture information and the second posture information on a display means in a display mode corresponding to an index used to search for the reference image. 2. The information processing device described in 1., wherein the index includes the similarity, and the display control means displays at least one of the first posture information and the second posture information on the display means in a display mode corresponding to the similarity. 3. The information processing device described in 2., wherein, when displaying the second posture information, the display control means associates the second posture information with a specific reference image from the plurality of reference images, and displays the second posture information on the display means in a display mode corresponding to the similarity between the second posture information and the first posture information corresponding to the specific reference image. 4. The information processing device described in any one of 1. to 3., wherein the first posture information includes a first model that is a posture estimation model related to a person shown in the reference image, and the second posture information includes a second model that is a posture estimation model shown to a subject included in the query image, and the posture estimation model is composed of a plurality of model elements. 5. The information processing device described in 4., wherein the index includes a weight indicating the degree of importance given to each of the model elements for determining the similarity between the posture estimation models in the predetermined posture, and the display control means causes the display means to display a plurality of model elements constituting at least one of the first model and the second model in a display mode corresponding to the weight. 6. The information processing device described in 4. or 5., wherein the similarity includes at least one of an overall similarity that is an overall similarity between the first model and the second model, and an element similarity that is a similarity for each corresponding model element between the first model and the second model.The information processing device described in any one of 4. to 8., wherein the index includes the similarity, and when displaying the second model, the display control means associates the second model with a specific reference image from the plurality of reference images and displays the second model on the display means in a display manner according to the similarity between the second model and a first model corresponding to the specific reference image, and the first model corresponding to the specific reference image is a first model having the largest overall similarity, a first model having the largest element similarity, or a first model designated by a user. 8. The information processing device described in 6., wherein when displaying the second model, the display control means calculates an average value of the element similarities for each of the model elements and displays the second model on the display means in a display manner according to the average value. 9. The information processing device described in any one of 4. to 8., wherein, when the similarity includes the element similarity, the display control means displays on the display means only model elements constituting at least one of the first model and the second model whose element similarity is equal to or greater than a first reference value, equal to or less than a second reference value, or whose weight is equal to or greater than a predetermined threshold value. 10. The information processing device according to any one of 4. to 9., further including a similarity acquisition means for determining the similarity using the first model and the second model. 11. The information processing device according to 10., wherein the similarity acquisition means includes an overall calculation means for determining the overall similarity between the first model and the second model. 12. The information processing device according to 11., wherein the similarity acquisition means further includes an element calculation means for determining the element similarity, which is the similarity for each corresponding model element between the first model and the second model, and the overall calculation means determines the overall similarity using the element similarity and a weight indicating the degree to which each of the model elements is emphasized in determining the similarity between the posture estimation models in the predetermined posture. 13. The model elements include joint elements corresponding to a plurality of joints, and trunk elements and bone elements corresponding to a trunk and a skeleton connecting the plurality of joints, respectively, and the display control means causes display means to display at least one of the trunk elements and the bone elements as a line with an arrow. 10. The information processing device according to claim 9, wherein:14. An information processing method in which a computer acquires first posture information indicating the posture of a person shown in each of a plurality of reference images associated with a predetermined posture and second posture information indicating the posture of a person shown in a query image, searches the plurality of reference images for a reference image indicating a person whose posture or behavior is similar to that of the person shown in the query image based on the similarity between the first posture information and the second posture information, and displays at least one of the first posture information and the second posture information on a display means in a display mode corresponding to an index used to search for the reference image. 16. A recording medium having recorded thereon a program for causing a computer to: acquire first posture information indicating the posture of a person shown in each of a plurality of reference images associated with a predetermined posture, and second posture information indicating the posture of a person shown in a query image; search for a reference image from among the plurality of reference images that indicates a person whose posture or behavior is similar to that of the person shown in the query image based on the similarity between the first posture information and the second posture information; and display at least one of the first posture information and the second posture information on a display means in a display mode corresponding to an index used to search for the reference image. A program for causing a computer to acquire first posture information indicating the posture of a person shown in each of a plurality of reference images associated with a predetermined posture, and second posture information indicating the posture of a person shown in a query image; searching for a reference image from among the plurality of reference images that indicates a person whose posture or behavior is similar to that of the person shown in the query image, based on the similarity between the first posture information and the second posture information; and displaying at least one of the first posture information and the second posture information on a display means in a display mode corresponding to an index used to search for the reference image.

[0214] REFERENCE SIGNS LIST 100 Information processing device 101 Photography unit 102 Analysis device 111 Image acquisition unit 112 Storage unit 112a Reference information 112b Weight information 113 Orientation acquisition unit 114 Similarity acquisition unit 114a Overall calculation unit 114b Element calculation unit 115 Estimation unit 116 Input unit 117 Search unit 118 Display unit 119 Display control unit

Claims

1. a posture acquiring means for acquiring first posture information indicating a posture of a person shown in each of a plurality of reference images associated with a predetermined posture, and second posture information indicating a posture of a person shown in a query image; a search means for searching the plurality of reference images for a reference image showing a person having a similar posture or behavior to a subject shown in the query image, based on a similarity between the first posture information and the second posture information; a display control means for causing a display means to display at least one of the first attitude information and the second attitude information in a display mode corresponding to an index used to search for the reference image; Information processing device.

2. The index includes the similarity, The display control means causes the display means to display at least one of the first attitude information and the second attitude information in a display mode according to the similarity. The information processing device according to claim 1 .

3. When displaying the second orientation information, the display control means associates the second orientation information with a specific reference image among the plurality of reference images, and causes the display means to display the second orientation information in a display mode according to the degree of similarity between the second orientation information and the first orientation information corresponding to the specific reference image. The information processing device according to claim 2 .

4. the first pose information includes a first model that is a pose estimation model for a person shown in the reference image; the second pose information includes a second model that is a pose estimation model shown to a subject included in the query image; The posture estimation model is composed of a plurality of model elements. The information processing device according to claim 1 .

5. the index includes a weight indicating a degree of importance to each of the model elements for determining a similarity between the pose estimation models in the predetermined pose; The display control means causes the display means to display a plurality of model elements constituting at least one of the first model and the second model in a display mode according to the weights. The information processing device according to claim 4.

6. The similarity includes at least one of an overall similarity, which is an overall similarity between the first model and the second model, and an element similarity, which is a similarity for each corresponding model element between the first model and the second model. The information processing device according to claim 4.

7. The index includes the similarity, when displaying the second model, the display control means associates the second model with a specific reference image among the plurality of reference images, and causes the display means to display the second model in a display mode according to the degree of similarity between the second model and the first model corresponding to the specific reference image; The first model corresponding to the particular reference image is a first model having the largest overall similarity, a first model having the largest element similarity, or a first model designated by a user. The information processing device according to claim 6.

8. When displaying the second model, the display control means calculates an average value of the element similarity for each of the model elements, and causes the display means to display the second model in a display mode corresponding to the average value. The information processing device according to claim 6.

9. The computer obtain first pose information indicating a pose of a person shown in each of a plurality of reference images associated with a predetermined pose, and second pose information indicating a pose of a person shown in a query image; searching for a reference image showing a person having a similar posture or behavior to the subject shown in the query image from among the plurality of reference images based on a similarity between the first posture information and the second posture information; At least one of the first attitude information and the second attitude information is displayed on a display means in a display mode corresponding to an index used to search for the reference image. Information processing methods.

10. On the computer, obtain first pose information indicating a pose of a person shown in each of a plurality of reference images associated with a predetermined pose, and second pose information indicating a pose of a person shown in a query image; searching for a reference image showing a person having a similar posture or behavior to the subject shown in the query image from among the plurality of reference images based on a similarity between the first posture information and the second posture information; A program for causing a display means to display at least one of the first attitude information and the second attitude information in a display mode corresponding to an index used to search for the reference image.