Information processing device, information processing method, and program

Abstract

An information processing device (100) comprises an estimation unit (115) and a display control unit (119). The estimation unit (115) estimates, on the basis of a plurality of query images obtained by imaging the subject to be imaged multiple times who performs a prescribed action and a reference image that shows the person associated with a prescribed posture, the posture of the imaged subject shown in each of the plurality of query images. When a query image is included in the plurality of query images, the estimation result of which is different, the display control unit (119) causes a display unit to display the query image with which a different estimation was made.

Description

Information processing device, information processing method, information processing system, and recording medium

[0001] The present invention relates to an information processing device, an information processing method, an information processing system, and a recording medium.

[0002] For example, the image search device described in Patent Document 1 includes a posture estimation unit, a feature extraction unit, a query generation unit, and an image search unit.

[0003] The posture estimation unit described in the document recognizes posture information of a search target, which is composed of multiple feature points, from an input image. The feature extraction unit described in the document extracts features from the posture information and the input image. The query generation unit described in the document generates a search query from an image database that stores feature amounts associated with input images and posture information specified by a user. The image search unit described in the document searches the image database for images containing similar postures according to the search query.

[0004] Patent Literature 2 describes a technology for calculating feature amounts for each of a plurality of key points of a human body included in an image, searching for images containing human bodies with similar postures or movements based on the calculated feature amounts, and classifying images with similar postures or movements together. Non-Patent Literature 1 describes a technology related to human skeleton estimation.

[0005] JP 2019-091138 A International Publication No. 2021 / 084677

[0006] Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields];, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, P. 7291-7299

[0007] Patent Literature 1 describes a technique for estimating posture or behavior based on an image. However, in Patent Literature 1, it is difficult to improve the accuracy of estimating the posture of a subject shown in an image because it is not clear whether the posture has been estimated correctly.

[0008] Note that neither Patent Document 2 nor Non-Patent Document 1 discloses a technique for improving the accuracy of detecting a person in a predetermined posture from an image of the person.

[0009] In view of the above-mentioned problems, one example of the object of the present invention is to provide an information processing device, an information processing method, an information processing system, and a recording medium that solve the problem of improving the accuracy of estimating the posture of a person shown in an image.

[0010] According to one aspect of the present invention, there is provided an information processing device comprising: an estimation means for estimating a posture of a person shown in each of a plurality of query images based on a plurality of query images obtained by taking multiple photographs while the person is performing a predetermined behavior and a reference image showing the person associated with a predetermined posture; and a display control means for, when a query image with a different estimation result is included among the plurality of query images, displaying the query image with the different estimation result on a display means.

[0011] According to one aspect of the present invention, there is provided an information processing system including: the information processing device described above; and one or more imaging units that perform the plurality of imaging operations.

[0012] According to one aspect of the present invention, there is provided an information processing method in which a computer estimates a posture of a person shown in each of a plurality of query images based on a plurality of query images obtained by photographing the person multiple times while the person is performing a predetermined action and a reference image showing the person corresponding to a predetermined posture, and if the plurality of query images include a query image with a different estimation result, the query image with the different estimation result is displayed on a display means.

[0013] According to one aspect of the present invention, a recording medium having recorded thereon a program for causing a computer to estimate the posture of a person shown in each of a plurality of query images based on a plurality of query images obtained by photographing a person multiple times while the person is performing a predetermined behavior and a reference image showing the person corresponding to a predetermined posture, and, if a query image with a different estimation result is included among the plurality of query images, displaying the query image with the different estimation result on a display means is provided.

[0014] According to one aspect of the present invention, it is possible to provide an information processing device, an information processing method, an information processing system, and a recording medium that solve the problem of improving the accuracy of estimating the posture of a person being photographed shown in an image.

[0015] 1 is a diagram illustrating an overview of an information processing device according to embodiment 1. FIG. 2 is a diagram illustrating an overview of an information processing system according to embodiment 1. FIG. 3 is a flowchart illustrating an overview of information processing according to embodiment 1. FIG. 4 is a diagram illustrating a detailed functional configuration example of an information processing system according to embodiment 1. FIG. 5 is a diagram illustrating a configuration example of reference information including a reference image associated with a calling posture. FIG. 6 is a diagram illustrating a configuration example of weight information indicating a weight associated with a calling posture. FIG. 7 is a diagram illustrating a functional configuration example of a similarity acquisition unit according to embodiment 1. FIG. 8 is a diagram illustrating an example physical configuration of an information processing device according to embodiment 1. FIG. 9 is a flowchart illustrating an example of posture estimation processing according to embodiment 1. FIG. 10 is a diagram illustrating an example of a method of thinning out a portion from a plurality of frame images. FIG. 11 is a flowchart illustrating a detailed example of similarity acquisition processing according to embodiment 1. FIG. 12 is a flowchart illustrating an example of estimation support processing according to embodiment 1. FIG. 13 is a diagram illustrating an example of an erroneous estimation pattern. FIG. 14 is a diagram illustrating a detailed functional configuration example of an information processing system S2 according to embodiment 2.

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0017] 1 is a diagram showing an overview of an information processing device 100 according to embodiment 1. The information processing device 100 includes an estimation unit 115 and a display control unit 119.

[0018] The estimation unit 115 estimates the posture of the person being photographed shown in each of the multiple query images based on multiple query images obtained by taking multiple photographs while the person is performing a specified action and a reference image showing a person associated with a specified posture.

[0019] When a query image with a different estimation result is included among the plurality of query images, the display control unit 119 causes the display unit to display the query image with the different estimation result.

[0020] According to this information processing device 100, it is possible to provide an information processing device 100 that solves the problem of improving the accuracy of estimating the posture of a person to be photographed shown in an image.

[0021] 2 is a diagram showing an overview of an information processing system S1 according to embodiment 1. The information processing system S1 includes an information processing device 100 and one or more image capturing units 101 that capture images multiple times.

[0022] According to this information processing system S1, it is possible to provide an information processing system S1 that solves the problem of improving the accuracy of estimating the posture of a person to be photographed shown in an image.

[0023] FIG. 3 is a flowchart showing an outline of information processing according to the first embodiment.

[0024] The estimation unit 115 estimates the posture of the person being photographed shown in each of the multiple query images based on multiple query images obtained by taking multiple photographs while the person is performing a specified action and a reference image showing a person associated with a specified posture (step S105).

[0025] When a query image with a different estimation result is included among the plurality of query images, the display control unit 119 causes the display unit to display the query image with the different estimation result (step S202).

[0026] According to this information processing, it is possible to provide an information processing system S1 that solves the problem of improving the accuracy of estimating the posture of a person to be photographed shown in an image.

[0027] (Details) A detailed example of the information processing system S1 according to the first embodiment will be described below.

[0028] 4 is a diagram showing a detailed example of the functional configuration of the information processing system S1 according to embodiment 1. The information processing system S1 includes an imaging unit 101, an information processing device 100, and an analysis device 102. The imaging unit 101, the information processing device 100, and the analysis device 102 are connected via a network N that may be wired, wireless, or a combination of these, and can transmit and receive information to and from each other.

[0029] The photographing unit 101 photographs a person (subject) performing a predetermined action. The photographing unit 101 is, for example, a camera installed in a branch of a financial institution such as a bank, and photographs an operator operating an automated teller machine (ATM).

[0030] In addition, the photographing unit 101 is not limited to a camera for photographing an ATM operator, but may be a camera for photographing the inside of a store such as a bank, or a camera installed in various stores other than financial institutions.

[0031] The image capturing unit 101 captures an image of a predetermined image capturing area and transmits image information representing a moving image to the information processing device 100 .

[0032] Specifically, the image capturing unit 101 captures images multiple times in chronological order at a predetermined frame rate. The image capturing unit 101 generates frame information including frame images for each capture. The image capturing unit 101 transmits the frame information including each of the chronological frame images to the information processing device 100 via the network N.

[0033] The analysis device 102 is a device that analyzes images. The analysis device 102 acquires image information generated by the imaging unit 101 via the network N. In this embodiment, an example will be described in which the analysis device 102 acquires image information from the imaging unit 101 via the information processing device 100, but the analysis device 102 may also acquire image information directly from the imaging unit 101.

[0034] The analysis device 102 is a device that analyzes the image included in the acquired image information.

[0035] Specifically, the analysis device 102 has one or more analysis functions that perform processing (analysis processing) to analyze an image. The analysis functions that the analysis device 102 has include one or more of: (1) an object detection function, (2) a face analysis function, (3) a human figure analysis function, (4) a posture analysis function, (5) a behavior analysis function, (6) an appearance attribute analysis function, (7) a gradient feature analysis function, (8) a color feature analysis function, and (9) a movement line analysis function.

[0036] (1) The object detection function detects people and objects from an image. The object detection function can also determine the positions of people and objects within an image. YOLO (You Only Look Once) is an example of a model that can be applied to object detection processing. The object detection function detects, for example, an operator, a mobile phone (including a smartphone), a wheelchair, etc. Furthermore, for example, the object detection function determines the positions of detected people and objects.

[0037] (2) The face analysis function detects human faces from images, extracts the features of the detected faces (facial feature values), and classifies the detected faces (classification). The face analysis function can also determine the position of the face within the image. The face analysis function can also determine the identity of people detected from different images based on the similarity between the facial feature values ​​of people detected from different images.

[0038] (3) The human morphology analysis function extracts the physical characteristics of people included in an image (for example, values ​​indicating overall characteristics such as whether they are fat or thin, their height, and their clothing), and classifies (classifies) people included in an image. The human morphology analysis function can also identify the position of a person within an image. The human morphology analysis function can also determine the identity of people included in different images based on the physical characteristics of the people included in different images.

[0039] (4) The posture analysis function generates posture information indicating the posture of a person. The posture information includes, for example, a posture estimation model of a person. The posture estimation model is a model connecting the joints of a person estimated from an image. The posture estimation model is composed of multiple model elements corresponding to joint elements corresponding to joints, trunk elements corresponding to the torso, bone elements corresponding to bones connecting the joints, etc. The posture analysis function, for example, detects the joint points of a person from an image and connects the joint points to create a posture estimation model.

[0040] The posture analysis function then uses information from the posture estimation model to estimate a person's posture, extract features of the estimated posture (posture features), and classify (classify) people included in the images. The posture analysis function can also determine the identity of people included in different images based on the posture features of people included in different images.

[0041] For example, the posture analysis function creates posture estimation models such as a talking posture and a wheelchair posture, and extracts posture features for those postures. The talking posture is the posture of talking on a mobile phone. The wheelchair posture is the posture of a person using a wheelchair.

[0042] The posture analysis function can apply, for example, the techniques disclosed in Patent Document 2 and Non-Patent Document 1. (5) The behavior analysis process can estimate a person's movements using information from a posture estimation model, changes in posture, etc., extract features of the person's movements (movement features), and classify (classify) people included in an image. The behavior analysis process can also estimate a person's height and identify the person's position in an image using information from a stick figure model. The behavior analysis process can estimate, for example, behaviors such as changes or transitions in posture and movements (changes or transitions in position) from an image and extract movement features of the behavior.

[0043] (6) The appearance attribute analysis function can recognize appearance attributes associated with a person. The appearance attribute analysis function extracts features (appearance attribute features) related to the recognized appearance attributes and classifies (classifies) people included in the image. Appearance attributes are attributes related to appearance, and include, for example, one or more of clothing color, shoe color, hairstyle, and whether or not a hat, tie, or glasses is worn.

[0044] (7) The gradient feature analysis function extracts gradient feature quantities (gradient feature quantities) in an image. For gradient feature detection processing, techniques such as SIFT, SURF, RIFF, ORB, BRISK, CARD, and HOG can be applied.

[0045] (8) The color feature analysis function can detect objects from an image, extract color features of the detected objects, and classify the detected objects. The color features are, for example, color histograms. The color feature analysis function can detect, for example, people and objects included in the image.

[0046] (9) The flow line analysis function can determine the flow line (trajectory of movement) of a person included in a video, for example, using the result of the identity determination in any of the above-described analysis functions (2) to (6). In more detail, for example, by connecting people determined to be the same in different images in a time series, the flow line of that person can be determined. Note that, in cases where video images captured by multiple image capture units 101 capturing different shooting areas are acquired, the flow line analysis function can also determine the flow line spanning multiple videos captured in different shooting areas.

[0047] Image features include, for example, the results of object detection by the object detection function, facial features, human body features, posture features, movement features, appearance attribute features, gradient features, color features, and movement lines.

[0048] Each of the analysis functions (1) to (9) may appropriately use the results of analysis performed by other analysis functions. The information processing device 100 may include an analysis unit having the functions of the analysis device 102.

[0049] The information processing device 100 according to the first embodiment is a device that estimates the posture of a person included in a frame image. As shown in Fig. 4 , the information processing device 100 functionally includes an image acquisition unit 111, a storage unit 112, a posture acquisition unit 113, a similarity acquisition unit 114, an estimation unit 115, an input unit 116, a determination unit 117, a display unit 118, and a display control unit 119.

[0050] The image acquisition unit 111 acquires image information indicating a moving image from the imaging unit 101. That is, the image acquisition unit 111 acquires a plurality of frame images in time series obtained by capturing images a plurality of times consecutively in time series.

[0051] More specifically, the image acquisition unit 111 acquires frame information including each of a plurality of frame images in time series from the imaging unit 101. The image acquisition unit 111 stores the acquired frame information.

[0052] The storage unit 112 is a storage unit for storing various types of information, and stores, in advance, for example, reference information 112a indicating a reference image, weight information 112b indicating a weight, and the like.

[0053] The reference image is an image of a person associated with a predetermined posture. The reference image is an image that is referred to in order to estimate the posture of a person included in a query image, and is appropriately selected and set in the storage unit 112. The predetermined posture is, for example, a conversation posture, a wheelchair posture, etc.

[0054] 5 is a diagram showing an example of the configuration of reference information 112a including reference images associated with talking postures. The reference information 112a shown in FIG. 5 includes, for example, positive examples and negative examples.

[0055] The positive examples are reference images of a person in a predetermined posture. The positive examples illustrated in Fig. 5 (i.e., reference images 1 to 4) are reference images of a person in a conversation posture, such as a person standing and talking on a mobile phone with their right or left hand.

[0056] Negative examples are reference images of people who are not in a predetermined posture. Images of people who are not in a predetermined posture but are similar to the predetermined posture may be selected as negative examples. The negative examples illustrated in FIG. 5 (i.e., reference images 5 to 7) are reference images of people who are not in a conversation posture, such as a person standing upright without a mobile phone.

[0057] The reference information 112a may include any number of reference images as long as it includes at least one reference image. The reference information 112a may also include only positive examples.

[0058] The weight is a value indicating the degree of importance given to each model element for determining the similarity between posture estimation models in a given posture. Weight information 112b includes the weight of each model element for each given posture.

[0059] FIG. 6 is a diagram showing an example of the configuration of weight information 112b indicating weights associated with a call posture. The weight information 112b shown in FIG. 5 associates element IDs and weights in a call posture. The element IDs are information for identifying model elements. The element IDs are, for example, numbers appropriately assigned to bone elements and joint elements corresponding to a trunk element, the upper and lower parts of the left and right arms, the thighs and lower legs, etc. Weights are determined for each model element of a given posture. While FIG. 5 shows an example in which the weights are integers equal to or greater than 0, the method for setting weights may be changed as appropriate.

[0060] For example, in a talking posture, the weight set for the arms is greater than the weight set for the legs because the mobile phone is held while talking, and in a talking posture when talking with the right hand, the weight set for the right hand is greater than the weight set for the left hand.

[0061] The posture acquisition unit 113 acquires, from the storage unit 112, a plurality of reference images associated with a predetermined posture, such as a talking posture, and acquires first posture information based on the acquired plurality of reference images.

[0062] The first pose information is information indicating the pose of a person shown in each of a plurality of reference images associated with a predetermined pose. The first pose information includes, for example, a first model that is a pose estimation model for the person shown in the reference image.

[0063] Furthermore, the orientation acquisition unit 113 acquires time-series frame images from the image acquisition unit 111 and acquires a query image by thinning out some of the time-series frame images. Then, the orientation acquisition unit 113 acquires second orientation information based on the acquired query image.

[0064] The second pose information is information indicating the pose of the subject shown in the query image. The second pose information includes, for example, a second model that is a pose estimation model for the subject shown in the query image.

[0065] In detail, for example, the posture acquisition unit 113 transmits each of the acquired reference image and query image to the analysis device 102 via the network N. When the reference image is transmitted to the analysis device 102, the posture acquisition unit 113 acquires first posture information including a first model related to the person shown in the reference image from the analysis device 102. When the query image is transmitted to the analysis device 102, the posture acquisition unit 113 acquires second posture information including a second model related to the person shown in the query image from the analysis device 102.

[0066] The similarity obtaining unit 114 obtains the similarity in the posture between the subject shown in the query image and the person shown in the reference image for each combination of a time-series query image and a plurality of reference images corresponding to a predetermined posture.

[0067] The similarity is a value that indicates the degree of similarity between pose estimation models in a given pose.

[0068] For example, the similarity acquisition unit 114 acquires a first model of a person shown in each of a plurality of reference images corresponding to a predetermined posture from the posture acquisition unit 113. The similarity acquisition unit 114 also acquires a second model of a person shown in each of the time-series query images from the posture acquisition unit 113. For each combination of the first model and the second model, the similarity acquisition unit 114 calculates the similarity using the first model and the second model.

[0069] The similarity includes an overall similarity and an element similarity.

[0070] The overall similarity is a value indicating the overall degree of similarity between the first model and the second model in a given posture, that is, the overall similarity between the first model and the second model.

[0071] The element similarity is the similarity for each corresponding model element between the first model and the second model in a predetermined posture, that is, the similarity for each corresponding model element between the first model and the second model.

[0072] The similarity may include at least one of the overall similarity and the element similarity.

[0073] 7 is a diagram showing an example of the functional configuration of the similarity acquisition unit 114 according to this embodiment. The similarity acquisition unit 114 includes an overall calculation unit 114a and an element calculation unit 114b.

[0074] The overall calculation unit 114a calculates the overall similarity between the first model and the second model. Specifically, the overall calculation unit 114a calculates the overall similarity using weights corresponding to the predetermined postures included in the weight information 112b and the element similarities calculated by the element calculation unit 114b.

[0075] For example, when overall calculation unit 114a obtains the similarity of each model element from element calculation unit 114b, it calculates the product of each similarity of the model element and the weight of the corresponding model element, and sums up the products obtained for each model element constituting the posture estimation model. The value obtained as a result of this summation is the overall similarity.

[0076] The element calculation unit 114b calculates element similarity, which is the degree of similarity for each corresponding model element between the first model and the second model. For example, the element calculation unit 114b calculates element similarity for each corresponding model element between the first model and the second model based on the size, length, inclination, etc.

[0077] The estimation unit 115 estimates the posture of the person being photographed shown in each of the multiple query images based on multiple query images obtained by taking multiple photographs while the person is performing a specified action and a reference image showing a person associated with a specified posture.

[0078] For example, the estimation unit 115 estimates the posture of the person being photographed shown in each of the time-series query images based on the similarity (for example, overall similarity) obtained by the similarity acquisition unit 114 .

[0079] In addition, the estimation unit 115 may estimate the posture of the person being photographed shown in at least one thinned frame image from the time series of frame images based on the at least one thinned frame image and a reference image.

[0080] In this case, the posture acquisition unit 113 acquires at least one of the thinned frame images from the image acquisition unit 111 and acquires a second model of the person shown in the frame image. The similarity acquisition unit 114 calculates an overall similarity based on the second model of the person shown in the frame image and the first models of the person shown in each of the multiple reference images. Then, the estimation unit 115 estimates the posture of the person shown in the at least one thinned frame image based on the overall similarity calculated by the similarity acquisition unit 114.

[0081] There are various methods for the estimation unit 115 to estimate the posture of the person being photographed based on the similarity. As examples, posture estimation methods 1 to 5 will be described below.

[0082] (Posture Estimation Method 1) For example, the estimation unit 115 may estimate the posture of the person shown in the query image or the frame image based on the reference image with the highest similarity among the positive examples and negative examples. In this case, for example, if the reference image with the highest similarity is a positive example, the estimation unit 115 estimates that the posture of the person is a predetermined posture corresponding to the reference image. If the reference image with the highest similarity is a negative example, the estimation unit 115 estimates that the posture of the person is not the predetermined posture corresponding to the reference image.

[0083] (Posture Estimation Method 2) For example, the estimation unit 115 may estimate the posture of the photographed person shown in the query image or the frame image based on the average positive example value and the average negative example value. The average positive example value is the average value of the similarity between the query image or the frame image and multiple positive examples associated with a predetermined posture. The average negative example value is the average value of the similarity between the query image or the frame image and multiple negative examples associated with a predetermined posture.

[0084] In this case, for example, if the positive example average value is greater than the negative example average value, the estimation unit 115 estimates that the posture of the photographed person is the predetermined posture corresponding to the reference image. If the positive example average value is equal to or less than the negative example average value, the estimation unit 115 estimates that the posture of the photographed person is not the predetermined posture corresponding to the reference image.

[0085] (Posture Estimation Method 3) Furthermore, for example, the estimation unit 115 may perform image matching between a query image or a frame image and a reference image, and estimate the posture of the person shown in the query image or the frame image based on the similarity between the reference image that matches in the image matching and the query image or the frame image. In this case, the estimation unit 115 may estimate the posture of the person shown in the query image or the frame image based on the average positive example value and the average negative example value among the reference images that match in the image matching.

[0086] In detail, for example, when the average value of positive examples among the reference images matched by image matching is greater than the average value of negative examples, the estimation unit 115 estimates that the posture of the photographed person is a predetermined posture corresponding to the reference images. When the average value of positive examples among the reference images matched by image matching is equal to or less than the average value of negative examples, the estimation unit 115 estimates that the posture of the photographed person is not a predetermined posture corresponding to the reference images.

[0087] Note that various known techniques may be applied to image matching. For example, the estimation unit 115 calculates the similarity between images based on feature amounts between the subject shown in the query image or frame image and the person shown in the reference image. The estimation unit 115 compares the similarity between the images with a threshold to determine whether the query image or frame image and the reference image match. For example, the estimation unit 115 determines that the images match (are similar) when the similarity between the images is equal to or greater than the threshold, and determines that the images do not match (are dissimilar) when the similarity between the images is less than the threshold.

[0088] (Posture Estimation Method 4) The estimation unit 115 may estimate the posture of the person being photographed using only image matching, without using similarity.

[0089] For example, the estimation unit 115 may estimate that the subject shown in the query image or the frame image is in a predetermined posture corresponding to the reference image when the query image or the frame image matches at least one positive example in the image matching. Alternatively, for example, the estimation unit 115 may estimate that the subject shown in the query image or the frame image is not in a predetermined posture corresponding to the reference image when the query image or the frame image matches at least one negative example in the image matching.

[0090] Furthermore, for example, the estimation unit 115 may estimate the posture of the subject based on the number of matches between positive examples and negative examples that match in image matching. In this case, for example, if the number of matches with positive examples is greater than the number of matches with negative examples, the estimation unit 115 estimates that the subject shown in the query image or frame image is in a predetermined posture corresponding to the reference image. Also, for example, if the number of matches with positive examples is equal to or less than the number of matches with negative examples, the estimation unit 115 estimates that the subject shown in the query image or frame image is not in a predetermined posture corresponding to the reference image.

[0091] In image matching, if a query image or a frame image does not match either a positive example or a negative example, the estimation unit 115 may determine that the query image or the frame image is different from either a positive example or a negative example, or may determine that the query image or the frame image is a negative example.

[0092] (Pose estimation method 5) The estimation unit 115 may determine whether a query image or the frame image conforms to (is similar to) a positive example or a negative example by using a learning model that has been trained through machine learning using a reference image. This learning model is a learning model that has been trained through machine learning for determining whether the subject conforms to a positive example or a negative example. In this case, the estimation unit 115 inputs image information including a query image or the frame image that shows the subject into the learning model, and obtains a determination result as to whether the query image or the frame image conforms to (is similar to) a positive example or a negative example.

[0093] The input data to the learning model during learning includes image information of a person. Furthermore, learning may be performed using supervised learning that includes labels (correct answers) that indicate whether the input data matches a positive example or a negative example.

[0094] The input unit 116 is a keyboard, a mouse, a touch panel, or the like that accepts input from the user.

[0095] The determination unit 117 determines, based on the estimation result by the estimation unit 115, whether or not a query image with different estimation results is included among the plurality of query images.

[0096] In detail, for example, the determination unit 117 determines whether or not the results of estimation for a plurality of query images correspond to a predetermined erroneous estimation pattern.

[0097] The erroneous estimation pattern is a pattern of the estimation result related to the posture of the photographed person included in each of the plurality of query images. The erroneous estimation pattern is, for example, determined in advance and stored in the storage unit 112.

[0098] The erroneous estimation pattern may include, for at least one query image, an estimation result that is different from that of the other query images. Thus, in step S201, the determination unit 117 determines whether the estimation result of the estimation unit 115 corresponds to the erroneous estimation pattern, thereby determining whether the plurality of query images includes a query image with a different estimation result.

[0099] The display unit 118 is a display or the like that displays various types of information. The display control unit 119 controls the display unit 118 and causes the display unit 118 to display various types of information. For example, when the estimation unit 115 detects a person to be photographed in a predetermined posture, the display control unit 119 causes the display unit 118 to display a query image or a frame image in which the person to be photographed is marked. The mark is, for example, a rectangular frame that surrounds the person to be photographed.

[0100] In addition, for example, based on the result of judgment by the judgment unit 117, if a query image with a different estimation result is included among the multiple query images, the display control unit 119 causes the query image with the different estimation to be displayed on the display unit 118.

[0101] (Physical configuration of information processing system S1) The information processing system S1 is physically composed of an imaging unit 101, an information processing device 100, and an analysis device 102, which are connected via a network N. The imaging unit 101, the information processing device 100, and the analysis device 102 are each composed of a physically different single device. The imaging unit 101 is, for example, a camera.

[0102] The information processing device 100 and the analysis device 102 may be physically configured as a single device, in which case the information processing device 100 and the analysis device 102 are connected using an internal bus 1010 (described later) instead of the network N. Furthermore, one or both of the information processing device 100 and the analysis device 102 may be physically configured as multiple devices connected via an appropriate communication line such as the network N.

[0103] 8 is a diagram showing an example of the physical configuration of an information processing device 100 according to this embodiment. The information processing device 100 is, for example, a general-purpose computer. The information processing device 100 includes, for example, a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070.

[0104] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0105] The processor 1020 is implemented as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0106] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0107] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules for realizing each function of the information processing device 100. The processor 1020 reads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module.

[0108] The network interface 1050 is an interface for connecting the information processing device 100 to the network N.

[0109] The input interface 1060 is an interface for a user to input information, and is configured with one or more of, for example, a keyboard, a mouse, a touch panel, and the like.

[0110] The output interface 1070 is an interface for presenting information to the user, and is configured, for example, by a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like.

[0111] The analysis device 102 is, for example, a general-purpose computer, and has a physical configuration similar to that of the information processing device 100 (see FIG. 8).

[0112] Storage device 1040 of analysis device 102 stores program modules for realizing each function of analysis device 102. Processor 1020 of analysis device 102 reads each of these program modules into memory 1030 and executes them, thereby realizing each function corresponding to the program module. Network interface 1050 of analysis device 102 is an interface for connecting analysis device 102 to network N. Except for these points, analysis device 102 may be physically configured in the same way as information processing device 100.

[0113] (Operation of Information Processing System S1) The information processing system S1 according to this embodiment executes information processing for estimating the posture of a subject included in a query image. The information processing executed by the information processing system S1 includes posture estimation processing and estimation support processing.

[0114] The posture estimation process is a process for estimating the posture of the subject included in the query image using a reference image associated with a predetermined posture. The estimation support process is a process for supporting the estimation of the posture of the subject.

[0115] 9 is a flowchart showing an example of the posture estimation process according to this embodiment. The posture estimation process is executed, for example, while the information processing system S1 is in operation.

[0116] The image acquisition unit 111 acquires a plurality of frame images in time series (step S101), and stores the acquired frame images.

[0117] More specifically, for example, the image acquisition unit 111 sequentially acquires a plurality of frame images in time series from time T1 to just before time T2, where time T2-time T1=time interval ΔT.

[0118] The image acquisition unit 111 thins out some of the frame images acquired in step S101 to acquire a query image (step S102).

[0119] In more detail, for example, the image acquisition unit 111 thins out some of the multiple frame images according to a predetermined rule. For example, FIG. 10 is a diagram illustrating an example of a method for thinning out some of the multiple frame images. As illustrated in FIG. 10, the image acquisition unit 111 thins out frame images acquired during a predetermined time interval ΔT (excluding both ends). As a result, the image acquisition unit 111 acquires query images in a time series at a predetermined constant time interval ΔT. Note that the method for thinning out some of the multiple frame images to acquire query images is not limited to this. For example, the time interval ΔT does not need to be constant and may be changed depending on the operation mode (a mode for tracking a subject, a mode for detecting the subject's posture). Alternatively, the query images may be a plurality of unthinned frame images.

[0120] The orientation acquisition unit 113 acquires first orientation information based on a plurality of reference images associated with a predetermined orientation, and second orientation information based on the query image acquired in step S102 (step S103).

[0121] In detail, for example, posture acquisition unit 113 acquires a plurality of reference images corresponding to a predetermined posture from storage unit 112. When the predetermined posture is a call posture and reference information 112a shown in FIG. 5 is stored in storage unit 112, posture acquisition unit 113 acquires reference image 1 to reference image 7. Posture acquisition unit 113 transmits the acquired reference image 1 to reference image 7 to analysis device 102. In response to this, analysis device 102 generates first posture information including a first model of a person represented by each of reference image 1 to reference image 7 and transmits the first posture information to information processing device 100. Posture acquisition unit 113 acquires the first posture information from analysis device 102.

[0122] The posture acquisition unit 113 acquires the query image acquired in step S102 from the image acquisition unit 111. The posture acquisition unit 113 transmits the acquired query image to the analysis device 102. In response to this, the analysis device 102 generates second posture information including a second model of the subject indicated by the query image and transmits the second posture information to the information processing device 100. The posture acquisition unit 113 acquires the second posture information from the analysis device 102.

[0123] The similarity obtaining unit 114 obtains the similarity between each of the first models included in the first orientation information and the second orientation information obtained in step S103 and the second model (step S104).

[0124] FIG. 11 is a flowchart showing a detailed example of the similarity obtaining process (step S104) according to this embodiment.

[0125] The element calculation unit 114b repeats steps S104b to S104c for each of the first models included in the first orientation information acquired in step S103 (step S104a).

[0126] The element calculation unit 114b calculates element similarity, which is the similarity for each corresponding model element between the first model and the second model (step S104b).

[0127] The overall calculation unit 114a acquires the weight information 112b stored in the memory unit 112, and calculates the overall similarity between the first model and the second model based on the element similarity and weight for each model element calculated in step S104b (step S104c).

[0128] For example, the overall calculation unit 114a calculates the sum of the products of the element similarities and weights of the corresponding model elements, and sets this sum as the overall similarity.

[0129] Overall calculation unit 114a repeats steps S104b to S104c for each of the first models included in the first posture information acquired in step S103, and as a result, calculates the overall similarity between each of the first models and the second model. After executing steps S104b to S104c for each of the first models included in the first posture information acquired in step S103, overall calculation unit 114a ends loop A (step S104a) and returns to the posture estimation process.

[0130] Referring again to Fig. 9, the estimation unit 115 estimates the posture of the person being photographed shown in the query image based on the query image acquired in step S102 and a plurality of reference images (step S105).

[0131] For example, the estimation unit 115 estimates the posture of the person shown in the query image acquired in step S102 based on the overall similarity between the query image and each of reference images 1 to 7. Note that in step S105, the estimation unit 115 may use any of the posture estimation methods 1 to 5 described above, or may use a method other than posture estimation methods 1 to 5, in order to estimate the posture of the person.

[0132] The estimation unit 115 determines whether or not a predetermined posture has been detected (step S106).

[0133] In detail, for example, the estimation unit 115 determines that the predetermined posture has been detected when the subject shown in the query image is estimated to be in the predetermined posture in step S105. The estimation unit 115 determines that the predetermined posture has not been detected when the subject shown in the query image is estimated to be not in the predetermined posture in step S105.

[0134] If it is determined that the predetermined posture has not been detected (step S106; No), the image acquisition unit 111 executes step S101 again.

[0135] If it is determined that the predetermined posture has been detected (step S106; No), the display control unit 119 causes the display unit 118 to display that the predetermined posture has been detected (step S107). After that, the image acquisition unit 111 executes step S101 again.

[0136] In step S107, the display control unit 119 causes a query image showing the subject in a predetermined posture to be displayed on the display unit 118. The query image displayed here may be an image in which the subject is marked, as described above.

[0137] The user can know that a person to be photographed in a predetermined posture has been detected by looking at the display unit 118. For example, if the person is in a talking posture while operating an ATM, there is a possibility that the person may be a victim of a bank transfer fraud or a suspicious person, and the user can take measures such as notifying a security guard near the ATM to check.

[0138] By repeatedly performing such posture estimation processing, the posture of the subject can be estimated for each of the time-series query images.

[0139] Here, the plurality of frame images are, for example, images based on photographs taken while an ATM is being operated. In this case, the plurality of frame images taken while the same person is operating the ATM and the query images therebetween are time-series images showing the same person being photographed.

[0140] Therefore, if the estimated postures of the subject shown in each of the time-series query images are different (i.e., if the postures are different from each other), there is a possibility that an error will be included in one of the estimation results in the posture estimation process. When a plurality of query images include query images with different estimation results, the display control unit 119 causes the display unit 118 to display the query images with different estimation results. The display control unit 119 may store the query images with different estimation results, and may cause the display unit 118 to display all of the query images with different estimation results in response to a user instruction or the like.

[0141] 12 is a flowchart showing an example of estimation support processing according to this embodiment. The estimation support processing is processing for displaying a query image whose posture may have been erroneously estimated in order to support estimation of the posture of the subject. The estimation support processing is executed while the posture estimation processing is being performed. The estimation support processing may be executed repeatedly.

[0142] The determination unit 117 acquires the estimation results from step S105, which are repeatedly executed, and determines whether the estimation results for the plurality of query images correspond to an erroneous estimation pattern (step S201). This makes it possible to detect estimation results that correspond to an erroneous estimation pattern.

[0143] Specifically, the erroneous estimation pattern is, for example, a pattern of estimation results for each of a predetermined number of time-series query images. Here, the predetermined number may be equal to or greater than 2. As described above, the erroneous estimation pattern may include an estimation result for at least one query image that is different from the other query images.

[0144] Fig. 13 is a diagram showing an example of such an erroneous estimation pattern. In Fig. 13, "OK" indicates a positive estimation, and "NG" indicates a negative estimation. A positive estimation is a result of estimation that the posture is a predetermined posture. A negative estimation is a result of estimation that the posture is not a predetermined posture.

[0145] Pattern 1 shown in Fig. 13(a) shows an example of a pattern in which the estimation results for four query images in time series are "OK / NG / OK / NG" in that order. This pattern 1 is an example of a pattern in which the estimation results for the query images in time series are different and repeated a predetermined number of times or more. Fig. 13(a) shows an example in which the predetermined number of times is two.

[0146] For example, it is extremely rare for a photographed person to frequently change whether or not they are talking on a mobile phone while operating an ATM, and therefore it is extremely rare for their posture to frequently change whether or not they are in a talking position. The same is true for wheelchair postures. Therefore, if positive and negative inferences are repeated for a query image showing a photographed person while the same person is operating an ATM, it is highly likely that one of these inference results is incorrect. Therefore, by detecting the inference results in step S105 that correspond to pattern 1, it is possible to detect inference results that are highly likely to be incorrect.

[0147] Pattern 2 shown in Fig. 13(b) shows an example of a pattern in which the estimation results for each of four query images in a time series are "OK / OK / OK / NG" in that order. This pattern 2 is an example of a pattern in which the query images in a time series are estimated to have a predetermined posture for a predetermined number of consecutive images, and then immediately thereafter are estimated to have a posture other than the predetermined posture. Fig. 13(b) shows an example in which the predetermined number is 3.

[0148] For example, if the same person is operating an ATM in accordance with the instructions of a fraudster committing a bank transfer fraud, the mobile phone conversation is likely to continue until the ATM operation is completed, and a negative inference is likely to be incorrect. On the other hand, for a wheelchair position, it is extremely rare for the wheelchair position to end, and a negative inference is likely to be incorrect. Therefore, by detecting the inference results in step S105 that correspond to pattern 2, it is possible to detect inference results that are likely to be incorrect.

[0149] As exemplified in patterns 1 and 2, when at least one query image contains an estimation result that differs from other query images, the estimation result is likely to contain an error. Therefore, by detecting at least one query image containing an estimation result that differs from other query images, it is possible to detect an estimation result that is likely to be incorrect.

[0150] Note that the erroneous estimation pattern is not limited to patterns such as patterns 1 and 2 in which at least one query image includes an estimation result that differs from the other query images. The erroneous estimation pattern may be any pattern in which a plurality of query images includes a query image with different estimation results. This makes it possible to detect estimation results that may be erroneous. An example of such an erroneous estimation pattern is a pattern in which an estimation result differs from at least one of its preceding and succeeding chronological results (when expressed as "OK" and "NG" as in FIG. 13 , one or both of a pattern in which the estimation results for each query image in the chronological order are "OK / NG" and a pattern in which "NG / OK" are respectively indicated).

[0151] Referring again to Fig. 12, when it is determined that the estimation results for the plurality of query images do not correspond to an erroneous estimation pattern (step S201; No), the determining unit 117 repeats step S201.

[0152] When it is determined that the estimation results for the multiple query images correspond to an incorrect estimation pattern (step S201; Yes), the display control unit 119 acquires query images related to the incorrect estimation pattern and displays the acquired query images on the display unit 118 (step S202). The query images related to the incorrect estimation pattern include query images that have been estimated differently from other query images among the multiple query images. At this time, the display control unit 119 may also display the estimation results for the acquired query images.

[0153] For example, assume that the query image corresponds to pattern 1 shown in Fig. 13(a). In this case, by executing step S202, if the estimation results for the time-series query images are different a predetermined number of times or more, the display control unit 119 causes the display unit 118 to display the query images for which the estimations are repeatedly different. In this case, the display control unit 119 may cause the display unit 118 to display at least one of the query images corresponding to pattern 1.

[0154] For example, suppose that the pattern corresponds to pattern 2 shown in Fig. 13(b) . In this case, by executing step S202, when a query image in a time series is estimated to be not in the predetermined posture immediately after being estimated to be in the predetermined posture for a predetermined number of consecutive query images or more, the display control unit 119 causes the display unit 118 to display the query image determined to be not in the predetermined posture.

[0155] Referring again to Fig. 12, the display control unit 119 determines whether the result estimated in step S105 for the query image displayed in step S202 is incorrect, for example, based on a user input (step S203).

[0156] In detail, for example, the user checks the query image displayed in step S202 and the estimation result for the query image on the display unit 118. Then, the user operates the input unit 116 to input whether or not the result estimated in step S105 for the displayed query image is incorrect.

[0157] If it is determined that the estimated result is not an error (step S203; No), the display control unit 119 returns to step S201.

[0158] If it is determined that the estimated result is incorrect (step S203; Yes), the display control unit 119 causes the display unit 118 to display the reference image (step S204).

[0159] In detail, for example, by executing step S204, the display control unit 119 causes at least one of one or more reference images to be displayed on the display unit 118 when there is a query image among the time series of query images for which different estimations have been made that has an incorrect estimation result.

[0160] The reference image displayed here is the reference image used in step S105 when estimating the query image displayed in step S202. More specifically, the reference image displayed here is a reference image showing a person whose similarity to the subject shown in the erroneously estimated query image satisfies a predetermined criterion. The predetermined criterion may be the highest similarity, a similarity equal to or greater than a threshold, etc.

[0161] The display control unit 119 determines whether or not a predetermined instruction for displaying thinned frame images has been received, for example, based on a user input (step S205).

[0162] If it is determined that a predetermined instruction has not been received (step S205; No), the display control unit 119 returns to step S201.

[0163] If it is determined that a predetermined instruction has been received (step S205; Yes), the display control unit 119 acquires the frame images thinned out in step S102 from the image acquisition unit 111 (step S206).

[0164] In detail, for example, the display control unit 119 acquires at least one frame image that is thinned out between the query images in the time series for which different determinations have been made.

[0165] For example, assume that the pattern corresponds to pattern 1 shown in Fig. 13(a) . In this case, for example, the display control unit 119 acquires at least one of the frame images acquired and thinned out by the image acquisition unit 111 during the time between the positively estimated query image and the negatively estimated query image.

[0166] For example, assume that the pattern corresponds to pattern 2 shown in Fig. 13(b) . In this case, for example, the display control unit 119 acquires at least one of the frame images acquired and thinned out by the image acquisition unit 111 during the time between the negatively estimated query image and the positively estimated query image.

[0167] It should be noted that the photographing unit 101 may store frame images, and the display control unit 119 may acquire thinned frame images from the photographing unit 101. This allows the number of frame images transmitted from the photographing unit 101 to the information processing device 100 to be reduced, thereby reducing communication charges between the photographing unit 101 and the information processing device 100.

[0168] Referring again to Fig. 12, the posture acquisition unit 113, the similarity acquisition unit 114, and the estimation unit 115 execute the same processes as steps S103 to S105 of the posture estimation process.

[0169] In detail, for example, the posture acquisition unit 113, upon receiving instructions from the display control unit 119, acquires first posture information based on a plurality of reference images associated with a predetermined posture and second posture information based on the frame image acquired in step S206 (step S103).

[0170] The similarity obtaining unit 114 obtains the similarity between each of the first models included in the first orientation information and the second orientation information obtained in step S103 and the second model (step S104).

[0171] The estimation unit 115 estimates the posture of the person shown in the frame image based on the frame image acquired in step S206 and a plurality of reference images (step S105). In this step S105, the estimation unit 115 estimates the posture of the person shown in the query image based on the overall similarity between the frame image acquired in step S206 and each of reference images 1 to 7.

[0172] That is, by executing step S105 here, the estimation unit 115 estimates the posture of the person being photographed shown in the frame image based on at least one thinned-out frame image and the reference image.

[0173] The display control unit 119 causes the display unit 118 to display the frame image acquired in step S206 and the result of estimation for that frame image in step S105 (step S207), and the process returns to step S201.

[0174] By executing step S207, the display control unit 119 causes the display unit 118 to display at least one frame image that has been thinned out between the time-series query images for which different determinations have been made.

[0175] By executing the estimation assistance process, query images that may have been estimated incorrectly, reference images related to the query images, and thinned frame images can be displayed on the display unit 118. Furthermore, the thinned frame images and reference images can be used to display the results of estimation of whether or not the posture is a predetermined posture on the display unit 118. This allows the user to know which query images were erroneously estimated. Furthermore, the user can know which reference images were used to make the erroneous estimation.

[0176] (Operations and Effects) As described above, according to this embodiment, the information processing device 100 includes the estimation unit 115 and the display control unit 119 .

[0177] The estimation unit 115 estimates the posture of the subject shown in each of the multiple query images based on multiple query images obtained by photographing the subject performing a specified behavior multiple times and a reference image showing a person associated with a specified posture.

[0178] When a query image with a different estimation result is included among the plurality of query images, the display control unit 119 causes the display unit 118 to display the query image with the different estimation result.

[0179] Generally, when multiple query images include query images with different estimation results, the estimation results may contain errors. By displaying such query images, the user can refer to the query images for which different estimations have been made. The user can then check the accuracy of the estimated pose for each query image and take measures to improve the accuracy of pose estimation, such as deleting reference images that may be causing the incorrect estimation. Therefore, it is possible to improve the accuracy of estimating the pose of the subject shown in the image.

[0180] According to this embodiment, the plurality of query images are time-series query images, which makes it possible to improve the accuracy of estimating the posture of the subject shown in the time-series query images.

[0181] According to this embodiment, when the estimation results for a time-series query image are repeatedly different a predetermined number of times or more, the display control unit 119 causes the display unit 118 to display the query image for which the repeatedly different estimations have been made.

[0182] This allows the user to check query images that are likely to have been erroneously estimated and take measures to improve the accuracy of estimating the posture, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0183] According to the present embodiment, the information processing device 100 includes a similarity obtaining unit 114 that obtains a similarity between the postures of a person shown in the query image and a person shown in the reference image for each combination of a time-series query image and one or more reference images.

[0184] The estimation unit 115 estimates the posture of the person shown in each of the time-series query images based on the similarity. When there is a query image with an incorrect estimation result among the time-series query images for which different estimations have been repeatedly made, the display control unit 119 causes the display unit 118 to display, among one or more reference images, a reference image showing a person whose similarity to the person shown in the query image with the incorrect estimation satisfies a predetermined criterion.

[0185] This makes it possible to improve the accuracy of estimating the posture, for example by checking reference images that may be the cause of the erroneous estimation and deleting the reference images that may be the cause of the erroneous estimation, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0186] According to this embodiment, the information processing device 100 includes an image acquisition unit 111 that acquires a time series of frame images obtained by capturing an image of a subject multiple times in a time-series manner. The multiple query images are frame images obtained by thinning out some of the time series of frame images.

[0187] The display control unit 119 further causes the display unit 118 to display at least one frame image that has been thinned out between the time-series query images for which different determinations have been made.

[0188] This allows for checking frame images captured at a time close to the query image, which is likely to have been erroneously estimated, and measures to improve the accuracy of posture estimation, thereby improving the accuracy of posture estimation of the subject shown in the image.

[0189] According to this embodiment, the estimation unit 115 further estimates the posture of the person being photographed shown in the frame image based on at least one of the thinned frame images and the reference image.

[0190] This allows for checking the estimation results for frame images captured at a time close to the query image, which is likely to have been erroneously estimated, and measures can be taken to improve the accuracy of estimating the posture, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0191] According to this embodiment, when a query image in a time series is estimated to be in a predetermined posture for a predetermined number of consecutive times or more, and then immediately thereafter is estimated to be in a different posture from the predetermined posture, the display control unit 119 causes the display unit 118 to display the query image that has been determined to be in a different posture from the predetermined posture.

[0192] This allows the user to check query images that are likely to have been erroneously estimated and take measures to improve the accuracy of estimating the posture, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0193] According to this embodiment, the information processing device 100 further includes a determination unit 117 that determines whether or not a plurality of query images include a query image with different estimation results.

[0194] This makes it possible to detect estimation results that are likely to be incorrect, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0195] According to this embodiment, the determination unit 117 determines whether the estimation results for the multiple query images correspond to a predetermined erroneous estimation pattern. The erroneous estimation pattern is a pattern of estimation results related to the posture of the subject included in each of the multiple query images, and includes an estimation result for at least one query image that is different from that for the other query images. When the determination unit 117 determines that the estimation results for the multiple query images correspond to an erroneous estimation pattern, the display control unit 119 causes the display unit 118 to display a query image related to the erroneous estimation pattern.

[0196] This makes it possible to detect estimation results that are likely to be incorrect and confirm the estimation results. Then, it is possible to check the accuracy of the estimated pose for each query image and take measures to improve the accuracy of pose estimation, such as deleting reference images that may be causing the incorrect estimation. Therefore, it is possible to improve the accuracy of estimating the pose of the subject shown in the image.

[0197] According to this embodiment, the erroneous estimation pattern includes at least one of a pattern in which the estimation results for a time-series query image are different a predetermined number of times or more, and a pattern in which a time-series query image is estimated to be in a predetermined posture for a predetermined number of consecutive times or more, and then immediately thereafter is estimated to be in a posture other than the predetermined posture.

[0198] This makes it possible to detect estimation results that are likely to be incorrect, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0199] According to this embodiment, multiple images are taken while the ATM is being operated, which improves the accuracy of estimating the posture of the person being photographed as shown in the images taken while the ATM is being operated.

[0200] According to this embodiment, the plurality of query images are images showing a common subject, which makes it possible to improve the accuracy of estimating the posture of the common subject shown in the images.

[0201] According to this embodiment, the information processing device 100 includes an estimation unit 115 and a display control unit 119 .

[0202] The estimation unit 115 estimates the posture of the person being photographed shown in the query image based on a query image obtained based on photographing while the person is performing a specified action and one or more reference images showing a person corresponding to a specified posture.

[0203] When the estimation unit 115 makes an incorrect estimation, the display control unit 119 causes the display unit 118 to display, from among one or more reference images, a reference image showing a person whose posture similarity to the subject shown in the query image for which the incorrect estimation was made meets a predetermined standard.

[0204] This makes it possible to improve the accuracy of estimating the posture, for example by checking reference images that may be the cause of the erroneous estimation and deleting the reference images that may be the cause of the erroneous estimation, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0205] Second Embodiment In the first embodiment, an example has been described in which multiple images are captured at successive (different) times in time. The multiple images may be captured from two or more different directions at the same time.

[0206] 14 is a diagram showing a detailed example of the functional configuration of an information processing system S2 according to embodiment 2. The information processing system S2 includes two image capturing units 101, an information processing device 100, and an analysis device 102. Note that the number of image capturing units 101 may be three or more.

[0207] The two image capturing units 101 are cameras that capture a common area, such as the area in front of an ATM. Therefore, the two image capturing units 101 can capture images of a common subject from different directions at the same time. Each of the image capturing units 101 may be functionally and physically similar to the image capturing unit 101 according to the first embodiment.

[0208] The information processing device 100 and the analysis device 102 may be functionally and physically similar to those of embodiment 1. For example, when a plurality of query images include a query image with a different estimation result, the display control unit 119 causes the display unit 118 to display the query image with the different estimation result. The information processing system S2 may operate in the same manner as the information processing system S1 according to embodiment 1.

[0209] (Operations and Effects) As described above, according to this embodiment, multiple photographing is performed from two or more different directions at the same time.

[0210] This allows the user to refer to query images that were taken around the same time and for which different estimations were made, check whether the estimated posture for each query image is correct, and take measures to improve the accuracy of posture estimation, such as deleting reference images that may be causing incorrect estimations. Therefore, it is possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0211] Although the embodiments and modifications of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.

[0212] In addition, although the flowcharts used in the above description show multiple steps (processes) in a sequential order, the order of steps executed in each embodiment is not limited to the order shown. In each embodiment, the order of steps shown in the drawings can be changed as long as it does not cause any problems in terms of content. Furthermore, the above-described embodiments and variations can be combined as long as the content is not contradictory.

[0213] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0214] 1. An information processing device comprising: an estimation means for estimating a posture of a person shown in a query image based on a query image obtained by photographing the person performing a predetermined behavior and one or more reference images showing a person corresponding to a predetermined posture; and a display control means for, when the estimation means makes an incorrect estimation, causing a display means to display, among the one or more reference images, a reference image showing a person whose posture similarity to the person shown in the query image for which the incorrect estimation was made satisfies a predetermined standard. 2. An information processing device comprising: an estimation means for estimating a posture of the person shown in each of the plurality of query images based on a plurality of query images obtained by photographing the person performing a predetermined behavior multiple times and a reference image showing a person associated with a predetermined posture; and a display control means for, when a query image with a different estimation result is included among the plurality of query images, causing a display means to display the query image with the different estimation result. 3. The information processing device described in 2., wherein the plurality of query images are query images in time series. 4. The information processing device according to 3., wherein, when the estimation results for the time-series query images are repeatedly different a predetermined number of times or more, the display control means causes the display means to display the query images for which different estimations have been made. 5. The information processing device according to 4., wherein the reference image is one or more, and the device further comprises a similarity acquisition means for determining a similarity regarding a posture between a subject shown in the query image and a person shown in the reference image for each combination of the time-series query image and the one or more reference images, the estimation means estimates a posture of the subject shown in each of the time-series query images based on the similarity, and when there is a query image for which the estimation result is incorrect among the time-series query images for which different estimations have been made repeatedly, the display control means causes the display means to display the reference image showing a person for which the similarity to the subject shown in the query image for which the incorrect estimation was made satisfies a predetermined standard, among the one or more reference images.6. The information processing device described in any one of 3. to 5., further comprising image acquisition means for acquiring time-series frame images obtained by photographing the subject a plurality of times in temporal succession, wherein the plurality of query images are frame images obtained by thinning out some of the time-series frame images, and the display control means further causes the display means to display at least one frame image thinned out between the time-series query images for which the different determination has been made. 7. The information processing device described in 6., wherein the estimation means further estimates the posture of the subject shown in the frame image based on the at least one thinned-out frame image and the reference image. 8. The information processing device described in 3., wherein, when a query image determined not to be in the predetermined posture is estimated not to be in the predetermined posture for a predetermined number or more consecutive query images in the time series, the display control means causes the display means to display the query image determined not to be in the predetermined posture. 9. The information processing device described in any one of 1. to 8., further comprising determination means for determining whether the plurality of query images include query images with different estimation results. 10. The information processing device described in 9., wherein the determination means determines whether or not estimation results for the plurality of query images correspond to a predetermined erroneous estimation pattern, the erroneous estimation pattern being a pattern of estimation results related to the posture of the person being photographed included in each of the plurality of query images, and including an estimation result for at least one query image that is different from that for other query images, and the display control means, when the determination means determines that the estimation results for the plurality of query images correspond to the erroneous estimation pattern, causes the display means to display a query image related to the erroneous estimation pattern. 11. The information processing device described in 10., wherein the erroneous estimation pattern includes at least one of a pattern in which estimation results for time-series query images are repeatedly different a predetermined number of times or more, and a pattern in which, immediately after a predetermined number of consecutive times or more of query images have been estimated to be the predetermined posture, the time-series query images are estimated to be not the predetermined posture. 12. The information processing device described in 2., wherein the plurality of photographs are photographs taken from two or more different directions at the same time.13. The information processing device described in any one of 2. to 12., wherein the multiple photographs are taken while an ATM is being operated. 14. The information processing device described in any one of 1. to 10., wherein the multiple query images are images showing the same person being photographed. 15. An information processing system comprising: the information processing device described in any one of 1. to 14.; and one or more photographing units that take the multiple photographs. 16. An information processing method, wherein a computer estimates a posture of the person being photographed shown in each of the multiple query images based on multiple query images obtained based on multiple photographs taken while the person is performing a predetermined action and a reference image that shows a person corresponding to a predetermined posture, and when the multiple query images include query images with different estimation results, the query images with the different estimations are displayed on a display means. 17. 17. A recording medium having recorded thereon a program for causing a computer to: estimate a posture of a person shown in each of a plurality of query images based on a plurality of query images obtained by photographing a plurality of times while a predetermined behavior is being performed and a reference image showing a person corresponding to a predetermined posture; and, if a query image with a different estimation result is included among the plurality of query images, display the query image with the different estimation on a display means. 18. A recording medium having recorded thereon a program for causing a computer to: estimate a posture of a person shown in each of a plurality of query images based on a plurality of query images obtained by photographing a plurality of times while a predetermined behavior is being performed and a reference image showing a person corresponding to a predetermined posture; and, if a query image with a different estimation result is included among the plurality of query images, display the query image with the different estimation on a display means.

[0215] S1, S2 Information processing system 100 Information processing device 101 Photography unit 102 Analysis device 111 Image acquisition unit 112 Storage unit 112a Reference information 112b Weight information 113 Orientation acquisition unit 114 Similarity acquisition unit 114a Overall calculation unit 114b Element calculation unit 115 Estimation unit 116 Input unit 117 Determination unit 118 Display unit 119 Display control unit

Claims

1. an estimation means for estimating a posture of the subject shown in each of the plurality of query images based on a plurality of query images obtained by photographing the subject performing a predetermined action a plurality of times and a reference image showing a person associated with a predetermined posture; a display control means for, when a query image with a different estimation result is included in the plurality of query images, displaying the query image with the different estimation result on a display means; Information processing device.

2. The plurality of query images are time-series query images. The information processing device according to claim 1 .

3. The display control means, when the estimation results for the time-series query images are different a predetermined number of times or more, causes the display means to display the query images for which the estimations are different. The information processing device according to claim 2 .

4. The reference image is one or more, a similarity obtaining unit for obtaining a similarity between a posture of a subject shown in the query image and a posture of a person shown in the reference image for each combination of the time-series query image and the one or more reference images; The estimation means estimates a posture of the subject shown in each of the time-series query images based on the similarity; and When there is a query image with an incorrect estimation result among the time-series query images for which different estimations have been repeatedly performed, the display control means displays, on the display means, a reference image showing a person whose similarity to the subject shown in the query image with the incorrect estimation satisfies a predetermined criterion among the one or more reference images. The information processing device according to claim 3 .

5. The image capturing device further includes an image capturing unit for capturing a time-series frame image obtained by capturing an image of the subject a plurality of times in succession over time, the plurality of query images are frame images obtained by thinning out a portion of the frame images in the time series, The display control means further causes the display means to display at least one frame image that is thinned out between the time-series query images for which different determinations have been made. The information processing device according to claim 2 .

6. The estimation means further estimates a posture of the person being photographed shown in the frame image based on at least one of the thinned frame images and the reference image. The information processing device according to claim 5 .

7. When the query image in the time series is estimated to be in the predetermined posture for a predetermined number of consecutive images, and then the query image is estimated to be in the predetermined posture but is not in the predetermined posture, the display control means causes the display means to display the query image determined to be in the predetermined posture. The information processing device according to claim 2 .

8. The plurality of query images may further include a determination unit for determining whether or not a query image with a different estimation result is included in the plurality of query images. The information processing device according to claim 1 .

9. The computer Based on a plurality of query images obtained by photographing a plurality of times a person performing a predetermined behavior and a reference image showing a person corresponding to a predetermined posture, a posture of the person shown in each of the plurality of query images is estimated; When a query image with a different estimation result is included in the plurality of query images, the query image with the different estimation result is displayed on a display means. Information processing methods.

10. On the computer, Based on a plurality of query images obtained by photographing a plurality of times a person performing a predetermined behavior and a reference image showing a person corresponding to a predetermined posture, a posture of the person shown in each of the plurality of query images is estimated; and a program for causing a display unit to display, when a query image with a different estimation result is included among the plurality of query images, the query image with the different estimation result.