Information processing device, information processing method, and program

Abstract

An information processing device (100) comprises an estimation unit (115) and a display control unit (119). The estimation unit (115) estimates, on the basis of a plurality of query images obtained by imaging the subject to be imaged multiple times who performs a prescribed action and a reference image that shows the person associated with a prescribed posture, the posture of the imaged subject shown in each of the plurality of query images. When a query image is included in the plurality of query images, the estimation result of which is different, the display control unit (119) causes a display unit to display the query image with which a different estimation was made.

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method. , and Beauty program Regarding. [Background technology]

[0002] For example, the image search device described in Patent Document 1 includes a posture estimation unit, a feature extraction unit, a query generation unit, and an image search unit.

[0003] The posture estimation unit described in the document recognizes posture information of a search target, which is composed of multiple feature points, from an input image. The feature extraction unit described in the document extracts features from the posture information and the input image. The query generation unit described in the document generates a search query from an image database that stores feature amounts associated with input images and posture information specified by a user. The image search unit described in the document searches the image database for images containing similar postures according to the search query.

[0004] Patent Document 2 describes a technology for calculating feature amounts for each of a plurality of key points of a human body included in an image, searching for images containing human bodies with similar postures or movements based on the calculated feature amounts, and classifying images with similar postures or movements together. Non-Patent Document 1 describes a technology related to human skeleton estimation. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-091138 [Patent Document 2] International Publication No. 2021 / 084677 [Non-patent literature]

[0006] [Non-Patent Document 1] Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields];, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, P. 7291-7299 Summary of the Invention [Problem to be solved by the invention]

[0007] Patent Document 1 describes a technology for estimating posture or behavior based on an image. However, Patent Document 1 does not know whether the posture has been correctly estimated, making it difficult to improve the accuracy of estimating the posture of a subject shown in an image.

[0008] Incidentally, neither Patent Document 2 nor Non-Patent Document 1 discloses a technique for improving the accuracy of detecting a person in a predetermined posture from an image of the person.

[0009] In view of the above-mentioned problems, one example of the object of the present invention is to provide an information processing device, an information processing method, an information processing system, and a recording medium that solve the problem of improving the accuracy of estimating the posture of a person shown in an image. [Means for solving the problem]

[0010] According to one aspect of the present invention, Perform the prescribed action The same subject multiple Round shot shadow By doing Obtained Ta an estimation means for estimating a posture of a person shown in each of a plurality of query images based on a plurality of query images and a reference image showing a person associated with a predetermined posture; a display control means for, when a query image with a different estimation result is included among the plurality of query images, displaying the query image with the different estimation result on a display means; 、 The plurality of query images are time-series query images. R An information processing device is provided.

[0012] According to one aspect of the present invention, The computer Perform the prescribed action The same subject multiple Round shot shadow By doing Obtained Ta estimating a posture of a person shown in each of the plurality of query images based on a plurality of query images and a reference image showing a person corresponding to a predetermined posture; When a query image with a different estimation result is included in the plurality of query images, the query image with the different estimation result is displayed on a display means. This includes: The plurality of query images are time-series query images. A method for processing information is provided.

[0013] According to one aspect of the present invention, On the computer, Perform the prescribed action The same subject multiple Round shot shadow By doing Obtained Ta estimating a posture of a person shown in each of the plurality of query images based on a plurality of query images and a reference image showing a person corresponding to a predetermined posture; When a query image with a different estimation result is included in the plurality of query images, the query image with the different estimation result is displayed on a display means. 、 The plurality of query images are time-series query images. Program Mu Provided. [Effects of the Invention]

[0014] According to one aspect of the present invention, it is possible to provide an information processing device, an information processing method, an information processing system, and a recording medium that solve the problem of improving the accuracy of estimating the posture of a person being photographed shown in an image. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an overview of an information processing device according to a first embodiment. [Figure 2] 1 is a diagram illustrating an overview of an information processing system according to a first embodiment. [Figure 3] 1 is a flowchart showing an outline of information processing according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a detailed functional configuration of the information processing system according to the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of the configuration of reference information including a reference image associated with a calling posture. [Figure 6] 10 is a diagram illustrating an example of the configuration of weight information indicating weights associated with a calling posture. FIG. [Figure 7] FIG. 2 is a diagram illustrating an example of the functional configuration of a similarity acquisition unit according to the first embodiment. [Figure 8] 1 is a diagram illustrating an example of the physical configuration of an information processing device according to a first embodiment. [Figure 9] 6 is a flowchart showing an example of a posture estimation process according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a method for thinning out a portion of a plurality of frame images. [Figure 11] 10 is a flowchart showing a detailed example of a similarity obtaining process according to the first embodiment. [Figure 12] 10 is a flowchart illustrating an example of estimation support processing according to the first embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of an erroneous estimation pattern. [Figure 14] FIG. 10 is a diagram illustrating an example of a detailed functional configuration of an information processing system S2 according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0017] <Embodiment 1> (overview) 1 is a diagram showing an overview of an information processing device 100 according to embodiment 1. The information processing device 100 includes an estimation unit 115 and a display control unit 119.

[0018] The estimation unit 115 estimates the posture of the person being photographed shown in each of the multiple query images based on multiple query images obtained by taking multiple photographs while the person is performing a specified action and a reference image showing a person associated with a specified posture.

[0019] When a query image with a different estimation result is included among the plurality of query images, the display control unit 119 causes the display unit to display the query image with the different estimation result.

[0020] According to this information processing device 100, it is possible to provide an information processing device 100 that solves the problem of improving the accuracy of estimating the posture of a person to be photographed shown in an image.

[0021] 2 is a diagram showing an overview of an information processing system S1 according to embodiment 1. The information processing system S1 includes an information processing device 100 and one or more imaging units 101 that perform imaging multiple times.

[0022] According to this information processing system S1, it is possible to provide an information processing system S1 that solves the problem of improving the accuracy of estimating the posture of a person to be photographed shown in an image.

[0023] FIG. 3 is a flowchart showing an outline of information processing according to the first embodiment.

[0024] The estimation unit 115 estimates the posture of the person being photographed shown in each of the multiple query images based on multiple query images obtained by taking multiple photographs while the person is performing a predetermined behavior and a reference image showing a person associated with a predetermined posture (step S105).

[0025] When a query image with a different estimation result is included among the plurality of query images, the display control unit 119 causes the display unit to display the query image with the different estimation result (step S202).

[0026] According to this information processing, it is possible to provide an information processing system S1 that solves the problem of improving the accuracy of estimating the posture of a person to be photographed shown in an image.

[0027] (detail) A detailed example of the information processing system S1 according to the first embodiment will be described below.

[0028] 4 is a diagram showing a detailed example of the functional configuration of the information processing system S1 according to embodiment 1. The information processing system S1 includes an imaging unit 101, an information processing device 100, and an analysis device 102. The imaging unit 101, the information processing device 100, and the analysis device 102 are connected via a network N configured by a wired or wireless connection or a combination thereof, and can transmit and receive information to and from each other.

[0029] The photographing unit 101 photographs a person (subject) performing a predetermined action. The photographing unit 101 is, for example, a camera installed in a branch of a financial institution such as a bank, and photographs an operator operating an automated teller machine (ATM).

[0030] The photographing unit 101 is not limited to a camera for photographing an ATM operator, but may be a camera for photographing the inside of a store such as a bank, or may be a camera installed in various stores other than financial institutions.

[0031] The photographing unit 101 photographs a predetermined photographing area and transmits image information representing a moving image to the information processing device 100.

[0032] Specifically, the image capturing unit 101 captures images multiple times in chronological order at a predetermined frame rate. The image capturing unit 101 generates frame information including frame images for each capture. The image capturing unit 101 transmits the frame information including each of the chronological frame images to the information processing device 100 via the network N.

[0033] The analysis device 102 is a device that analyzes an image. The analysis device 102 acquires image information generated by the imaging unit 101 via the network N. In this embodiment, an example will be described in which the analysis device 102 acquires image information from the imaging unit 101 via the information processing device 100, but the analysis device 102 may acquire image information directly from the imaging unit 101.

[0034] The analysis device 102 is a device that analyzes the image contained in the acquired image information.

[0035] Specifically, analysis device 102 has one or more analysis functions that perform processing (analysis processing) to analyze images. The analysis functions that analysis device 102 has are one or more of: (1) object detection function, (2) face analysis function, (3) human figure analysis function, (4) posture analysis function, (5) behavior analysis function, (6) appearance attribute analysis function, (7) gradient feature analysis function, (8) color feature analysis function, and (9) movement line analysis function.

[0036] (1) The object detection function detects people and objects from an image. The object detection function can also determine the location of people and objects within an image. An example of a model that can be applied to object detection processing is YOLO (You Only Look Once). The object detection function detects, for example, an operator, a mobile phone (including a smartphone), a wheelchair, etc. Furthermore, for example, the object detection function determines the location of the detected people and objects.

[0037] (2) The face analysis function detects human faces from images, extracts the features of the detected faces (facial feature values), and classifies the detected faces (classification). The face analysis function can also determine the position of the face within the image. The face analysis function can also determine the identity of people detected from different images based on the similarity between the facial feature values ​​of people detected from different images.

[0038] (3) The human morphology analysis function extracts the physical characteristics of people in images (for example, values ​​that indicate overall characteristics such as whether they are fat or thin, height, and clothing), and classifies (classifies) people in images. The human morphology analysis function can also identify the position of a person in an image. The human morphology analysis function can also determine the identity of people in different images based on the physical characteristics of people in different images.

[0039] (4) The posture analysis function generates posture information that indicates a person's posture. The posture information includes, for example, a posture estimation model of a person. The posture estimation model is a model that connects the joints of a person estimated from an image. The posture estimation model is composed of multiple model elements corresponding to the joint elements, the trunk elements corresponding to the torso, and the bone elements corresponding to the bones connecting the joints. For example, the posture analysis function detects the joint points of a person from an image and connects the joint points to create a posture estimation model.

[0040] The posture analysis function then uses information from the posture estimation model to estimate a person's posture, extract features of the estimated posture (posture features), and classify (classify) people included in the images. The posture analysis function can also determine the identity of people included in different images based on the posture features of people included in different images.

[0041] For example, the posture analysis function creates posture estimation models such as a talking posture and a wheelchair posture, and extracts posture features for those postures. The talking posture is the posture of talking on a mobile phone. The wheelchair posture is the posture of a person using a wheelchair.

[0042] For example, the techniques disclosed in Patent Document 2 and Non-Patent Document 1 can be applied to the posture analysis function. (5) The behavior analysis process can estimate a person's movements using information from the posture estimation model, posture changes, etc., extract features of the person's movements (movement features), and classify (classify) people included in the image. The behavior analysis process can also estimate a person's height and identify the person's position in the image using information from the stick figure model. The behavior analysis process can estimate behaviors such as posture changes or transitions, and movement (position changes or transitions) from the image, and extract movement features of the behavior.

[0043] (6) The appearance attribute analysis function can recognize appearance attributes associated with people. The appearance attribute analysis function extracts features related to the recognized appearance attributes (appearance attribute features) and classifies (classifies) people in the image. Appearance attributes are attributes of appearance, and include one or more of, for example, clothing color, shoe color, hairstyle, and whether or not a hat, tie, or glasses are worn.

[0044] (7) The gradient feature analysis function extracts gradient features from an image. For gradient feature detection, technologies such as SIFT, SURF, RIFF, ORB, BRISK, CARD, and HOG can be applied.

[0045] (8) The color feature analysis function can detect objects from an image, extract the color features of the detected objects, and classify the detected objects. An example of a color feature is a color histogram. The color feature analysis function can detect, for example, people and objects contained in an image.

[0046] (9) The flow line analysis function can determine the flow line (trajectory of movement) of a person included in a video, for example, using the result of the identity determination in any of the above-mentioned analysis functions (2) to (6). In more detail, for example, by connecting people who are determined to be the same in different images in a time series, the flow line of the person can be determined. Note that, when acquiring videos captured by multiple image capturing units 101 capturing different shooting areas, the flow line analysis function can also determine the flow line across multiple videos captured in different shooting areas.

[0047] Image features include, for example, the results of object detection by the object detection function, facial features, human body features, posture features, movement features, appearance attribute features, gradient features, color features, and movement lines.

[0048] Each of the analysis functions (1) to (9) may appropriately use the results of analysis performed by other analysis functions. Information processing device 100 may include an analysis unit having the functions of analysis device .

[0049] The information processing device 100 according to the first embodiment is a device that estimates the posture of a person included in a frame image. As shown in Fig. 4, the information processing device 100 functionally includes an image acquisition unit 111, a storage unit 112, a posture acquisition unit 113, a similarity acquisition unit 114, an estimation unit 115, an input unit 116, a determination unit 117, a display unit 118, and a display control unit 119.

[0050] The image acquisition unit 111 acquires image information indicating a moving image from the imaging unit 101. That is, the image acquisition unit 111 acquires a plurality of frame images in time series obtained by capturing images a plurality of times consecutively in time series.

[0051] In detail, the image acquisition section 111 acquires frame information including each of a plurality of frame images in time series from the imaging section 101. The image acquisition section 111 stores the acquired frame information.

[0052] The storage unit 112 is a storage unit for storing various types of information, and stores, in advance, for example, reference information 112a indicating a reference image, weight information 112b indicating a weight, and the like.

[0053] The reference image is an image of a person associated with a predetermined posture. The reference image is an image that is referred to in order to estimate the posture of a person included in a query image, and is appropriately selected and set in the storage unit 112. The predetermined posture is, for example, a talking posture, a wheelchair posture, etc.

[0054] 5 is a diagram showing an example of the configuration of reference information 112a including reference images associated with talking postures. The reference information 112a shown in FIG. 5 includes, for example, positive examples and negative examples.

[0055] The positive examples are reference images of a person in a specific posture. The positive examples shown in Fig. 5 (i.e., reference images 1 to 4) are reference images of a person in a talking posture, such as a person standing and talking on a mobile phone with their right or left hand.

[0056] Negative examples are reference images of people who are not in a predetermined posture. Images of people who are not in a predetermined posture but are similar to the predetermined posture may be selected as negative examples. The negative examples (i.e., reference images 5 to 7) shown in FIG. 5 are reference images of people who are not in a posture for talking, such as a person standing upright without holding a mobile phone.

[0057] The reference information 112a may include any number of reference images as long as it includes at least one reference image. The reference information 112a may also include only positive examples.

[0058] The weight is a value indicating the degree of importance given to each model element for determining the similarity between pose estimation models in a given pose. Weight information 112b includes the weight of each model element for each given pose.

[0059] FIG. 6 is a diagram showing an example of the configuration of weight information 112b indicating the weight associated with the calling posture. 6The weight information 112b shown in the example in FIG. 1 associates element IDs and weights in a talking posture. The element IDs are information for identifying model elements. The element IDs are numbers appropriately assigned to bone elements and joint elements corresponding to, for example, the trunk element, the upper and lower parts of the left and right arms, the thighs and lower legs, etc. The weights are determined for each model element in a given posture. 6 Although an example in which the weight is an integer equal to or greater than 0 is shown, the method for setting the weight may be changed as appropriate.

[0060] For example, in a talking posture, the weight set for the arms is greater than the weight set for the legs because the mobile phone is held while talking, and in a talking posture when talking with the right hand, the weight set for the right hand is greater than the weight set for the left hand.

[0061] The posture acquisition unit 113 acquires, from the storage unit 112, a plurality of reference images associated with a predetermined posture such as a talking posture, and acquires first posture information based on the acquired plurality of reference images.

[0062] The first pose information is information indicating the pose of a person shown in each of a plurality of reference images associated with a predetermined pose. The first pose information includes, for example, a first model that is a pose estimation model for the person shown in the reference image.

[0063] Furthermore, the orientation acquisition unit 113 acquires time-series frame images from the image acquisition unit 111 and acquires a query image by thinning out some of the time-series frame images. Then, the orientation acquisition unit 113 acquires second orientation information based on the acquired query image.

[0064] The second pose information is information indicating the pose of the subject shown in the query image. The second pose information includes, for example, a second model that is a pose estimation model for the subject shown in the query image.

[0065] In detail, for example, pose acquisition unit 113 transmits each of the acquired reference image and query image to analysis device 102 via network N. When the reference image is transmitted to analysis device 102, pose acquisition unit 113 acquires first pose information including a first model of a person shown in the reference image from analysis device 102. When the query image is transmitted to analysis device 102, pose acquisition unit 113 acquires second pose information including a second model of a person shown in the query image from analysis device 102.

[0066] The similarity obtaining unit 114 obtains the similarity regarding the posture between the subject shown in the query image and the person shown in the reference image for each combination of a time-series query image and a plurality of reference images corresponding to a predetermined posture.

[0067] The similarity is a value that indicates the degree of similarity between pose estimation models in a given pose.

[0068] For example, the similarity acquisition unit 114 acquires a first model of a person shown in each of a plurality of reference images corresponding to a predetermined posture from the posture acquisition unit 113. The similarity acquisition unit 114 also acquires a second model of a person shown in each of the time-series query images from the posture acquisition unit 113. For each combination of the first model and the second model, the similarity acquisition unit 114 calculates the similarity using the first model and the second model.

[0069] The similarity includes an overall similarity and an element similarity.

[0070] The overall similarity is a value indicating the overall degree of similarity between the first model and the second model in a given pose, that is, the overall similarity between the first model and the second model.

[0071] The element similarity is the similarity for each corresponding model element between the first model and the second model in a predetermined posture, that is, the similarity for each corresponding model element between the first model and the second model.

[0072] The similarity may include at least one of the overall similarity and the element similarity.

[0073] 7 is a diagram showing an example of the functional configuration of the similarity acquisition unit 114 according to this embodiment. The similarity acquisition unit 114 includes an overall calculation unit 114a and an element calculation unit 114b.

[0074] The overall calculation unit 114a calculates the overall similarity between the first model and the second model. Specifically, the overall calculation unit 114a calculates the overall similarity using the weight corresponding to the predetermined posture included in the weight information 112b and the element similarity calculated by the element calculation unit 114b.

[0075] For example, when overall calculation unit 114a obtains the similarity of each model element from element calculation unit 114b, it calculates the product of each similarity of the model element and the weight of the corresponding model element, and sums up the products obtained for each model element constituting the posture estimation model. The value obtained as a result of this summation is the overall similarity.

[0076] The element calculation unit 114b calculates element similarity, which is the degree of similarity for each corresponding model element between the first model and the second model. For example, the element calculation unit 114b calculates element similarity for each corresponding model element between the first model and the second model based on the size, length, inclination, etc.

[0077] The estimation unit 115 estimates the posture of the person being photographed shown in each of the multiple query images based on multiple query images obtained by taking multiple photographs while the person is performing a specified action and a reference image showing a person associated with a specified posture.

[0078] For example, the estimation unit 115 estimates the posture of the person being photographed shown in each of the query images in time series based on the similarity (for example, overall similarity) obtained by the similarity acquisition unit 114.

[0079] Furthermore, the estimation unit 115 may estimate the posture of the person being photographed shown in at least one thinned-out frame image from among the frame images in the time series, based on the at least one thinned-out frame image and a reference image.

[0080] In this case, the posture acquisition unit 113 acquires at least one of the thinned frame images from the image acquisition unit 111 and acquires a second model of the person shown in the frame image. The similarity acquisition unit 114 calculates an overall similarity based on the second model of the person shown in the frame image and the first models of the person shown in each of the multiple reference images. Then, the estimation unit 115 estimates the posture of the person shown in the at least one thinned frame image based on the overall similarity calculated by the similarity acquisition unit 114.

[0081] There are various methods for the estimation unit 115 to estimate the posture of the person being photographed based on the similarity, and posture estimation methods 1 to 5 will be described below as examples.

[0082] (Posture estimation method 1) For example, the estimation unit 115 may estimate the posture of the person being photographed shown in the query image or the frame image based on the reference image with the highest similarity among the positive and negative examples. In this case, for example, if the reference image with the highest similarity is a positive example, the estimation unit 115 estimates that the posture of the person being photographed is a predetermined posture corresponding to the reference image. If the reference image with the highest similarity is a negative example, the estimation unit 115 estimates that the posture of the person being photographed is not a predetermined posture corresponding to the reference image.

[0083] (Posture estimation method 2) Furthermore, for example, the estimation unit 115 may estimate the posture of the photographed person shown in the query image or the frame image based on the positive example average value and the negative example average value. The positive example average value is the average value of the similarity between the query image or the frame image and multiple positive examples associated with a predetermined posture. The negative example average value is the average value of the similarity between the query image or the frame image and multiple negative examples associated with a predetermined posture.

[0084] In this case, for example, if the positive example average value is greater than the negative example average value, the estimation unit 115 estimates that the posture of the photographed person is the predetermined posture corresponding to the reference image. If the positive example average value is equal to or less than the negative example average value, the estimation unit 115 estimates that the posture of the photographed person is not the predetermined posture corresponding to the reference image.

[0085] (Posture estimation method 3) Furthermore, for example, the estimation unit 115 may perform image matching between a query image or a frame image and a reference image, and estimate the posture of the person shown in the query image or the frame image based on the similarity between the reference image that matches in the image matching and the query image or the frame image. In this case, the estimation unit 115 may estimate the posture of the person shown in the query image or the frame image based on the average positive example value and the average negative example value among the reference images that match in the image matching.

[0086] In detail, for example, when the average value of positive examples among the reference images matched by image matching is greater than the average value of negative examples, the estimation unit 115 estimates that the posture of the photographed person is a predetermined posture corresponding to the reference images. When the average value of positive examples among the reference images matched by image matching is equal to or less than the average value of negative examples, the estimation unit 115 estimates that the posture of the photographed person is not a predetermined posture corresponding to the reference images.

[0087] Note that various known techniques may be applied to image matching. For example, the estimation unit 115 calculates the similarity between images based on the feature amounts of the subject shown in the query image or frame image and the person shown in the reference image. The estimation unit 115 compares the similarity between the images with a threshold to determine whether the query image or frame image and the reference image match. For example, the estimation unit 115 determines that the images match (are similar) when the similarity between the images is equal to or greater than the threshold, and determines that the images do not match (are dissimilar) when the similarity between the images is less than the threshold.

[0088] (Posture estimation method 4) The estimation unit 115 may estimate the posture of the person being photographed using only image matching, without using the similarity.

[0089] For example, the estimation unit 115 may estimate that the subject shown in the query image or the frame image is in a predetermined posture corresponding to the reference image when the image matching matches at least one positive example. Alternatively, for example, the estimation unit 115 may estimate that the subject shown in the query image or the frame image is not in a predetermined posture corresponding to the reference image when the image matching matches at least one negative example.

[0090] Furthermore, for example, the estimation unit 115 may estimate the posture of the subject based on the number of matches between positive examples and negative examples that match in image matching. In this case, for example, if the number of matches with positive examples is greater than the number of matches with negative examples, the estimation unit 115 estimates that the subject shown in the query image or frame image is in a predetermined posture corresponding to the reference image. Also, for example, if the number of matches with positive examples is equal to or less than the number of matches with negative examples, the estimation unit 115 estimates that the subject shown in the query image or frame image is not in a predetermined posture corresponding to the reference image.

[0091] In image matching, if a query image or a frame image does not match either a positive example or a negative example, the estimation unit 115 may determine that the query image or the frame image is different from either a positive example or a negative example, or may determine that the query image or the frame image is a negative example.

[0092] (Posture estimation method 5) The estimation unit 115 may determine whether the query image or the frame image conforms to (is similar to) a positive example or a negative example by using a learning model that has been trained through machine learning using a reference image. This learning model is a trained learning model that has been trained through machine learning for determining whether the photographed person conforms to a positive example or a negative example. In this case, the estimation unit 115 inputs image information including the query image or the frame image that shows the photographed person into the learning model, and obtains a determination result as to whether the query image or the frame image conforms to (is similar to) a positive example or a negative example.

[0093] The input data to the learning model during learning includes image information of people. Furthermore, learning may be performed using supervised learning that includes labels (correct answers) that indicate whether the input data matches a positive example or a negative example.

[0094] The input unit 116 is a keyboard, a mouse, a touch panel, or the like that accepts input from the user.

[0095] Based on the result of estimation by the estimation unit 115, the determination unit 117 determines whether or not a query image with different estimation results is included among the plurality of query images.

[0096] In detail, for example, the determination unit 117 determines whether or not the results of estimation for a plurality of query images correspond to a predetermined erroneous estimation pattern.

[0097] The erroneous estimation pattern is a pattern of the estimation result related to the posture of the photographed person included in each of the plurality of query images. The erroneous estimation pattern is, for example, determined in advance and stored in the storage unit 112.

[0098] The erroneous estimation pattern may include, for at least one query image, an estimation result that is different from that of the other query images. Thus, in step S201, the determination unit 117 determines whether the estimation result of the estimation unit 115 corresponds to the erroneous estimation pattern, thereby making it possible to determine whether the plurality of query images includes a query image with a different estimation result.

[0099] The display unit 118 is a display or the like that displays various types of information. The display control unit 119 controls the display unit 118 to display various types of information on the display unit 118. For example, when the estimation unit 115 detects a person to be photographed in a predetermined posture, the display control unit 119 causes the display unit 118 to display a query image or a frame image in which the person to be photographed is marked. The mark is, for example, a rectangular frame that surrounds the person to be photographed.

[0100] Furthermore, for example, based on the result of the judgment by the judgment unit 117, if a query image with a different estimation result is included among the multiple query images, the display control unit 119 causes the display unit 118 to display the query image with the different estimation result.

[0101] (Physical configuration of information processing system S1) The information processing system S1 is physically composed of an imaging unit 101, an information processing device 100, and an analysis device 102, which are connected via a network N. The imaging unit 101, the information processing device 100, and the analysis device 102 are each composed of a physically separate single device. The imaging unit 101 is, for example, a camera.

[0102] The information processing device 100 and the analysis device 102 may be physically configured as a single device, in which case the information processing device 100 and the analysis device 102 are connected using an internal bus 1010 (described later) instead of the network N. Alternatively, one or both of the information processing device 100 and the analysis device 102 may be physically configured as multiple devices connected via an appropriate communication line such as the network N.

[0103] 8 is a diagram showing an example of the physical configuration of an information processing device 100 according to this embodiment. The information processing device 100 is, for example, a general-purpose computer. The information processing device 100 includes, for example, a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070.

[0104] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070. However, the method of connecting the processor 1020 and other components to each other is not limited to bus connection.

[0105] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0106] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0107] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), etc. The storage device 1040 stores program modules for realizing each function of the information processing device 100. The processor 1020 reads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module.

[0108] The network interface 1050 is an interface for connecting the information processing device 100 to the network N.

[0109] The input interface 1060 is an interface for a user to input information, and is configured with one or more of, for example, a keyboard, a mouse, a touch panel, and the like.

[0110] The output interface 1070 is an interface for presenting information to the user, and is configured, for example, by a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like.

[0111] The analysis device 102 is physically a general-purpose computer, for example, and is configured in a similar manner to the information processing device 100 (see FIG. 8).

[0112] Storage device 1040 of analysis device 102 stores program modules for realizing each function of analysis device 102. Processor 1020 of analysis device 102 reads each of these program modules into memory 1030 and executes them, thereby realizing each function corresponding to the program module. Network interface 1050 of analysis device 102 is an interface for connecting analysis device 102 to network N. Except for these points, analysis device 102 may be physically configured in the same way as information processing device 100.

[0113] (Operation of information processing system S1) The information processing system S1 according to this embodiment executes information processing for estimating the posture of a subject included in a query image. The information processing executed by the information processing system S1 includes posture estimation processing and estimation support processing.

[0114] The posture estimation process is a process for estimating the posture of the subject included in the query image using a reference image associated with a predetermined posture. The estimation support process is a process for supporting the estimation of the posture of the subject.

[0115] 9 is a flowchart showing an example of the posture estimation process according to this embodiment. The posture estimation process is executed, for example, while the information processing system S1 is in operation.

[0116] The image acquisition unit 111 acquires a plurality of frame images in time series (step S101). The image acquisition unit 111 stores the acquired frame images.

[0117] In detail, for example, the image acquisition unit 111 sequentially acquires a plurality of frame images in time series from time T1 to just before time T2, where time interval ΔT is defined as time T2−time T1=time interval ΔT.

[0118] The image acquisition unit 111 thins out some of the frame images acquired in step S101 to acquire a query image (step S102).

[0119] In more detail, for example, the image acquisition unit 111 thins out some of the multiple frame images in accordance with a predetermined rule. For example, FIG. 10 is a diagram showing an example of a method for thinning out some of the multiple frame images. As shown in the diagram, the image acquisition unit 111 thins out frame images acquired during a predetermined time interval ΔT (excluding both end times). In this way, the image acquisition unit 111 acquires query images in a time series at a predetermined constant time interval ΔT. Note that the method for acquiring query images by thinning out some of the multiple frame images is not limited to this; for example, the time interval ΔT does not have to be constant and may be changed depending on the operation mode (a mode for tracking a subject, a mode for detecting the posture of the subject). Alternatively, the query images may be a plurality of frame images that have not been thinned out.

[0120] The orientation acquisition unit 113 acquires first orientation information based on a plurality of reference images associated with a predetermined orientation, and second orientation information based on the query image acquired in step S102 (step S103).

[0121] In detail, for example, posture acquisition unit 113 acquires a plurality of reference images corresponding to a predetermined posture from storage unit 112. When the predetermined posture is a call posture and reference information 112a shown in FIG. 5 is stored in storage unit 112, posture acquisition unit 113 acquires reference image 1 to reference image 7. Posture acquisition unit 113 transmits the acquired reference image 1 to reference image 7 to analysis device 102. In response to this, analysis device 102 generates first posture information including a first model of a person represented by each of reference image 1 to reference image 7 and transmits the first posture information to information processing device 100. Posture acquisition unit 113 acquires the first posture information from analysis device 102.

[0122] The posture acquisition unit 113 acquires the query image acquired in step S102 from the image acquisition unit 111. The posture acquisition unit 113 transmits the acquired query image to the analysis device 102. In response to this, the analysis device 102 generates second posture information including a second model of the subject indicated by the query image and transmits it to the information processing device 100. The posture acquisition unit 113 acquires the second posture information from the analysis device 102.

[0123] The similarity obtaining unit 114 obtains the similarity between each of the first models included in the first posture information and the second posture information obtained in step S103 and the second model (step S104).

[0124] FIG. 11 is a flowchart showing a detailed example of the similarity obtaining process (step S104) according to this embodiment.

[0125] The element calculation unit 114b repeats steps S104b to S104c for each of the first models included in the first posture information acquired in step S103 (step S104a).

[0126] The element calculation unit 114b calculates the element similarity, which is the similarity for each corresponding model element between the first model and the second model (step S104b).

[0127] The overall calculation unit 114a acquires the weight information 112b stored in the storage unit 112, and calculates the overall similarity between the first model and the second model based on the element similarity and weight for each model element calculated in step S104b (step S104c).

[0128] For example, the overall calculation unit 114a calculates the sum of the products of the element similarities and weights of the corresponding model elements, and sets this sum as the overall similarity.

[0129] The overall calculation unit 114a repeats steps S104b to S104c for each of the first models included in the first posture information acquired in step S103, and as a result, of After performing steps S104b to S104c for each of the first models included in the first posture information acquired in step S103, overall calculation section 114a ends loop A (step S104a) and returns to the posture estimation process.

[0130] Referring again to FIG. The estimation unit 115 estimates the posture of the person being photographed shown in the query image, based on the query image acquired in step S102 and a plurality of reference images (step S105).

[0131] For example, the estimation unit 115 estimates the posture of the person shown in the query image acquired in step S102 based on the overall similarity between the query image and each of reference images 1 to 7. Note that in step S105, the estimation unit 115 may use any of the posture estimation methods 1 to 5 described above, or may use a method other than posture estimation methods 1 to 5, in order to estimate the posture of the person.

[0132] The estimation unit 115 determines whether or not a predetermined posture has been detected (step S106).

[0133] In detail, for example, if the subject shown in the query image is estimated to be in a predetermined posture in step S105, the estimation unit 115 determines that the predetermined posture has been detected. If the subject shown in the query image is estimated to be not in a predetermined posture in step S105, the estimation unit 115 determines that the predetermined posture has not been detected.

[0134] If it is determined that the predetermined posture has not been detected (step S106; No), the image acquisition unit 111 executes step S101 again.

[0135] If it is determined that a predetermined posture has been detected (step S106; Yes ), the display control unit 119 causes the display unit 118 to display that the predetermined posture has been detected (step S107). After that, the image acquisition unit 111 executes step S101 again.

[0136] In step S107, the display control unit 119 causes a query image showing the subject in a predetermined posture to be displayed on the display unit 118. The query image displayed here may be an image in which the subject is marked, as described above.

[0137] The user can know that a person to be photographed in a predetermined posture has been detected by looking at the display unit 118. For example, if the person is in a talking posture while operating an ATM, there is a possibility that the person may be a victim of a bank transfer fraud or a suspicious person, so the user can take measures such as notifying a security guard near the ATM to check.

[0138] By repeatedly performing such posture estimation processing, the posture of the subject can be estimated for each of the time-series query images.

[0139] Here, the plurality of frame images are, for example, images based on photography taken while an ATM is being operated. In this case, the plurality of frame images taken while the same person is operating the ATM and the query images therebetween are time-series images showing the same person being photographed.

[0140] Therefore, if the estimated postures of the subject shown in each of the time-series query images are different (i.e., if the postures are different from each other), there is a possibility that an error will be included in one of the estimation results in the posture estimation process. When a plurality of query images include query images with different estimation results, the display control unit 119 causes the display unit 118 to display the query images with different estimation results. The display control unit 119 may store the query images with different estimation results, and may cause the display unit 118 to display all of the query images with different estimation results in response to a user instruction or the like.

[0141] 12 is a flowchart showing an example of estimation support processing according to this embodiment. The estimation support processing is processing for displaying a query image whose posture may have been erroneously estimated in order to support estimation of the posture of the subject. The estimation support processing is executed while posture estimation processing is being performed. The estimation support processing may be executed repeatedly.

[0142] The determination unit 117 acquires the estimation results in step S105, which are repeatedly executed, and determines whether the estimation results for a plurality of query images correspond to an erroneous estimation pattern (step S201). This makes it possible to detect estimation results that correspond to an erroneous estimation pattern.

[0143] More specifically, the erroneous estimation pattern is, for example, a pattern of estimation results for each of a predetermined number of time-series query images. Here, the predetermined number may be equal to or greater than 2. As described above, the erroneous estimation pattern may include an estimation result for at least one query image that is different from the other query images.

[0144] Fig. 13 is a diagram showing an example of such an erroneous estimation pattern. In Fig. 13, "OK" indicates a positive estimation, and "NG" indicates a negative estimation. A positive estimation is a result of estimation that the posture is a predetermined posture. A negative estimation is a result of estimation that the posture is not a predetermined posture.

[0145] Pattern 1 shown in Fig. 13(a) shows an example of a pattern in which the estimation results for four query images in time series are "OK / NG / OK / NG" in that order. This pattern 1 is an example of a pattern in which the estimation results for the query images in time series are different and repeated a predetermined number of times or more. Fig. 13(a) shows an example in which the predetermined number of times is two.

[0146] For example, it is extremely rare for a photographed person to frequently change whether or not they are talking on a mobile phone while operating an ATM, and therefore it is extremely rare for their posture to frequently change whether or not they are in a talking position. The same is true for wheelchair postures. Therefore, if positive and negative inferences are repeated for a query image showing a photographed person while the same person is operating an ATM, it is highly likely that one of these inference results is incorrect. Therefore, by detecting the inference results in step S105 that correspond to pattern 1, it is possible to detect inference results that are highly likely to be incorrect.

[0147] Pattern 2 shown in Figure 13(b) shows an example of a pattern in which the estimation results for each of four query images in a time series are "OK / OK / OK / NG" in that order. This pattern 2 is an example of a pattern in which the image is estimated to be in a predetermined posture for a predetermined number of consecutive query images in a time series, and then immediately thereafter is estimated to be in a posture other than the predetermined posture. Figure 13(b) shows an example in which the predetermined number is 3.

[0148] For example, if the same person is operating an ATM in accordance with the instructions of a fraudster committing a bank transfer fraud, the mobile phone conversation is likely to continue until the ATM operation is completed, and a negative inference is likely to be incorrect. On the other hand, for a wheelchair position, it is extremely rare for the wheelchair position to end, and a negative inference is likely to be incorrect. Therefore, by detecting the inference results in step S105 that correspond to pattern 2, it is possible to detect inference results that are likely to be incorrect.

[0149] As exemplified in patterns 1 and 2, when at least one query image contains an estimation result that is different from other query images, the estimation result is likely to contain an error. Therefore, by detecting at least one query image containing an estimation result that is different from other query images, it is possible to detect estimation results that are likely to be incorrect.

[0150] Note that the erroneous estimation pattern is not limited to patterns such as patterns 1 and 2, in which at least one query image includes an estimation result that is different from that of other query images. The erroneous estimation pattern may be a pattern that includes a query image with different estimation results among multiple query images. This makes it possible to detect estimation results that may be incorrect. An example of such an erroneous estimation pattern is a pattern that includes an estimation result that is different from at least one of its preceding and succeeding ones in chronological order (if represented by "OK" and "NG" as in FIG. 13, one or both of a pattern in which the estimation results for each query image in the chronological order are "OK / NG" and a pattern in which "NG / OK" are respectively indicated).

[0151] Referring again to FIG. When it is determined that the estimation results for the plurality of query images do not correspond to an erroneous estimation pattern (step S201; No), the determining unit 117 repeats step S201.

[0152] When it is determined that the estimation results for the multiple query images correspond to an incorrect estimation pattern (step S201; Yes), the display control unit 119 acquires a query image related to the incorrect estimation pattern and displays the acquired query image on the display unit 118 (step S202). The query image related to the incorrect estimation pattern includes a query image that has been estimated differently from other query images among the multiple query images. At this time, the display control unit 119 may also display the estimation results for the acquired query image.

[0153] For example, suppose that the pattern corresponds to pattern 1 shown in Fig. 13(a). In this case, by executing step S202, if the estimation results for the time-series query images are different repeatedly a predetermined number of times or more, the display control unit 119 causes the display unit 118 to display the query images for which different estimations have been repeatedly made. In this case, the display control unit 119 may cause the display unit 118 to display at least one of the query images that correspond to pattern 1.

[0154] For example, suppose that the situation corresponds to pattern 2 shown in Fig. 13(b). In this case, by executing step S202, when a query image in a time series is estimated to be not in the predetermined posture immediately after being estimated to be in the predetermined posture for a predetermined number of consecutive query images or more, the display control unit 119 causes the display unit 118 to display the query image determined to be not in the predetermined posture.

[0155] Referring again to FIG. The display control unit 119 determines, for example, based on a user input, whether or not the result estimated in step S105 for the query image displayed in step S202 is incorrect (step S203).

[0156] In detail, for example, the user checks the query image displayed in step S202 and the estimation result for the query image by viewing them on the display unit 118. Then, the user operates the input unit 116 to input whether or not the result estimated in step S105 for the displayed query image is incorrect.

[0157] If it is determined that the estimated result is not an error (step S203; No), the display control unit 119 returns to step S201.

[0158] If it is determined that the estimated result is incorrect (step S203; Yes), the display control unit 119 causes the display unit 118 to display the reference image (step S204).

[0159] In detail, for example, by executing step S204, the display control unit 119 causes at least one of one or more reference images to be displayed on the display unit 118 when there is a query image with an incorrect estimation result among the query images in a time series for which different estimations have been made.

[0160] The reference image displayed here is the reference image used in step S105 when estimating the query image displayed in step S202. More specifically, the reference image displayed here is a reference image showing a person whose similarity to the subject shown in the erroneously estimated query image satisfies a predetermined criterion. The predetermined criterion may be the highest similarity, a similarity equal to or greater than a threshold, etc.

[0161] The display control unit 119 determines whether or not a predetermined instruction for displaying the thinned frame images has been received, for example, based on a user input (step S205).

[0162] If it is determined that a predetermined instruction has not been received (step S205; No), the display control unit 119 returns to step S201.

[0163] If it is determined that a predetermined instruction has been received (step S205; Yes), the display control unit 119 acquires the frame images thinned out in step S102 from the image acquisition unit 111 (step S206).

[0164] In detail, for example, the display control unit 119 acquires at least one frame image that is thinned out between the query images in the time series for which different determinations have been made.

[0165] For example, suppose that the case corresponds to pattern 1 shown in Fig. 13(a). In this case, for example, the display control unit 119 acquires at least one of the frame images acquired and thinned out by the image acquisition unit 111 during the time between the positively inferred query image and the negatively inferred query image.

[0166] For example, suppose that the case corresponds to pattern 2 shown in Fig. 13(b). In this case, for example, the display control unit 119 acquires at least one of the frame images acquired and thinned out by the image acquisition unit 111 during the time between the negatively inferred query image and the positively inferred query image.

[0167] The image capturing unit 101 may store the frame images, and the display control unit 119 may acquire the thinned frame images from the image capturing unit 101. In this way, the frame images transmitted from the image capturing unit 101 to the information processing device 100 may be Number of Therefore, the communication fee between the image capturing unit 101 and the information processing device 100 can be reduced.

[0168] Referring again to FIG. The posture acquisition unit 113, the similarity acquisition unit 114, and the estimation unit 115 perform the same processes as steps S103 to S105 of the posture estimation process.

[0169] In detail, for example, the posture acquisition unit 113 receives an instruction from the display control unit 119 to acquire first posture information based on a plurality of reference images associated with a predetermined posture and second posture information based on the frame image acquired in step S206 (step S103).

[0170] The similarity obtaining unit 114 obtains the similarity between each of the first models included in the first posture information and the second posture information obtained in step S103 and the second model (step S104).

[0171] The estimation unit 115 estimates the posture of the person shown in the frame image acquired in step S206 based on the frame image and a plurality of reference images (step S105). In step S105, the estimation unit 115 estimates the posture of the person shown in the query image based on the overall similarity between the frame image acquired in step S206 and each of reference images 1 to 7.

[0172] That is, by executing step S105 here, the estimation unit 115 estimates the posture of the person being photographed shown in the frame image based on at least one thinned-out frame image and the reference image.

[0173] The display control unit 119 causes the frame image acquired in step S206 and the result of estimation for that frame image in step S105 to be displayed on the display unit 118 (step S207), and the process returns to step S201.

[0174] By executing step S207, the display control unit 119 causes the display unit 118 to display at least one frame image that has been thinned out between the query images in the time series for which different determinations have been made.

[0175] By executing the estimation assistance process, query images that may have been estimated incorrectly, reference images related thereto, and thinned frame images can be displayed on the display unit 118. Furthermore, the thinned frame images and reference images can be used to display the results of estimation of whether or not the posture is a predetermined posture on the display unit 118. This allows the user to know which query images were erroneously estimated. Furthermore, the user can know which reference images were used to make the erroneous estimation.

[0176] (Actions and Effects) As described above, according to this embodiment, the information processing device 100 includes the estimation unit 115 and the display control unit 119.

[0177] The estimation unit 115 estimates the posture of the subject shown in each of the multiple query images based on multiple query images obtained by photographing the subject performing a predetermined behavior multiple times and a reference image showing a person associated with a predetermined posture.

[0178] When a query image with a different estimation result is included among the plurality of query images, the display control unit 119 causes the display unit 118 to display the query image with the different estimation result.

[0179] Generally, when multiple query images include query images with different estimation results, the estimation results may contain errors. By displaying such query images, the user can refer to the query images for which different estimations have been made. The user can then check the accuracy of the estimated pose for each query image and take measures to improve the accuracy of pose estimation, such as deleting reference images that may be causing the incorrect estimation. Therefore, it is possible to improve the accuracy of estimating the pose of the subject shown in the image.

[0180] According to this embodiment, the plurality of query images are time-series query images, which makes it possible to improve the accuracy of estimating the posture of the subject shown in the time-series query images.

[0181] According to this embodiment, when the estimation results for a time-series query image are repeatedly different a predetermined number of times or more, the display control unit 119 causes the display unit 118 to display the query image for which the repeatedly different estimations have been made.

[0182] This allows the user to check query images that are likely to have been erroneously estimated and take measures to improve the accuracy of estimating the posture, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0183] According to this embodiment, the information processing device 100 includes a similarity obtaining unit 114 that obtains a similarity between the postures of a person shown in the query image and a person shown in the reference image for each combination of a time-series query image and one or more reference images.

[0184] The estimation unit 115 estimates the posture of the person shown in each of the time-series query images based on the similarity. When there is a query image with an incorrect estimation result among the time-series query images for which different estimations have been repeatedly performed, the display control unit 119 causes the display unit 118 to display, among one or more reference images, a reference image showing a person whose similarity to the person shown in the query image with the incorrect estimation satisfies a predetermined criterion.

[0185] This makes it possible to improve the accuracy of estimating the posture, for example by checking reference images that may be the cause of the erroneous estimation and deleting the reference images that may be the cause of the erroneous estimation, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0186] According to this embodiment, the information processing device 100 includes an image acquisition unit 111 that acquires time-series frame images obtained by capturing images of a subject multiple times in a time-series manner. The multiple query images are frame images obtained by thinning out some of the time-series frame images.

[0187] The display control unit 119 further causes the display unit 118 to display at least one frame image that has been thinned out between the query images in the time series for which different determinations have been made.

[0188] This allows for checking frame images captured at a time close to the query image, which is likely to have been erroneously estimated, and measures to improve the accuracy of posture estimation, thereby improving the accuracy of posture estimation of the subject shown in the image.

[0189] According to this embodiment, the estimation section 115 further estimates the posture of the person being photographed shown in the frame image based on at least one of the thinned frame images and the reference image.

[0190] This allows for checking the estimation results for frame images captured at a time close to the query image, which is likely to have been erroneously estimated, and measures can be taken to improve the accuracy of estimating the posture, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0191] According to this embodiment, when a predetermined posture is estimated to be present for a predetermined number of consecutive query images in a time series, and then the posture is estimated to be not present, the display control unit 119 determines that the posture is not present. Estimate The selected query image is displayed on the display unit 118.

[0192] This allows the user to check query images that are likely to have been erroneously estimated and take measures to improve the accuracy of estimating the posture, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0193] According to this embodiment, the information processing device 100 further includes a determination unit 117 that determines whether or not a query image with different estimation results is included among the plurality of query images.

[0194] This makes it possible to detect estimation results that are likely to be incorrect, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0195] According to this embodiment, the determination unit 117 determines whether or not the estimation results for the multiple query images correspond to a predetermined erroneous estimation pattern. The erroneous estimation pattern is a pattern of estimation results related to the posture of the subject included in each of the multiple query images, and includes an estimation result for at least one query image that is different from that for the other query images. When the determination unit 117 determines that the estimation results for the multiple query images correspond to an erroneous estimation pattern, the display control unit 119 causes the display unit 118 to display a query image related to the erroneous estimation pattern.

[0196] This makes it possible to detect estimation results that are likely to be incorrect and confirm the estimation results. Then, it is possible to check the accuracy of the estimated pose for each query image and take measures to improve the accuracy of pose estimation, such as deleting reference images that may be causing the incorrect estimation. Therefore, it is possible to improve the accuracy of estimating the pose of the subject shown in the image.

[0197] According to this embodiment, the erroneous estimation pattern includes at least one of a pattern in which the estimation results for a time-series query image are different a predetermined number of times or more, and a pattern in which a time-series query image is estimated to be in a predetermined posture for a predetermined number of consecutive times or more, and then immediately thereafter is estimated to be in a posture that is not the predetermined posture.

[0198] This makes it possible to detect estimation results that are likely to be incorrect, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0199] According to this embodiment, multiple images are captured while the ATM is being operated, which improves the accuracy of estimating the posture of the person being photographed in the images captured while the ATM is being operated.

[0200] According to this embodiment, the plurality of query images are images showing a common subject, which makes it possible to improve the accuracy of estimating the posture of the common subject shown in the images.

[0201] According to this embodiment, the information processing device 100 includes an estimation unit 115 and a display control unit 119.

[0202] The estimation unit 115 estimates the posture of the person being photographed shown in the query image based on a query image obtained by photographing the person while performing a predetermined action and one or more reference images showing a person corresponding to a predetermined posture.

[0203] When the estimation unit 115 makes an incorrect estimation, the display control unit 119 causes the display unit 118 to display, from among one or more reference images, a reference image showing a person whose posture similarity with the subject shown in the query image in which the incorrect estimation was made satisfies a predetermined standard.

[0204] This makes it possible to improve the accuracy of estimating the posture, for example by checking reference images that may be the cause of the erroneous estimation and deleting the reference images that may be the cause of the erroneous estimation, thereby improving the accuracy of estimating the posture of the subject shown in the image.

[0205] <Embodiment 2> In the first embodiment, an example has been described in which multiple images are taken at successive (different) times in terms of time. However, the multiple images may be taken from two or more different directions at the same time.

[0206] 14 is a diagram showing a detailed example of the functional configuration of an information processing system S2 according to embodiment 2. The information processing system S2 includes two imaging units 101, an information processing device 100, and an analysis device 102. Note that the number of imaging units 101 may be three or more.

[0207] The two imaging units 101 are cameras that capture a common area, such as the area in front of an ATM. Therefore, the two imaging units 101 can capture images of a common subject from different directions at the same time. Each of the imaging units 101 may be functionally and physically similar to the imaging unit 101 according to the first embodiment.

[0208] The information processing device 100 and the analysis device 102 may be functionally and physically similar to those of the first embodiment. For example, when a plurality of query images includes a query image with a different estimation result, the display control unit 119 causes the display unit 118 to display the query image with the different estimation. The information processing system S2 may operate in the same manner as the information processing system S1 according to the first embodiment.

[0209] (Actions and Effects) As described above, according to this embodiment, multiple photographs are taken from two or more different directions at the same time.

[0210] This allows the user to refer to query images that were taken around the same time and for which different estimations were made, check whether the estimated posture for each query image is correct, and take measures to improve the accuracy of posture estimation, such as deleting reference images that may be causing incorrect estimations. Therefore, it is possible to improve the accuracy of estimating the posture of the subject shown in the image.

[0211] Although the embodiments and modifications of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.

[0212] In addition, although the flowcharts used in the above description show multiple steps (processes) in a sequential order, the order of steps performed in each embodiment is not limited to the order shown. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of content. Furthermore, the above-described embodiments and variations can be combined as long as the content is not contradictory.

[0213] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0214] 1. An estimation means for estimating a posture of a person shown in a query image based on a query image obtained by photographing the person performing a predetermined action and one or more reference images showing a person corresponding to a predetermined posture; and a display control means for, when the estimation means makes an erroneous estimation, displaying, on a display means, one of the one or more reference images showing a person whose similarity in posture to the subject shown in the query image in which the erroneous estimation was made satisfies a predetermined standard. Information processing device. 2. An estimation means for estimating a posture of the subject shown in each of the plurality of query images based on a plurality of query images obtained by photographing the subject performing a predetermined action multiple times and a reference image showing a person associated with a predetermined posture; a display control means for, when a query image with a different estimation result is included among the plurality of query images, displaying the query image with the different estimation result on a display means; Information processing device. 3. The plurality of query images are time-series query images. 2. An information processing device according to the present invention. 4. When the estimation results for the time-series query images are different a predetermined number of times or more, the display control means causes the display means to display the query images for which the estimations are different repeatedly. 3. An information processing device according to the present invention. 5. The reference image is one or more; a similarity obtaining unit for obtaining a similarity between a posture of a subject shown in the query image and a posture of a person shown in the reference image for each combination of the time-series query image and the one or more reference images; the estimation means estimates a posture of the person shown in each of the time-series query images based on the similarity; When there is a query image with an incorrect estimation result among the time-series query images for which different estimations have been repeatedly performed, the display control means displays, on the display means, the reference image showing a person whose similarity to the subject shown in the query image with the incorrect estimation satisfies a predetermined criterion among the one or more reference images. 4. An information processing device according to the present invention. 6. The image capturing device further comprises an image capturing means for capturing time-series frame images obtained by capturing the subject a plurality of times in a time-series manner, the plurality of query images are frame images obtained by thinning out a portion of the time-series frame images, The display control means further causes the display means to display at least one frame image thinned out between the query images in the time series for which different determinations have been made. 6. The information processing device according to any one of 3. to 5. 7. The estimation means further estimates the posture of the person being photographed shown in the frame image based on the at least one thinned-out frame image and the reference image. 6. An information processing device according to the present invention. 8. The display control means causes the display means to display the query image determined not to be in the predetermined posture when the query image is determined not to be in the predetermined posture immediately after being determined to be in the predetermined posture for a predetermined number of consecutive query images in the time series. 3. An information processing device according to the present invention. 9. The apparatus further includes a determination unit for determining whether or not the plurality of query images includes a query image with a different estimation result. 9. An information processing device according to any one of 1. to 8. 10. The determination means determines whether or not the estimation results for the plurality of query images correspond to a predetermined erroneous estimation pattern; the erroneous estimation pattern is a pattern of estimation results regarding the posture of the subject included in each of a plurality of query images, and includes an estimation result for at least one query image that is different from an estimation result for other query images; When the determination means determines that the estimation results for the plurality of query images correspond to the erroneous estimation pattern, the display control means causes the display means to display a query image related to the erroneous estimation pattern. 9. An information processing device according to the present invention. 11. The erroneous estimation pattern includes at least one of a pattern in which the estimation results for the time-series query images are different a predetermined number of times or more, and a pattern in which the posture is estimated to be different from the predetermined posture immediately after the posture is estimated to be the predetermined posture for the time-series query images a predetermined number of times or more consecutively. 10. An information processing device according to the present invention. 12. The multiple photographs are taken simultaneously from two or more different directions. 2. An information processing device according to the present invention. 13. The multiple photographs are taken while the ATM is being operated. 13. An information processing device according to any one of 2. to 12. 14. The plurality of query images are images showing a common subject. 10. An information processing device according to any one of 1. to 10. 15. An information processing device according to any one of 1. to 14.; one or more imaging units that perform the multiple imaging operations; Information processing system. 16. The computer estimating a posture of a person shown in each of the plurality of query images based on a plurality of query images obtained by taking a plurality of photographs while the person is performing a predetermined action and a reference image showing the person corresponding to a predetermined posture; When a query image with a different estimation result is included in the plurality of query images, the query image with the different estimation result is displayed on a display means. Information processing methods. 17. To the computer, estimating a posture of a person shown in each of the plurality of query images based on a plurality of query images obtained by taking a plurality of photographs while the person is performing a predetermined action and a reference image showing the person corresponding to a predetermined posture; A recording medium having a program recorded thereon for causing a display means to display, when the plurality of query images includes a query image with a different estimation result, the query image with the different estimation result. 18. To the computer, estimating a posture of a person shown in each of the plurality of query images based on a plurality of query images obtained by taking a plurality of photographs while the person is performing a predetermined action and a reference image showing the person corresponding to a predetermined posture; and a program for causing a display unit to display, when the plurality of query images includes a query image with a different estimation result, the query image with the different estimation result. [Explanation of symbols]

[0215] S1, S2 Information Processing System 100 Information processing device 101 Photography Department 102 Analysis equipment 111 Image acquisition unit 112 Storage section 112a References 112b Weight information 113 Attitude acquisition part 114 Similarity acquisition unit 114a Overall calculation section 114b Element calculation part 115 Estimation Department 116 Input section 117 Judgment section 118 Display section 119 Display control unit

Claims

1. an estimation means for estimating a posture of the subject shown in each of the plurality of query images based on a plurality of query images obtained by photographing the same subject performing a predetermined action multiple times and a reference image showing a person associated with a predetermined posture; a display control means for, when a query image with a different estimation result is included among the plurality of query images, displaying the query image with the different estimation result on a display means; The plurality of query images are time-series query images. Information processing device.

2. The display control means, when the estimation results for the time-series query images are different a predetermined number of times or more, causes the display means to display the query images for which the estimations are different repeatedly. The information processing device according to claim 1 .

3. the reference image is one or more; a similarity obtaining unit for obtaining a similarity between a posture of a subject shown in the query image and a posture of a person shown in the reference image for each combination of the time-series query image and the one or more reference images; the estimation means estimates a posture of the person shown in each of the time-series query images based on the similarity; When there is a query image with an incorrect estimation result among the time-series query images for which different estimations have been repeatedly performed, the display control means displays, on the display means, the reference image showing a person whose similarity to the subject shown in the query image with the incorrect estimation satisfies a predetermined criterion among the one or more reference images. The information processing device according to claim 2 .

4. further comprising an image acquisition means for acquiring time-series frame images obtained by photographing the subject multiple times in succession over time; the plurality of query images are frame images obtained by thinning out a portion of the time-series frame images, The display control means further causes the display means to display at least one frame image thinned out between the query images in the time series for which different determinations have been made. The information processing device according to claim 1 .

5. The estimation means further estimates the posture of the person being photographed shown in the frame image based on the at least one thinned-out frame image and the reference image. The information processing device according to claim 4 .

6. When the query images in the time series are estimated to be in the predetermined posture for a predetermined number of consecutive times or more, and then the query images are estimated to be in a different posture from the predetermined posture, the display control means causes the display means to display the query images determined to be in a different posture from the predetermined posture. The information processing device according to claim 1 .

7. The device further includes a determination unit for determining whether or not a query image with a different estimation result is included among the plurality of query images. The information processing device according to claim 1 .

8. The computer Estimating a posture of the subject shown in each of the plurality of query images based on a plurality of query images obtained by photographing the same subject performing a predetermined behavior multiple times and a reference image showing a person corresponding to a predetermined posture; When a query image with a different estimation result is included among the plurality of query images, displaying the query image with the different estimation result on a display means, The plurality of query images are time-series query images. Information processing methods.

9. On the computer, Estimating a posture of the subject shown in each of the plurality of query images based on a plurality of query images obtained by photographing the same subject performing a predetermined behavior multiple times and a reference image showing a person corresponding to a predetermined posture; When a query image with a different estimation result is included among the plurality of query images, the query image with the different estimation result is displayed on a display means; The program, wherein the plurality of query images are time-series query images.

Images