Information processing device, information processing method, and program

Abstract

An information processing device (100) comprises a subject image acquisition unit (101) and a position estimation unit (110). The subject image acquisition unit (101) acquires an estimation subject image which includes an image of a subject site of an estimation subject. The position estimation unit (110), on the basis of a learning result obtained by performing learning using a reference image including an image of the subject site, and a subject image including an image of the subject site, extracts a subject site feature pertaining to a part corresponding to the subject site from a feature of the estimation subject image, and estimates the position of the subject site on the basis of the subject site feature.

Description

Information processing device, information processing method, information processing system, and recording medium

[0001] The present disclosure relates to an information processing device, an information processing method, an information processing system, and a recording medium.

[0002] Various techniques have been proposed for acquiring information about human eyes from a target image.

[0003] For example, Patent Document 1 discloses an object tracking method that uses a tracking technique based on template matching to detect a specific object from captured images acquired in time series and track its position.

[0004] For example, Patent Document 2 discloses a technology for determining whether a second image capturing at least one of a user's right eye and left eye is of the user's left eye or right eye based on a first image including the user's entire body.

[0005] JP 2014-063280 A International Publication No. 2020 / 079741

[0006] The present disclosure aims to improve upon the techniques described in the prior art documents mentioned above.

[0007] According to one aspect of the disclosure, there is provided an information processing device comprising: a target image acquisition means for acquiring an estimation target image including an image of a target part in an estimation target; and a position estimation means for extracting target part features for a part corresponding to the target part from features of the estimation target image based on a learning result obtained by using a reference image including an image of the target part and a target image including the image of the target part, and estimating a position of the target part based on the target part features.

[0008] According to one aspect of the present disclosure, there is provided an information processing system including the above-described information processing device and an imaging device for imaging the estimation target.

[0009] According to one aspect of the present disclosure, there is provided an information processing method in which one or more computers acquire an estimation target image including an image of a target part in an estimation target, extract target part features related to a part corresponding to the target part from features of the estimation target image based on a learning result obtained by using a reference image including the image of the target part and a target image including the image of the target part, and estimate a position of the target part based on the target part features.

[0010] According to one aspect of the disclosure, there is provided a recording medium having recorded thereon a program for causing one or more computers to acquire an estimation target image including an image of a target part in an estimation target, extract target part features relating to a part corresponding to the target part from features of the estimation target image based on a learning result obtained using a reference image including an image of the target part and a target image including the image of the target part, and estimate the position of the target part based on the target part features.

[0011] 1 is a diagram illustrating an overview of an information processing device according to embodiment 1. FIG. 1 is a diagram illustrating an overview of information processing according to embodiment 1. FIG. 2 is a diagram illustrating an example of an estimation target image according to embodiment 1. FIG. 3 is a diagram illustrating an example of a reference image according to embodiment 1. FIG. 4 is a diagram illustrating an example of a functional configuration of an information processing device according to embodiment 1. FIG. 5 is a diagram illustrating an example of a similarity map according to embodiment 1. FIG. 6 is a diagram illustrating an example of a physical configuration of an information processing device according to embodiment 1. FIG. 7 is a flowchart illustrating a detailed example of information processing according to embodiment 1. FIG. 8 is a flowchart illustrating a detailed example of extraction processing according to embodiment 1. FIG. 9 is a diagram illustrating an example of a change in data size from an estimation target image to acquisition of feature amounts in embodiment 1. FIG. 10 is a diagram conceptually illustrating an example of a method for generating similarity information according to embodiment 1. FIG. 11 is a flowchart illustrating an example of similarity generation processing according to embodiment 1. FIG. 12 is a flowchart illustrating a detailed example of estimation processing according to embodiment 1. FIG. 13 is a conceptual diagram illustrating an example of the data size and data configuration of a reference feature amount Rref according to embodiment 2. FIG. 14 is a diagram illustrating an example of a functional configuration of an information processing device according to embodiment 3. FIG. 15 is a flowchart illustrating an example of information processing according to embodiment 3. FIG. 16 is a diagram illustrating an example of a functional configuration of an information processing device according to embodiment 4. FIG. 17 is a flowchart illustrating an example of gaze acquisition processing according to embodiment 4. FIG. 18 is a diagram illustrating an example of a functional configuration of an information processing device according to embodiment 5. FIG. 19 is a flowchart illustrating an example of learning processing according to embodiment 5. FIG. 19 is a flowchart illustrating an example of learning processing according to embodiment 5. 10 is a diagram illustrating an example of the functional configuration of an information processing device according to embodiment 8. FIG. 11 is a flowchart illustrating an example of display processing according to embodiment 8. FIG. 12 is a diagram illustrating an example of a display image according to embodiment 8. FIG. 13 is a diagram illustrating an example of the configuration of an information processing system according to embodiment 9.

[0012] Hereinafter, embodiments will be described with reference to the drawings. In all drawings, similar components are denoted by similar reference numerals, and descriptions thereof will be omitted where appropriate.

[0013] 1 is a diagram showing an overview of an information processing device 100 according to embodiment 1. The information processing device 100 includes a target image acquisition unit 101 and a position estimation unit 110.

[0014] The target image acquisition unit 101 acquires an estimation target image including an image of a target part in an estimation target.

[0015] The position estimation unit 110 extracts target part features relating to the part corresponding to the target part from the features of the estimated target image based on the learning results obtained using a reference image including an image of the target part and a target image including an image of the target part, and estimates the position of the target part based on the target part features.

[0016] According to this information processing device 100, it is possible to estimate the position of the target area with high accuracy.

[0017] FIG. 2 is a diagram illustrating an overview of information processing according to the first embodiment.

[0018] The target image acquisition unit 101 acquires an estimation target image including an image of a target part in an estimation target (step S110).

[0019] The position estimation unit 110 extracts target part features relating to the part corresponding to the target part from the features of the estimated target image based on the learning results obtained using a reference image including an image of the target part and a target image including an image of the target part, and estimates the position of the target part based on the target part features (step S110).

[0020] This information processing makes it possible to estimate the position of the target area with high accuracy.

[0021] A detailed example of the first embodiment will be described below.

[0022] (Detailed Example) For example, in the object tracking method described in Patent Document 1, the position of an object is detected by template matching. In template matching, as described in Patent Document 1, similarity is calculated at each position while shifting the relative positional relationship between a matching target area and a template, and the position of the object is detected based on the similarity. However, with such template matching, it is difficult to detect the position of the object in more detail than the amount of shift in the relative positional relationship.

[0023] Furthermore, for example, even if the technology described in Patent Document 2 can determine whether the second image is an image of the left or right eye, it does not disclose a technology for accurately estimating the position of a target part, such as the pupil.

[0024] There is a demand for a technique for accurately estimating the position of a target part, such as the position of a person's pupil.

[0025] In view of the above circumstances, one example of the purpose of this disclosure is to provide an information processing device, an information processing method, a recording medium having a program recorded thereon, and the like that solve the problem of estimating the position of a target region with high accuracy.

[0026] The information processing device 100 is a device for estimating the position of a target part in an estimation object. The information processing device 100 acquires an estimation object image including the target part in the estimation object, and estimates the position of the target part based on a learning result obtained by learning using a reference image and the target image.

[0027] The estimation target is a target for which the position of a target part is to be estimated. The target is, for example, a person, an object, etc. If the target is an object, the target is, for example, a machine, an animal, a plant, etc. The machine is, for example, a robot, an automobile, a machine tool, etc.

[0028] The estimation target according to this embodiment is a person.

[0029] The target part is a part of the part included in the estimation object that is predetermined as a part whose position is to be estimated. When the estimation object is a person, the target part is, for example, at least one of the center of the pupil (pupil center), the outer corner of the eye, the inner corner of the eye, the tip of the nose, the center of the nostrils, the corner of the mouth, etc. Furthermore, for example, in the case of multiple parts provided on the left and right sides of the estimation object, the target parts may be predetermined by distinguishing between the left and right, such as the pupil center of the left eye and the pupil center of the right eye.

[0030] The target area according to this embodiment is the center of the pupil.

[0031] The estimation target image includes an image of a target part of the estimation target. Fig. 3 is a diagram showing an example of the estimation target image according to this embodiment. The estimation target image according to this embodiment is a binocular image including both eyes of the estimation target. Note that the estimation target image may also be a single-eye image.

[0032] Both the reference image and the target image include an image of the target region.

[0033] The reference image is an image used as a reference for estimating the position of a target part in an estimation object, and is prepared in advance. The reference image is used in common in both cases of performing learning to estimate the position of a target part in an estimation object and estimating the position of a target part in an estimation object.

[0034] 4 is a diagram showing an example of a reference image according to this embodiment. In this embodiment, the reference image is a right-eye image. The reference image may be a right-eye image of an appropriate object (a person in this embodiment) different from the estimation object.

[0035] The reference image may also be an image generated by image processing, etc. For example, the reference image may be an average image obtained by averaging right-eye images of multiple objects (people in this embodiment).

[0036] The reference image may be, for example, a left-eye image as long as it includes only a predetermined one of the left and right eyes. Alternatively, the reference image may be a one-eye image including only one of the eyes (a mixed image including both a right-eye image and a left-eye image).

[0037] The target image is an image used when performing learning to estimate the position of a target part in an estimation target. The target image according to this embodiment is a binocular image including both eyes of a person, similar to the estimation target image. The target part included in the target image may be the target part of the estimation target, or may be the target part of an object different from the estimation target.

[0038] (Detailed Example of Functional Configuration of Information Processing Apparatus 100) FIG. 5 is a diagram showing an example of the functional configuration of the information processing apparatus 100 according to the first embodiment.

[0039] The information processing device 100 functionally comprises a target image acquisition unit 101 , a reference image acquisition unit 102 , a storage unit 103 , and a position estimation unit 110 .

[0040] The target image acquisition unit 101 acquires an estimation target image as described above.

[0041] The estimation target image is, for example, an image captured in real time. Examples of estimation target images captured in real time include images captured for biometric authentication such as iris authentication and face authentication, and surveillance images captured by a surveillance camera.

[0042] The target image acquisition unit 101 may acquire, for example, estimation target images obtained by each of one or more imaging devices, such as a visible light camera or a near-infrared camera, that capture an image of the estimation target via a communication network, etc. Alternatively, for example, the target image acquisition unit 101 may acquire, via a communication network, an estimation target image obtained by capturing an image using an imaging device mounted on a terminal device (e.g., a smartphone or a tablet terminal) used by the estimation target. The target image acquisition unit 101 may acquire multiple estimation target images for the same estimation target.

[0043] The reference image acquisition unit 102 acquires a reference image. The reference image is stored in advance in, for example, the storage unit 103. In this case, the reference image acquisition unit 102 may acquire the reference image from the storage unit 103. Note that the reference image acquisition unit 102 may also acquire the reference image from another device (not shown) via a communication network or the like.

[0044] The storage unit 103 stores various types of information. The information stored in the storage unit 103 may include, for example, a learning result obtained by learning using a reference image and a target image. Also, for example, the information stored in the storage unit 103 may include a reference image acquired by the target image acquisition unit 101. Furthermore, for example, the storage unit 103 may store an estimated target image acquired by the target image acquisition unit 101.

[0045] The learning result includes, for example, at least one of a feature extraction model and a position estimation model. The learning methods for the feature extraction model and the position estimation model will be described in other embodiments.

[0046] (Learning Result Example 1: Feature Extraction Model) The feature extraction model is a trained model that has undergone machine learning to extract features of an estimation target image.

[0047] For example, the feature quantity of the estimation target image can be extracted using a feature quantity extraction model with the estimation target image as input. Also, for example, the reference feature quantity can be extracted using a feature quantity extraction model with the reference image as input. The reference feature quantity is the feature quantity of the reference image.

[0048] The feature extraction model is a model including a CNN (Convolutional Neural Network) represented by, for example, ResNet, VGGNet, GoogleNet, ResNext, SENet, EfficientNet, etc. In this case, the feature is a so-called CNN feature.

[0049] (Learning result example 2: position estimation model) The position estimation model is a trained model that has undergone machine learning to estimate an in-region position. The in-region position indicates the position of the target part in the corresponding region. The corresponding region is the region that corresponds to the target part feature amount. The target part feature amount is part of the feature amount of the estimation target image, and is a feature amount related to the part that corresponds to the target part.

[0050] The corresponding region according to this embodiment is a region u included in similarity information, which will be described later. Note that the corresponding region is not limited to this, and may be, for example, a region in the estimation target image, a region in real space corresponding to the estimation target image, or the like.

[0051] For example, the position of the target part in the estimation target area can be estimated using a position estimation model with the feature amount of the estimation target image as input.

[0052] The location estimation model is a model including, for example, a linear regression model, a neural network (NN), a CNN, and the like.

[0053] (Position estimation unit 110) The position estimation unit 110 extracts target part features from the features of the estimation target image based on the learning results obtained using the reference image and the target image, and estimates the position of the target part in the estimation target based on the target part features.

[0054] In this embodiment, an example will be described in which the position estimation unit 110 estimates the pupil center of the right eye of the estimation target. Note that the position estimation unit 110 may estimate the pupil centers of both eyes included in the estimation target image based on a reference image that is a right-eye image.

[0055] As shown in FIG. 5 , the position estimation unit 110 includes an extraction unit 120 and an estimation unit 130 .

[0056] (Extraction Unit 120) The extraction unit 120 generates similarity information, which will be described later, and extracts target region features from the features of the estimation target image based on the similarity information.

[0057] As shown in FIG. 5 , the extraction unit 120 includes, for example, a feature acquisition unit 121 , a reference feature acquisition unit 122 , a generation unit 123 , and a feature extraction unit 124 .

[0058] The feature amount acquiring unit 121 receives the estimation target image as input and acquires the feature amount of the estimation target image using a feature amount extraction model.

[0059] The reference feature acquisition unit 122 receives a reference image as an input and acquires reference features, which are features of the reference image, using a feature extraction model. The reference features may be stored in advance in the storage unit 103. In this case, the reference feature acquisition unit 122 may acquire the reference features from the storage unit 103, and the reference image acquisition unit 102 may not be provided in the information processing device 100.

[0060] The generating unit 123 generates similarity information based on the feature amount of the estimation target image and the reference feature amount.

[0061] In more detail, for example, the generation unit 123 calculates the similarity between each of a plurality of portions included in the feature of the estimation target image and the reference feature. Each of the plurality of portions included in the feature has, for example, the same size as the reference feature. The similarity is an index that indicates a larger value the more similar a portion of the feature and the reference feature are. In this embodiment, the similarity is the spatial cosine similarity between the portion of the feature of the estimation target image and the reference feature.

[0062] The similarity is not limited to this, and for example, an index showing a smaller value may be used as the similarity between the feature portion and the reference feature portion increases.

[0063] (Similarity Information) The similarity information is information indicating the similarity between the feature of the image to be estimated and the reference feature. The similarity information includes, for example, a similarity map indicating, for each of a plurality of parts included in the feature, the region u corresponding to that part and the similarity between that part and the reference feature calculated for that part in association with each other.

[0064] 6 is a diagram showing an example of a similarity map according to this embodiment. Note that the similarity map may be any information that associates a region u corresponding to each of a plurality of portions included in the feature with the similarity for that portion, and may be represented, for example, as a table that associates the position of the region u with the similarity.

[0065] In this embodiment, since the reference image is a right-eye image, the similarity is large in the region u corresponding to the vicinity of the pupil of the right eye to be estimated. Therefore, Fig. 6 shows an enlarged view of the vicinity of the pupil of the right eye out of the entire region of the similarity map.

[0066] The overall region of the similarity map corresponds to the entire feature of the image to be estimated. The similarity map is divided into a plurality of regions u (small, roughly square regions surrounded by dotted lines in FIG. 6 ). As will be described in detail later, each of the regions u corresponds one-to-one to a corresponding one of the portions included in the feature of the image to be estimated.

[0067] In this embodiment, an example will be described in which the entire region of the similarity map is composed of 32 regions u arranged in the x direction (horizontal direction in FIG. 6 ) and 8 regions u arranged in the y direction (vertical direction in FIG. 6 ). Also, in this embodiment, an example will be described in which each region u is identified based on its position in each of the x and y directions, counting from the top left of the region map shown in FIG. 6 . Specifically, the region u that is the ith region in the x direction (i is an integer from 1 to 32) and the jth region in the y direction (j is an integer from 1 to 32), counting from the top left of FIG. 6 , is represented as region u[i, j]. That is, in FIG. 6 , regions u where i is 1 to 14 are shown within the entire region.

[0068] In the similarity map shown in Fig. 6, multiple regions u are represented by a density (grayscale) according to the similarity between the corresponding feature portion and the reference feature. The greater the corresponding similarity of a region u, the darker the region u is filled in. In the similarity map shown in Fig. 6, for example, the region u[9,5] is filled in the darkest, which indicates that the similarity of region u[9,5] is the greatest.

[0069] As a result, in the similarity map, the regions u corresponding to each of the multiple portions included in the feature amount are associated with the similarity.

[0070] The regions u constituting the overall region of the similarity map may be rectangular or have other shapes other than squares. The number of regions u constituting the overall region and the method for identifying each of them may be changed as appropriate. Furthermore, the similarity in the similarity map is not limited to grayscale intensity, but may be represented by the similarity itself (i.e., a value), a graded index corresponding to the similarity, or at least one of color and intensity. The index may be a letter, a symbol, a number, or the like.

[0071] Referring again to Fig. 5, the feature amount extraction unit 124 extracts target portion features from the features of the estimation target image based on the similarity information.

[0072] In detail, for example, the feature amount extraction unit 124 extracts target portion feature amounts from the feature amounts of the estimation target image based on predetermined specific conditions and similarity information.

[0073] The specific condition is a condition for identifying a corresponding region from among the regions u included in the similarity information. The specific condition according to this embodiment is that the region u has the highest similarity. The corresponding region can also be referred to as a region u that is highly likely to correspond to the target region (in this embodiment, the center of the pupil).

[0074] The specific condition is not limited to this, and may be, for example, a threshold value for the similarity or more, etc. Furthermore, a plurality of corresponding regions may be identified based on the specific condition.

[0075] The feature extraction unit 124 according to this embodiment identifies the maximum similarity from the similarity information in accordance with specific conditions. Then, the feature extraction unit 124 extracts a portion corresponding to the identified similarity from the feature of the estimation target image. The portion of the feature extracted in this way is the target portion feature.

[0076] That is, in this embodiment, the target portion feature amount is a portion of the feature amount of the estimation target image that corresponds to the specified similarity. In other words, the portion that corresponds to the specified similarity is the portion that was used to calculate the specified similarity. Therefore, the similarity information may further include information that associates the similarity with the portion of the feature amount that was used to calculate it.

[0077] (Estimation Unit 130) The estimation unit 130 estimates an in-region position using a position estimation model with target region feature amounts as input, and estimates the position of the target region based on the in-region position.

[0078] As shown in FIG. 5 , the estimation unit 130 includes, for example, a region identification unit 131 , a first estimation unit 132 , a second estimation unit 133 , and a conversion unit 134 .

[0079] The region specifying unit 131 specifies the region position, which indicates the position of the region u (corresponding region) corresponding to the target region feature amount.

[0080] For example, the region identifying unit 131 may identify, as the corresponding region, a region u that corresponds to the target region feature extracted by the feature extracting unit 124, based on the similarity information, and identify the position of the corresponding region. Alternatively, for example, the region identifying unit 131 may identify the corresponding region based on a specifying condition and the similarity information. In this case, since the specifying condition according to this embodiment is that the region u has the highest similarity, the region identifying unit 131 may refer to the similarity information and identify the position of the region u associated with the highest similarity as the position of the corresponding region.

[0081] The position identified by the region identifying unit 131 is represented, for example, by a position in the similarity information (i.e., a position in the xy coordinate system described with reference to FIG. 6). In this embodiment, when the corresponding region is region u[i,j], the position of the corresponding region will be described using an example where it is represented as [i,j].

[0082] The position identified by the region identifying unit 131 is not limited to the position in the similarity information, but may be expressed as, for example, a position in the estimation target image or a position in real space corresponding to the estimation target image.

[0083] The first estimation unit 132 receives the target part feature amount as input and estimates the position within the region using a position estimation model.

[0084] The target region feature amounts input to the position estimation model are the target region feature amounts extracted by the feature amount extraction unit 124. The intra-region position estimated by the first estimation unit 132 is represented, for example, by a position in the corresponding region. The position in the corresponding region is represented, for example, by a difference from a representative position of region u, which is the corresponding region.

[0085] The representative position of region u is a position determined in advance for region u, such as the center of region u or one of the four corners, such as the lower right corner. The difference is a value in the coordinate system of the similarity information. Specifically, for example, the coordinate system of the similarity information is an xy coordinate system with the upper left corner of the entire similarity map shown in FIG. 6 as its origin, and the representative position is the lower right corner. In this case, the position of region u[i,j] in the xy coordinate system can be expressed as (i,j). Furthermore, Δx and Δy, which represent the position (Δx, Δy) within the region, each have a value greater than or equal to -1 and less than 0.

[0086] When the representative position is the center of the region u, each of Δx and Δy representing the position (Δx, Δy) within the region is, for example, a value between −0.5 and +0.5. For example, when the representative position is the upper left corner, each of Δx and Δy representing the position (Δx, Δy) within the region is, for example, a value between 0 and 1.

[0087] The difference may be expressed using values ​​in a coordinate system in the estimation target image and a coordinate system in real space corresponding to the estimation target image.

[0088] The second estimation unit 133 estimates the position of the target part of the estimation target in the entire region based on the region position identified by the region identification unit 131 and the intra-region position estimated by the first estimation unit 132.

[0089] The position estimated by the second estimation unit 133 is represented, for example, by a position in the similarity information (for example, a position in the xy coordinate system described above). Note that the position estimated by the second estimation unit 133 is not limited to a position in the similarity information, and may be represented, for example, by a position in the estimation target image or a position in real space corresponding to the estimation target image.

[0090] The conversion unit 134 converts the coordinate system of the entire region to convert the position estimated by the second estimation unit 133 into the position of the target part of the estimation target in the estimation target image.

[0091] In this embodiment, as described above, the position estimated by the second estimation unit 133 is represented by a position in the similarity information. The conversion unit 134, for example, performs coordinate conversion on the position estimated by the second estimation unit 133 to obtain a position in the estimation target image that corresponds to the position estimated by the second estimation unit 133.

[0092] Note that when the second estimation unit 133 estimates a position in the estimation target image, the conversion unit 134 does not need to be included in the estimation unit 130. Furthermore, the conversion unit 134 may convert the position estimated by the second estimation unit 133 into a position in real space corresponding to the estimation target image.

[0093] Up to now, an example of the functional configuration of the information processing apparatus 100 according to this embodiment has been described. From here, an example of the physical configuration of the information processing apparatus 100 according to this embodiment will be described.

[0094] 7 is a diagram showing an example of the physical configuration of the information processing device 100 according to this embodiment. The information processing device 100 physically includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070.

[0095] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0096] The processor 1020 is a processor realized by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The memory 1030 is a main storage device realized by a RAM (Random Access Memory), etc.

[0097] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules for realizing each function of the information processing device 100. The processor 1020 reads each of these program modules into the memory 1030 and executes them to realize the function corresponding to that program module.

[0098] The network interface 1050 is an interface for connecting the information processing device 100 to a communication network. The input interface 1060 is an interface for a user to input information, and includes, for example, a touch panel, a keyboard, a mouse, etc. The output interface 1070 is an interface for presenting information to a user, and includes, for example, a liquid crystal panel, an organic EL (Electro-Luminescence) panel, etc.

[0099] The information processing device 100 may be configured from a plurality of devices having a similar physical configuration to the information processing device 100 shown in Fig. 7. In this case, the plurality of devices may be configured to be able to transmit and receive information to and from each other via a communication network or the like.

[0100] So far, an example of the physical configuration of the information processing device 100 according to this embodiment has been described. From here, an example of the operation of the information processing device 100 according to this embodiment will be described.

[0101] (Example of Operation of Information Processing Device 100) Fig. 8 is a flowchart showing a detailed example of information processing according to this embodiment. The information processing is processing for estimating the position of a target part in an estimation target.

[0102] The information processing is started, for example, when the image capturing device captures an image of the estimated target. The image capturing device may capture the image of the estimated target while the target is standing still in front of the image capturing device, or may capture the image of the estimated target while walking when the target reaches a focus point.

[0103] Furthermore, for example, the image capturing device may continuously capture images at predetermined time intervals, and in this case, the information processing may be executed repeatedly.

[0104] As shown in FIG. 8, the target image acquisition unit 101 acquires an estimation target image (step S101).

[0105] In detail, for example, the target image acquisition unit 101 acquires an estimation target image as exemplified in Fig. 3. Such an estimation target image can be acquired, for example, by placing an imaging device so as to capture images of the vicinity of both eyes of the estimation target.

[0106] Furthermore, for example, the target image acquisition unit 101 may cut out a predetermined portion, such as the face of the target, from the captured image. In this case, for example, the target image acquisition unit 101 may detect images of both eyes from the face image using pattern matching or the like, which is a predetermined method with a relatively light processing load, and cut out the detected images of both eyes from the face image.

[0107] The estimation target image is preferably a high-definition image with a relatively large number of images, for example, an image with a height of 1000 px (pixels) and a width of 4000 px.

[0108] Although not shown, the target image acquisition unit 101 may determine whether the image of both eyes, which is the estimation target image, is included in the image acquired from the image capture device in step S101 by using pattern matching or the like. If it is determined that the estimation target image is included, step S102, which will be described later, may be executed. Alternatively, if it is determined that the estimation target image is not included, the target image acquisition unit 101 may repeatedly execute step S101.

[0109] As shown in FIG. 8, the reference image acquisition unit 102 acquires a reference image (step S102).

[0110] In more detail, for example, the reference image acquisition unit 102 acquires a reference image from the storage unit 103. The reference image is, for example, a right-eye image as shown in FIG.

[0111] The position estimation unit 110 executes the above-described position estimation process (step S110). That is, in step S110, the position estimation unit 110 extracts object part features from the features of the estimation object image based on the learning results obtained by using the reference image and the object image, and estimates the position of the object part in the estimation object based on the object part features.

[0112] As shown in FIG. 8, the position estimation process (step S110) includes an extraction process (step S120) and an estimation process (step S130).

[0113] The extraction unit 120 generates similarity information and extracts target region features from the features of the estimation target image based on the similarity information (step S120).

[0114] FIG. 9 is a flowchart showing a detailed example of the extraction process (step S120) according to this embodiment.

[0115] The feature amount acquiring unit 121 acquires the feature amounts of the estimation target image acquired in step S101 as an input, using a feature amount extraction model (step S111).

[0116] In detail, for example, the feature amount acquiring unit 121 reduces the size of the estimation target image in order to reduce the processing load and speed up the processing.

[0117] 10A and 10B are diagrams showing an example of changes in data size from the estimation target image to the acquisition of feature amounts in this embodiment. Fig. 10A shows an example of the data size of the estimation target image acquired in step S101. Fig. 10B shows an example of the data size of a reduced estimation target image. Fig. 10C shows an example of the data size of the feature amounts of the estimation target image.

[0118] 10(a) and 10(b), the feature acquisition unit 121 reduces the data size of the estimation target image from 1000px x 4000px to 64px x 256px, for example. In this example, the aspect ratio remains the same before and after reduction. Note that the reduction rate in the reduction process may be changed as appropriate, or the reduction process may not be performed at all.

[0119] The feature acquisition unit 121 acquires features of the reduced estimation target image using a feature extraction model as input. The features are, for example, CNN features, and are three-dimensional data including the dimension of the number of channels. In this embodiment, the data size of the features is 128×8×32 as shown in FIG. 10( c), when expressed as Csrch×Hsrch×Wsrch, where Csrch is the number of channels, Hsrch is the height, and Wsrch is the width (horizontal).

[0120] The data size of the feature amount (Csrch×Hsrch×Wsrch) may be changed as appropriate. Also, the aspect ratio of the feature amount (e.g., Hsrch / Wsrch) is the same as that of the estimation target image in the above example, but may be different from that of the estimation target image.

[0121] The reference feature amount acquiring unit 122 acquires reference features using the feature amount extraction model with the reference image acquired in step S102 as an input (step S112).

[0122] The feature extraction model used in step S112 may be the same as the feature extraction model used in step S111. The reference feature is, like the feature of the image to be estimated, three-dimensional data including, for example, the dimension of the number of channels. In this embodiment, the data size of the reference feature is 128×3×11 when expressed as Cref×Href×Wref, where Cref is the number of channels, Href is the height, and Wref is the width (horizontal). In other words, the number of channels of the reference feature, Cref, is the same as the number of channels of the feature, Csrch.

[0123] Although an example in which the vertical and horizontal dimensions of the reference feature are 3×11 will be described, the vertical and horizontal dimensions of the reference feature may be changed as appropriate. Furthermore, the number of channels of the feature and the reference feature may differ.

[0124] Furthermore, if the storage unit 103 stores the reference feature amounts, the reference feature amount acquisition unit 122 may acquire the reference feature amounts from the storage unit 103 in step S112, and step S102 may not be executed.

[0125] Referring again to Fig. 9, the generation unit 123 generates similarity information based on the feature amount of the estimation target image acquired in step S111 and the reference feature amount acquired in step S112 (step S113).

[0126] (Method for Generating Similarity Information) An overview of a method for generating similarity information (similarity map) will be described with reference to FIG. 11 . FIG. 11 is a diagram conceptually illustrating an example of a method for generating similarity information according to this embodiment. FIG. 11(a) is a diagram illustrating an example of the relationship between the entire feature amount Rsrch and parts of the image to be estimated. FIG. 11(b) is a diagram illustrating an example of the data size of the reference feature amount Rref. FIG. 11(c) is a diagram illustrating an example of the relationship between the entire region and region u of the similarity map.

[0127] The generation unit 123 extracts a portion Rsrch[i, j] of the same data size as the reference feature Rref shown in Fig. 11(b) from the entire feature Rsrch of the image to be estimated, where i is an integer from 1 to 32 and j is an integer from 1 to 8, as described above.

[0128] The portions Rsrch[i, j] are different from one another within the entire Rsrch of the feature amount of the image to be estimated.

[0129] In detail, for example, the generation unit 123 extracts the partial Rsrch[i, j] so that each of the components (i, j, 1:128) of the matrix representing the entire Rsrch matches the central component of the reference feature Rref when viewed from the channel direction.

[0130] In this embodiment, the partial Rsrch[i, j] has the same data size as the reference feature Rref, and is therefore composed of components (i-5:i+5, j-1:j+1, 1:128) among the components that make up the entire Rsrch. Also, in this embodiment, the central component of the reference feature Rref when viewed from the channel direction is the component (2, 6, 1:128).

[0131] Here, ":" represents a range, and when p and q (q>p) are arbitrary integers, "p:q" represents integers from p to q. For example, the component (2, 6, 1:128) represents the component at (2, 6) when viewing the entire Rsrch from the channel direction, that is, the entire component (2, 6, r) when r is an integer from 1 to 128.

[0132] As a result of this extraction process, the widthwise adjacent portions Rsrch[i,j] and Rsrch[i+1,j] are shifted by one component horizontally in the overall Rsrch, and the vertically adjacent portions Rsrch[i,j] and Rsrch[i,j+1] are shifted by one component vertically in the overall Rsrch.

[0133] In this extraction process, when some of the components constituting the partial Rsrch[i, j] are not included in the entire Rsrch, such as when i = 1 and j = 1, their values ​​may be values ​​interpolated from outside the entire Rsrch. This interpolation process may use various padding processes such as zero padding, reflection padding, and replication padding.

[0134] In addition, the method by which the whole Rsrch extracts the part Rsrch[i, j] may be changed as appropriate. For example, the amount of deviation between adjacent parts in the width direction and the vertical direction is not limited to one component, but may be determined as appropriate in each direction.

[0135] The generation unit 123 calculates the spatial cosine similarity between each of the extracted portions Rsrch[i,j] and the reference feature Rref, thereby obtaining the similarity of the region u[i,j] from the entire region of the similarity map shown in FIG.

[0136] The cosine similarity is calculated using, for example, formula (1). In formula (1), u represents the position [i, j]. That is, in formula (1), Map[u] and cos θ[u] represent the similarity of the region u[i, j]. Rsrch[u] represents the feature amount of the portion Rsrch[i, j]. In this embodiment, the cosine similarity calculated using formula (1) is expressed using a one-dimensional vector including, for example, 4224 (= 128 × 3 × 11) components.

[0137] For example, the generation unit 123 may calculate the cosine similarity for the portion Rsrch[i, j] each time the portion Rsrch[i, j] is extracted. Alternatively, for example, the generation unit 123 may calculate the cosine similarity for each portion Rsrch[i, j] after extracting all of the portions Rsrch[i, j].

[0138] 12 is a flowchart showing an example of the similarity generation process (step S113) according to this embodiment. The flowchart shown in Fig. 12 shows an example of calculating the cosine similarity for the portion Rsrch[i,j] each time the portion Rsrch[i,j] is extracted.

[0139] The generation unit 123 repeatedly executes steps S113b to S113d until j is between 1 and 8 (step S113a; loop A). ​​The generation unit 123 repeatedly executes steps S113c to S113d until i is between 1 and 32 (step S113b; loop B).

[0140] The generator 123 performs interpolation around the entire Rsrch as necessary to extract a portion Rsrch[i, j] having the same data size as the reference feature Rref from the entire Rsrch (step S113c). The generator 123 calculates the spatial cosine similarity between the portion Rsrch[i, j] extracted in step S113c and the reference feature Rref, for example, using the above-mentioned formula (1) (step S113d).

[0141] The generating unit 123 generates similarity information (step S113d).

[0142] As can be seen from the procedures of steps S113a to S113d, the region u[i,j] and its respective similarity correspond one-to-one to the portion Rsrch[i,j] of the feature amount of the estimation target image. In step S113d, the generation unit 123 generates similarity information in which the region u[i,j] and its respective similarity are further associated with the feature amount of the portion Rsrch[i,j], for example.

[0143] The method for generating similarity information described here is one example. It is sufficient to be able to calculate the similarity between each of the portions Rsrch[i,j] (portions having the same data size as the reference feature Rref) extracted from the entire feature Rsrch of the estimation target image according to a predetermined vertical and horizontal slide amount and the reference feature Rref. The specific method for doing so may be changed as appropriate. Furthermore, the similarity included in the similarity information is not limited to the similarity between feature amounts, but may also be similarity based on pixel values ​​between the portion of the estimation target image and the reference image.

[0144] Referring again to Fig. 9, the feature amount extraction unit 124 extracts target region feature amounts from the feature amounts of the estimation target image based on the similarity information generated in step S113 (step S114).

[0145] In more detail, for example, the feature extraction unit 124 extracts target region feature amounts from the overall Rsrch of feature amounts of the estimation target image based on specific conditions and similarity information. If the specific condition specifies that the region has the greatest similarity, the feature extraction unit 124 identifies the maximum value of similarity (maximum similarity) from the similarity information. Then, the feature extraction unit 124 extracts, from the overall Rsrch of feature amounts of the estimation target image, the portion corresponding to the identified maximum similarity as the target region feature amount.

[0146] As described above, the similarity included in the similarity information corresponds one-to-one to the portion Rsrch[i,j] of the feature of the estimation target image. In this embodiment, the feature extraction unit 124 extracts the portion Rsrch[i,j] corresponding to the maximum similarity as the target region feature.

[0147] In the example of similarity information (similarity map) shown in Figure 6, the darkest (black) region u[9,5] has the highest similarity. In this case, the feature extraction unit 124 extracts the portion Rsrch[9,5] of the feature of the estimation target image corresponding to this maximum similarity as the target region feature. The portion Rsrch[9,5] is composed of components (4:14, 4:6, 1:128) out of the components that make up the entire Rsrch.

[0148] Note that, in this embodiment, an example will be described in which one of the partial Rsrch[i, j] is extracted as the target region feature, but the target region feature may be any portion of the entire Rsrch of the feature of the estimation target image that corresponds to the highest similarity. For example, the portion of the target region feature may have a predetermined data size different from the partial Rsrch[i, j] according to this embodiment. In this case, the target region feature may be, for example, any portion that shares a common center with the partial Rsrch[i, j] that corresponds to the highest similarity, and may be composed of components (i-s:i+s, j+t:j-t, 1:128) that make up the entire Rsrch. Here, s and t represent any integers equal to or greater than 0.

[0149] After executing step S114, the feature extraction unit 124 returns to the position estimation process (step S110).

[0150] Referring again to Fig. 8, the estimation unit 130 estimates the position within the region using a position estimation model with the target region feature amount extracted in step S114 as input, and estimates the position of the target region based on the position within the region (step S130).

[0151] FIG. 13 is a flowchart showing a detailed example of the estimation process (step S130) according to this embodiment.

[0152] The region specifying unit 131 specifies the region position (step S131).

[0153] In detail, for example, the region identifying unit 131 identifies the region u[i, j] corresponding to the target region feature amount extracted in step S114 as the corresponding region based on the similarity information generated in step S113. Then, the region identifying unit 131 identifies the position of the identified corresponding region.

[0154] As in the example above, when the target region feature amount is the portion Rsrch[9,5], the region identification unit 131 identifies the representative position of the region u[9,5] corresponding to the target region feature amount (i.e., for example, the ninth position horizontally and the fifth position vertically). When the representative position is the lower right corner of the region u[i,j], the region identification unit 131 determines the position (9,5) in the xy coordinate system as the representative position of the region u[9,5].

[0155] The first estimation unit 132 estimates an intra-region position using a position estimation model with the target region feature amount extracted in step S114 as input (step S132).

[0156] More specifically, for example, if the target region feature amount is partial Rsrch[9,5], this partial Rsrch[9,5] is composed of components (4:14, 4:6, 1:128) among the components that make up the overall Rsrch. Therefore, the first estimation unit 132 uses the components (4:14, 4:6, 1:128) among the components that make up the overall Rsrch as the input f. Then, for example, the first estimation unit 132 uses the components (4:14, 4:6, 1:128) as the input f and a linear regression model W to determine the difference (Δx, Δy) (= Wf). This difference is the difference from the representative position within the corresponding region, region u[9,5].

[0157] The linear regression model W may be configured to receive target region feature amounts of a predetermined data size as input and output a difference (Δx, Δy).

[0158] The second estimation unit 133 estimates the position of the target part of the estimation target in the entire region based on the region position identified in step S131 and the intra-region position estimated in step S132 (step S133).

[0159] In detail, for example, the second estimation unit 133 adds the corresponding x and y components of the difference (Δx, Δy) to the x and y components of the region position identified in step S131. In this way, the second estimation unit 133 obtains an estimate of the position of the target region to be estimated in the entire region. As in the above example, if the representative position is set to the lower right corner of region u and position (9, 5) is identified in step S131, the second estimation unit 133 adds the difference (Δx, Δy) to this position to obtain the estimate (9 + Δx, 5 + Δy).

[0160] The conversion unit 134 converts the coordinate system (xy coordinate system) of the entire region to convert the position estimated in step S132 into the position of the target part of the estimation target in the estimation target image (step S134).

[0161] 10 and 11 , in this embodiment, the data size of the estimation target image for the entire region is reduced by 8 / 1000 times vertically and 32 / 4000 times horizontally. Accordingly, the conversion unit 134 multiplies each component of the position estimated in step S132 by the reciprocal of the vertical and horizontal reduction rates. That is, the conversion unit 134 multiplies the x component of the estimated position by 4000 / 32 and the y component of the estimated position by 1000 / 8. In this way, the conversion unit 134 determines the position of the target part of the estimation target in the estimation target image.

[0162] After executing step S134, the conversion unit 134 returns to the position estimation process (step S110) and ends the information processing as shown in Fig. 8. By executing the information processing, the position of the target part can be determined with higher accuracy than the region position.

[0163] So far, the first embodiment of this disclosure has been described.

[0164] (Actions and Effects) According to this embodiment, the information processing device 100 includes a target image acquisition unit 101 and a position estimation unit 110. The target image acquisition unit 101 acquires an estimation target image including an image of a target part in an estimation target. The position estimation unit 110 extracts target part feature amounts related to a part corresponding to the target part from the feature amounts of the estimation target image based on a learning result obtained using a reference image including an image of the target part and a target image including an image of the target part, and estimates the position of the target part based on the target part feature amounts.

[0165] In this way, since the position of the target part is estimated based on the target part feature amount, the position of the target part can be determined with higher accuracy than the region position, and therefore the position of the target part can be estimated with high accuracy.

[0166] According to this embodiment, the position estimation unit 110 generates similarity information indicating the similarity between the features of the image to be estimated and reference features, which are features of a reference image, and includes an extraction unit 120 that extracts target part features from the features of the image to be estimated based on the similarity information.

[0167] In this way, the target region feature is extracted based on similarity information indicating the similarity between the feature of the estimation target image and the reference feature. The similarity between the feature of the estimation target image and the reference feature can be extracted quickly with a relatively light processing load. Therefore, the position of the target region can be estimated quickly and accurately.

[0168] The extraction unit 120 includes a generation unit 123 and a feature extraction unit 124. The generation unit 123 generates similarity information based on the feature amounts of the estimation target image and the reference feature amounts. The feature extraction unit 124 extracts target portion feature amounts from the feature amounts of the estimation target image based on the similarity information.

[0169] In this way, the target region feature is extracted based on similarity information indicating the similarity between the feature of the estimation target image and the reference feature. The similarity between the feature of the estimation target image and the reference feature can be extracted quickly with a relatively light processing load. Therefore, the position of the target region can be estimated quickly and accurately.

[0170] According to this embodiment, the learning result includes a trained feature extraction model that has undergone machine learning to extract feature quantities of the image to be estimated. The extraction unit 120 further includes a feature acquisition unit 121 that acquires feature quantities of the image to be estimated using the feature extraction model with the image to be estimated as input.

[0171] In this way, the feature extraction model, which is the result of learning, is used to acquire the feature quantities of the estimation target image. The process of acquiring the feature quantities of the estimation target image using the feature extraction model has a relatively light processing load and can be performed quickly. Therefore, it is possible to estimate the position of the target region quickly and accurately.

[0172] According to this embodiment, the learning result includes a trained position estimation model that has undergone machine learning to estimate an intra-region position that indicates the position of the target region in a corresponding region that corresponds to the target region feature amount. The position estimation unit 110 includes an estimation unit 130 that estimates the intra-region position using the position estimation model with the target region feature amount as input, and estimates the position of the target region based on the intra-region position.

[0173] In this way, the position within the region is estimated using the position estimation model that is the result of learning, and the position of the target part is estimated based on the position within the region. The process of estimating the position within the region using the position estimation model with the target part feature amount as input has a relatively light processing load and can be performed at high speed.

[0174] Furthermore, the intra-region position indicates the position of the target part in the corresponding region corresponding to the target part feature amount, i.e., the position inside the corresponding region. Therefore, the position of the target part can be estimated more accurately than when the position of the corresponding region is used as the estimation result of the position of the target part.

[0175] Therefore, it becomes possible to estimate the position of the target area quickly and accurately.

[0176] According to this embodiment, the estimation unit 130 includes a region identification unit 131, a first estimation unit 132, and a second estimation unit 133. The region identification unit 131 identifies a region position indicating the position of the corresponding region. The first estimation unit 132 estimates an intra-region position using a position estimation model with the target region feature amount as input. The second estimation unit 133 estimates the position of the target region based on the region position and the intra-region position.

[0177] In this way, the position of the target region is estimated based on the region position and the intra-region position. This process has a relatively light processing load and can be performed quickly. Therefore, the position of the target region can be estimated quickly and accurately.

[0178] Furthermore, by estimating the position of the target part based on the region position and the intra-region position, the position of the target part in the entire region can be obtained. This makes it easy to grasp the position of the target part in the entire region. Therefore, it is possible to improve convenience.

[0179] According to this embodiment, the estimation unit 130 further includes a conversion unit 134 that converts the estimated position of the target region into a position in the estimation target image.

[0180] This allows the estimated position of the target region to be reflected in the estimation target image, thereby improving convenience.

[0181] According to this embodiment, the target area includes at least one of the center of the pupil, the outer corner of the eye, and the inner corner of the eye.

[0182] This makes it possible to estimate the positions of various target areas with high accuracy.

[0183] According to this embodiment, the estimation target is a person. The target part is the center of the pupil. The reference image is a single-eye image including only a predetermined eye. The target image is a binocular image including both eyes. The similarity is spatial cosine similarity.

[0184] Processing using these can be easily performed using equation (1) or the like. Therefore, it becomes possible to estimate the position of the target area easily and accurately. In particular, it becomes possible to estimate the position of the pupil center easily and accurately. The pupil center is useful for extracting an iris image used in iris authentication, etc., and it becomes possible to improve the accuracy of iris authentication.

[0185] (Variation 1) Based on the position of a target part (e.g., the center of the pupil) of the estimation target in the estimation target image, the conversion unit 134 may cut out an image of a region of interest within a predetermined range based on the target part. The region of interest may be, for example, the iris. The conversion unit 134 may output the image of the region of interest via a network to, for example, an authentication device that performs iris authentication. This increases the likelihood of accurately acquiring an image including the iris, thereby enabling accurate authentication.

[0186] (Variation 2) The target part may be, for example, one or more of the eyes, nose, mouth, etc. In this case, the reference image, target image, and estimation target image may be facial images. The estimation target image may include a background, etc. This variation also makes it possible to accurately estimate the position of the target part in the estimation target, as in the first embodiment.

[0187] For example, as described in Modification Example 1, the conversion unit 134 may cut out a facial image from the estimated target image, with the face as the region of interest, based on the position of the estimated target part. Then, the conversion unit 134 may output the image of the region of interest to, for example, an authentication device that performs facial color authentication via a network. This increases the likelihood that a facial image including the face can be accurately acquired, thereby enabling accurate authentication.

[0188] In the first embodiment, an example has been described in which the target region is the center of the pupil. However, a plurality of target regions may be predetermined. Such target regions may be, for example, a plurality of regions selected from the center of the pupil (pupil center), the outer corner of the eye, the inner corner of the eye, the tip of the nose, the center of the nostrils, and the corners of the mouth.

[0189] In this embodiment, for the sake of simplicity, differences from the first embodiment will be mainly described, and overlapping descriptions will be omitted as appropriate.

[0190] In this embodiment, the target areas are the center of the pupil, the outer corner of the eye, and the inner corner of the eye, all of which are included in the eyes, which are a common facial feature.

[0191] Therefore, the reference image according to this embodiment may be the same as that of the first embodiment, for example, a right-eye image. The estimation target image and the target image according to this embodiment may also be the same as that of the first embodiment, for example, a both-eye image.

[0192] For example, if the target region includes at least one of the tip of the nose and the center of the nostrils, the reference image, the estimated target image, and the target image each include a nose image. For example, if the target region is the corner of the mouth, the reference image, the estimated target image, and the target image each include a mouth image. That is, the reference image, the estimated target image, and the target image may be prepared for each part (e.g., face part) that includes the target region.

[0193] The information processing device 100 according to this embodiment may have substantially the same functions as those of embodiment 1. That is, the position estimation unit 110 according to this embodiment may extract target region features from features of the estimation target image based on a learning result obtained by learning using a reference image and a target image, as in embodiment 1, and estimate the position of each target region in the estimation target based on the target region features.

[0194] However, in this embodiment, the outer and inner corners of the eyes are added to the target regions of embodiment 1. Therefore, in this embodiment, the variables (e.g., matrices) handled by each of the extraction unit 120 and the estimation unit 130 are expanded so as to add components corresponding to the added target regions.

[0195] As a result, the position of each target part in the estimation object can be estimated using the extraction unit 120 and the estimation unit 130 that have the same functions as in embodiment 1. Furthermore, by performing the same processes as in embodiment 1 (steps S101, S102, and S110), the position of each target part in the estimation object can be estimated.

[0196] More specifically, for example, FIG. 14 is a conceptual diagram showing an example of the data size and data structure of the reference feature Rref according to this embodiment. The reference feature Rref includes portions (components) corresponding to each target part. The reference feature acquisition unit 122 may acquire the reference feature Rref as shown in FIG. 14 using a feature extraction model with the basic image as input. Note that the data size and data structure of the reference feature Rref may be changed as appropriate.

[0197] The feature acquisition unit 121 may acquire feature amounts of the estimation target image including each portion of the feature amount of the estimation target image, for example, by using a feature extraction model with the estimation target image as input. Each portion of the feature amount of the estimation target image may also have the same data size and data configuration as the reference feature amount Rref. That is, for example, each of Rsrch[i, j] includes a portion (component) corresponding to each target region (in this embodiment, the center of the pupil, the inner corner of the eye, and the outer corner of the eye).

[0198] The generating unit 123 may generate similarity information including a matrix composed of components for each target region. This similarity information includes a similarity map for each target region.

[0199] The feature amount extraction unit 124 may extract target region feature amounts for each target region from the feature amounts of the estimation target image based on the similarity information and in accordance with specific conditions.

[0200] For example, if the specific condition in this embodiment is the same as in embodiment 1, the specific condition is the region with the highest similarity for each target region. In this case, the feature extraction unit 124 may identify the region corresponding to the highest similarity for each target region based on the similarity information, and extract target region feature values ​​corresponding to each of the regions identified for each target region from the feature values ​​of the estimation target image.

[0201] The feature amount extraction section 124 may then integrate the extracted target region feature amounts for each target region to extract a target region feature amount (matrix) corresponding to the entire set of target regions.

[0202] The region identifying unit 131 may identify the region position for each target region based on the similarity information and the target region feature amount, or based on the identifying conditions and the similarity information, as in embodiment 1. The output from the region identifying unit 131 may be, for example, a value (matrix) including the region position for each target region as a component.

[0203] The first estimation unit 132 may estimate the position within the region for each target region using a position estimation model with the target region feature amounts as input. Here, the position estimation model may be configured to, for example, receive target region feature amounts corresponding to all of the multiple target regions as input and output a value (matrix) whose components include the position within the region for each target region.

[0204] As in the first embodiment, the second estimation unit 133 estimates the position of each target part of the estimation target in the entire region based on the region position identified by the region identification unit 131 and the intra-region position estimated by the first estimation unit 132. The output from the second estimation unit 133 may be, for example, a value (matrix) including the position of each target part as a component.

[0205] As in the first embodiment, the conversion unit 134 may convert the positions estimated by the second estimating unit 133 to obtain the positions of each target part.

[0206] The information processing device 100 according to this embodiment may be physically configured in the same manner as in the first embodiment.

[0207] So far, the second embodiment of this disclosure has been described.

[0208] (Operations and Effects) According to this embodiment, the plurality of target regions further includes the outer corner and inner corner of the eye.

[0209] This makes it possible to estimate the positions of the outer and inner corners of the eyes with high accuracy.

[0210] Third Embodiment In the second embodiment, an example was described in which the variables handled by each of the extraction unit 120 and the estimation unit 130 are expanded in order to estimate the position of each of a plurality of target parts. However, the method for estimating the position of each of a plurality of target parts is not limited to this. For example, the information processing device may be provided with a position estimation unit 110 for each target part. Also, for example, the information processing device may be provided with a position estimation unit 110 for each part (e.g., facial parts) that includes the target part. In this embodiment, an example will be described in which the information processing device is provided with a position estimation unit 110 for each target part.

[0211] In this embodiment, for the sake of simplicity, differences from the second embodiment will be mainly described, and overlapping descriptions will be omitted as appropriate. That is, the target areas in this embodiment are the center of the pupil and the outer and inner corners of the eye, as in the second embodiment.

[0212] 15 is a diagram showing an example of the functional configuration of an information processing device 300 according to this embodiment. The information processing device 300 includes a target image acquisition unit 101, a reference image acquisition unit 102, and a storage unit 103, similar to those in the first embodiment, and multiple position estimation units 110a to 110c corresponding to the number of target regions.

[0213] Each of the position estimation units 110a to 110c may be similar to that in embodiment 1 and has a function corresponding to each of the target parts. For example, the position estimation unit 110a has a function for estimating the position of the pupil center of the estimation target, similar to embodiment 1. The position estimation units 110b to 110c have a function for estimating the positions of the outer corner and inner corner of the eye of the estimation target, respectively.

[0214] Each of the position estimation units 110a to 110c extracts target part features from the features of the estimation target image based on the learning results obtained using a reference image and a target image, and estimates the position of the target part according to its function based on the target part features.

[0215] Each of the position estimation units 110 may use an estimation target image and a reference image according to the corresponding function. In addition, each of the feature extraction model and the position estimation model may be prepared for each target body part and stored in the storage unit 103, for example.

[0216] So far, an example of the functional configuration of the information processing device 300 according to this embodiment has been described. The information processing device 300 according to this embodiment may be physically configured in the same manner as the information processing device 100 according to embodiment 1. From here, an example of the operation of the information processing device 300 according to this embodiment will be described.

[0217] (Example of operation of information processing device 300) Fig. 16 is a flowchart showing an example of information processing according to this embodiment. The information processing according to this embodiment includes steps S101 and S102 similar to the information processing according to embodiment 1. Furthermore, the information processing according to this embodiment includes a position estimation process (steps S110a to S110c) instead of the position estimation process (step S110) according to embodiment 1.

[0218] The position estimation process (steps S110a to S110c) is a process for estimating the position of each of a plurality of target parts. That is, the position estimation unit 110a executes the position estimation process (step S110a). The position estimation unit 110b executes the position estimation process (step S110b). The position estimation unit 110c executes the position estimation process (step S110c).

[0219] Each of the position estimation processes (steps S110a to S110c) may be substantially similar to the position estimation process (step S110) according to the first embodiment.

[0220] That is, the position estimation process (step S110a) may be the same as the position estimation process (step S110) according to embodiment 1. The position estimation process (step S110b) corresponds to the process in which the pupil center in the position estimation process (step S110) according to embodiment 1 is replaced with the outer corner of the eye. The position estimation process (step S110c) corresponds to the process in which the pupil center in the position estimation process (step S110) according to embodiment 1 is replaced with the inner corner of the eye.

[0221] So far, the third embodiment of this disclosure has been described.

[0222] (Actions and Effects) This embodiment provides the same effects as those of embodiment 2. (Variation 3) The multiple target regions may be, for example, the pupil center of the right eye and the pupil center of the left eye. In this case, the reference image may include a right-eye image and a left-eye image. According to this variation, the information processing device 300 according to embodiment 2 or 3 can be used to accurately estimate the positions of the pupil centers of the right and left eyes of the estimation target.

[0223] Fourth Embodiment In the second and third embodiments, examples of estimating the positions of the pupil center and other parts of the eye have been described. These may be used to estimate the gaze direction of an estimation target. In this embodiment, for the sake of simplicity, differences from the third embodiment will be mainly described, and overlapping descriptions will be omitted as appropriate.

[0224] 17 is a diagram showing an example of the functional configuration of an information processing device 400 according to this embodiment. The information processing device 400 includes a target image acquisition unit 101, a reference image acquisition unit 102, a storage unit 103, and multiple position estimation units 110a to 110c, similar to those in embodiment 2. In addition to these, the information processing device 400 includes a line-of-sight acquisition unit 401 and an alert unit 402.

[0225] The gaze acquisition unit 401 estimates the gaze direction of the person being the estimation target based on the positions of the pupil center, outer corner and inner corner of the eye estimated by the position estimation units 110a to 110c, respectively.

[0226] The position of the predetermined part used to estimate the gaze direction is not limited to the positions of the pupil center, the outer corner, and the inner corner of the eye, but may be the positions of multiple target parts. Such multiple target parts are, for example, parts included in the eyes, and may be one or more of the pupil center, the outer corner, the inner corner, the upper eyelid (upper end), and the lower eyelid (upper end).

[0227] The alert unit 402 outputs an alert when the gaze direction of the estimation target estimated by the gaze acquisition unit 401 satisfies a predetermined alert condition.

[0228] So far, an example of the functional configuration of the information processing device 400 according to this embodiment has been described. The information processing device 400 according to this embodiment may be physically configured in the same manner as the information processing device 100 according to embodiment 1. From here, an example of the operation of the information processing device 400 according to this embodiment will be described.

[0229] (Example of operation of information processing device 400) Fig. 18 is a flowchart showing an example of gaze acquisition processing according to this embodiment. The gaze acquisition processing is processing for estimating the gaze direction of an estimation target. The gaze acquisition processing may be executed, for example, following the information processing according to embodiment 3. That is, the gaze acquisition processing is started when, for example, the positions of the pupil center, outer corner and inner corner of the eye of the estimation target are estimated as a result of executing the information processing.

[0230] The gaze acquisition unit 401 acquires the positions of the pupil center, outer corner and inner corner of the eye of the target estimated by executing information processing (step S401).

[0231] In detail, for example, the gaze acquisition unit 401 acquires the positions of the pupil center, outer corner and inner corner of the eye as estimation targets from each of the plurality of position estimation units 110a to 110b.

[0232] The gaze acquisition unit 401 estimates the gaze direction of the person to be estimated based on the positions of the pupil center, the outer corner and the inner corner of the eye acquired in step S401 (step S402).

[0233] In detail, for example, the gaze acquisition unit 401 estimates the gaze direction of the estimation target using a gaze estimation model with the positions of the pupil center, outer corner, and inner corner of the eye of the estimation target as input. The gaze direction is, for example, a direction in the estimation target image or a direction in real space corresponding to the estimation target image.

[0234] The gaze estimation model is a model for estimating the gaze direction of an estimation target, and is a trained model that has undergone machine learning using the positions of the pupil center and the outer and inner corners of the eyes as input. Input data to the learning model during training includes the positions of the pupil center and the outer and inner corners of the eyes. In the machine learning, for example, supervised learning may be performed using ground truth data that includes gaze directions corresponding to the positions of the pupil center and the outer and inner corners of the eyes included in the input data.

[0235] The alert unit 402 determines whether a predetermined alert condition is met (step S403). Examples of the alert condition will be described later.

[0236] If it is determined that the alert condition is not satisfied (step S403; No), the alert unit 402 ends the gaze acquisition process. If it is determined that the alert condition is satisfied (step S403; Yes), the alert unit 402 outputs an alert (step S404) and ends the gaze acquisition process.

[0237] (Example 1 of Alert Conditions and Alerts) When the information processing device 400 is installed in an authentication device for performing biometric authentication of an object to be authenticated (for example, iris authentication or face authentication), the gaze acquisition unit 401 may estimate the gaze direction of a person to be authenticated as the object to be authenticated. The authentication device is equipped with a camera for capturing an image of the person to be authenticated.

[0238] In this case, the alert condition may be that the gaze direction of the person to be authenticated is facing in a direction different from the direction of the camera lens, and the alert unit 402 may output an alert when the gaze direction of the estimated target is facing in a direction different from the direction of the camera lens.

[0239] The alert is, for example, information that prompts the authentication target to face the camera. The alert may be output as a sound, or as characters or figures displayed on the display unit of the authentication device, or may be lit by a lamp associated with the camera.

[0240] This increases the likelihood of obtaining a captured image suitable for biometric authentication of an object to be authenticated, thereby improving the accuracy of authentication.

[0241] (Example 2 of Alert Condition and Alert) When the information processing device 400 is mounted on a camera, the gaze acquisition unit 401 may estimate the gaze direction of a person who is a subject, as an estimation target. In this case, the alert condition may be that the gaze direction of the subject is facing in a direction different from that of the camera lens. The alert unit 402 may output an alert when the estimated gaze direction of the estimation target is facing in a direction different from that of the camera lens.

[0242] The alert is, for example, a mark superimposed on the captured image displayed on the display unit of the camera. The mark is, for example, a frame displayed in a predetermined color such as red or black or in a predetermined manner (for example, flashing or constantly displayed), and is positioned so as to enclose a subject whose line of sight is facing in a direction different from the camera lens, thereby enabling the subject to be identified.

[0243] This allows the photographer, when taking a group photo or the like, to easily identify subjects whose line of sight is pointing in a direction different from the camera lens, and the photographer can then encourage the subjects to turn their gaze toward the camera.

[0244] (Example 3 of Alert Conditions and Alerts) The information processing device 400 may estimate the gaze direction of a passerby or the like as an estimated target based on an image captured by a surveillance camera. In this case, the alert condition may be that the estimated target is looking at a security guard, or that the estimated target is looking at a security guard at a predetermined frequency or more. The alert unit 402 may output an alert when an estimated target that satisfies the alert condition is detected. The output destination of the alert unit 402 may be, for example, a terminal device (e.g., a smartphone, a tablet terminal, etc.) carried by the security guard, or a device provided at a security center.

[0245] This allows for the early detection of suspicious individuals, which allows for quick action to be taken against them, thereby improving safety.

[0246] So far, the fourth embodiment of this disclosure has been described.

[0247] According to the present embodiment, there are a plurality of target parts, and the information processing device 400 further includes a gaze acquisition unit 401 that estimates the gaze direction of the person who is the estimation target, based on the positions of each of the plurality of target parts.

[0248] Since the positions of multiple target parts can be estimated with high accuracy, the gaze direction can be estimated. The gaze direction can be used for various purposes, which improves convenience.

[0249] According to this embodiment, the information processing device 400 further includes an alert unit 402 that outputs an alert regarding the estimation target based on the gaze direction.

[0250] By outputting an alert, information based on the gaze direction of the target can be communicated to the target and other people. As a result, as described above, it becomes possible to obtain a captured image suitable for authentication. Therefore, it is possible to improve convenience.

[0251] Fifth Embodiment In this embodiment, an example will be described in which an information processing device has a learning function for the learning models (feature extraction model and position estimation model) described in Embodiment 1. Note that the learning function may be provided in a device other than the information processing device, and the learning results may be used in the information processing device 100.

[0252] In this embodiment, for the sake of simplicity, differences from the first embodiment will be mainly described, and overlapping descriptions will be omitted as appropriate.

[0253] 19 is a diagram showing an example of the functional configuration of an information processing device 500 according to this embodiment. The information processing device 500 includes the target image acquisition unit 101, reference image acquisition unit 102, storage unit 103, and position estimation unit 110, similar to those in embodiment 1. In addition to these, the information processing device 500 includes a similarity loss acquisition unit 501, a position loss acquisition unit 502, and a correction unit 510.

[0254] The similarity loss obtaining unit 501 obtains the similarity loss based on the similarity information and the similarity loss function L.

[0255] The similarity loss function L is a function for calculating the similarity loss, which is a loss related to similarity information.

[0256] The position loss acquisition unit 502 obtains the position loss based on the position loss function M.

[0257] The position loss function M is a function for calculating the position loss. The position loss is a loss related to the position within a region estimated using a position estimation model with the target portion feature amount among the feature amounts of the target image as input.

[0258] The correction unit 510 corrects the feature extraction model and the position estimation model based on the similarity loss and the position loss. Note that the correction unit 510 may correct only one of the feature extraction model and the position estimation model. As a result of the correction or improvement performed by the correction unit 510, at least one of the feature extraction model and the position estimation model is corrected. Therefore, the correction unit 510 may also be referred to as an improvement unit, an optimization unit, or the like.

[0259] In detail, for example, as shown in FIG. 19 , the correction unit 510 includes a loss integration unit 511, a feature quantity extraction model correction unit 512, and a position estimation model correction unit 513.

[0260] The loss integrating unit 511 integrates the similarity loss obtained by the similarity loss obtaining unit 501 and the position loss obtained by the position loss obtaining unit 502 to obtain an integrated loss.

[0261] The feature extraction model correction unit 512 corrects the feature extraction model based on the integrated loss calculated by the loss integration unit 511.

[0262] The location estimation model correction unit 513 corrects the location estimation model based on the integrated loss calculated by the loss integration unit 511 .

[0263] So far, an example of the functional configuration of the information processing device 500 according to this embodiment has been described. The information processing device 500 according to this embodiment may be physically configured in the same manner as the information processing device 100 according to embodiment 1. From here, an example of the operation of the information processing device 500 according to this embodiment will be described.

[0264] 20 and 21 are flowcharts showing an example of the learning process according to this embodiment. The learning process is a process for machine learning of a feature extraction model and a position estimation model.

[0265] The learning process is started, for example, when a start instruction is received from a user. Thereafter, the learning process may be repeatedly executed until a termination condition is met. The termination condition may be, for example, the number of repetitions, or the similarity loss, position loss, and integrated loss being equal to or less than a predetermined threshold.

[0266] The information processing apparatus 500 executes steps S501 to S506, which are processes corresponding to steps S101 to S102, S111 to S114, and S132 according to the first embodiment.

[0267] Specifically, the target image acquisition unit 101 acquires a target image (step S501).

[0268] That is, in this embodiment, the target image acquisition unit 101 acquires a target image instead of the estimated target image in S101 according to embodiment 1. As described in embodiment 1, the target image is, for example, a binocular image.

[0269] The reference image acquisition unit 102 acquires a reference image (step S502) in the same manner as in step S102 according to the first embodiment. The reference image acquired here may be the same as the reference image acquired in step S102 according to the first embodiment.

[0270] The feature acquisition unit 121 receives the target image acquired in step S501 as an input and acquires the feature of the target image using a feature extraction model (step S503).

[0271] That is, in this embodiment, the feature amount acquiring unit 121 acquires the feature amount of the target image instead of the feature amount of the estimation target image in step S111 according to the first embodiment.

[0272] More specifically, in step S503, the feature acquisition unit 121 acquires features of the target image based on an input different from that in step S111 according to embodiment 1. The feature extraction model used in step S503 is also different from step S111 according to embodiment 1 in that it is a model under training. Except for these points, the processing in step S503 may be similar to the processing in step S111 according to embodiment 1.

[0273] The reference feature amount acquiring unit 122 receives the reference image acquired in step S502 as an input and acquires reference features using the feature amount extraction model in the same manner as in step S112 according to the first embodiment (step S504).

[0274] More specifically, the feature extraction model used in step S504 is a model under training, which is a difference from step S112 according to embodiment 1. Except for this, the processing in step S504 may be the same as the processing in step S112 according to embodiment 1. That is, for example, the reference image used for input in step S502 may be the same as the reference image used for input in step S112 according to embodiment 1.

[0275] The generating unit 123 generates similarity information based on the feature amounts of the target image and the reference feature amounts acquired in steps S503 and S504, respectively, in the same manner as in step S113 according to the first embodiment (step S505).

[0276] More specifically, in step S505, the generation unit 123 generates similarity information based on an input different from that in step S113 according to embodiment 1. That is, except for the different input, the processing in step S505 may be the same as the processing in step S113 according to embodiment 1.

[0277] The feature amount extraction unit 124 extracts target region feature amounts from the feature amounts of the target image based on the similarity information generated in step S505, similarly to step S114 according to the first embodiment (step S506).

[0278] More specifically, in step S506, the feature extraction unit 124 extracts target region features based on an input different from that in step S114 according to embodiment 1. That is, except for the different input, the processing in step S506 may be similar to the processing in step S114 according to embodiment 1.

[0279] The first estimation unit 132 estimates the intra-region position using the position estimation model with the target portion feature amount extracted in step S506 as input (step S507).

[0280] More specifically, in step S507, the first estimation unit 132 estimates the intra-area position based on an input different from that in step S132 according to embodiment 1. The position estimation model used in step S507 is also different from step S132 according to embodiment 1 in that it is a model under training. Except for these points, the processing in step S507 may be similar to the processing in step S132 according to embodiment 1.

[0281] See Fig. 21. The similarity loss obtaining unit 501 obtains the similarity loss based on the similarity information and the similarity loss function L (step S508).

[0282] In detail, for example, the similarity loss function L is a function that finds the difference between the correct answer map input as correct answer data and the similarity included in the similarity information generated in step S505.

[0283] The correct answer map may be, for example, a two-dimensional map in which only the eye positions are labeled with "1" and the rest are labeled with "0".

[0284] The similarity loss function L may be, for example, a function that randomizes the similarity included in the similarity information by multiplying it by a softmax function in the map direction, and calculates the cross entropy with the correct map y. The similarity loss function L may also use CosFace or the like to emphasize the similarity. Specifically, for example, as shown in Equation (2), correction hyperparameters s and m may be applied when calculating the BCE (binary cross entropy error) with the correct map y. Equation (2) is an example of the similarity loss function L.

[0285] The position loss acquisition unit 502 calculates the position loss based on the position estimated in step S507 and the position loss function M (step S509).

[0286] In detail, for example, the position loss function M may be an absolute error or a squared error as shown in formula (3). Formula (3) is an example of the position loss function M. In formula (3), (X, Y) represents the position estimated in step S507, and (Xg, Yg) represents the correct position.

[0287] The loss integration unit 511 integrates the similarity loss calculated in step S508 and the position loss calculated in step S509 to calculate an integrated loss S (step S510).

[0288] In detail, for example, the loss integration unit 511 calculates the integrated loss S (= the value obtained using the similarity loss function L + the value obtained using the position loss function M) by adding the similarity loss and the position loss together.

[0289] The feature extraction model correction unit 512 corrects the feature extraction model based on the integrated loss S calculated in step S510 (step S511).

[0290] In detail, for example, the feature extraction model correction unit 512 corrects the feature extraction model by updating the parameters included in the feature extraction model so that the integrated loss S becomes smaller.

[0291] The location estimation model correction unit 513 corrects the location estimation model based on the integrated loss S calculated in step S510 (step S512), and ends the learning process.

[0292] In detail, for example, the position estimation model correction unit 513 corrects the position estimation model by updating the parameters included in the position estimation model so that the integrated loss S becomes smaller.

[0293] By performing such a learning process, the parameters included in the feature extraction model and the position estimation model can be corrected.

[0294] So far, the fifth embodiment of this disclosure has been described.

[0295] (Actions and Effects) According to this embodiment, the information processing device 500 further includes a position loss acquisition unit 502 and a correction unit 510. The position loss acquisition unit 502 calculates a position loss, which is a loss related to an intra-area position estimated using a position estimation model with a target portion feature amount among the feature amounts of a target image as input, based on a position loss function M. The correction unit 510 corrects the position estimation model based on the position loss.

[0296] This improves the accuracy of estimation using the position estimation model, thereby enabling the position of the target area to be estimated with even greater accuracy.

[0297] According to this embodiment, the learning result includes a trained feature extraction model that has undergone machine learning to extract features of the image to be estimated. The information processing device 500 further includes a similarity loss acquisition unit 501 that calculates a similarity loss, which is a loss related to the similarity information, based on the similarity information and a similarity loss function. The correction unit 510 further corrects the feature extraction model based on the similarity loss.

[0298] This makes it possible to estimate the position of the target area by acquiring more appropriate features using the feature extraction model, thereby enabling the position of the target area to be estimated with even greater accuracy.

[0299] According to this embodiment, the correction unit 510 includes a loss integrating unit 511, a feature extraction model correction unit 512, and a location estimation model correction unit 513. The loss integrating unit 511 calculates an integrated loss by integrating the location loss and the similarity loss. The feature extraction model correction unit 512 corrects the location estimation model based on the integrated loss. The location estimation model correction unit 513 corrects the feature extraction model based on the integrated loss.

[0300] This makes it possible to acquire more appropriate features using the feature extraction model and improve the estimation accuracy using the position estimation model, thereby making it possible to estimate the position of the target area with even greater accuracy.

[0301] Furthermore, the integrated loss can be used to modify both the feature extraction model and the localization model overall to improve estimation accuracy, thereby enabling the location of the target area to be estimated with even greater accuracy.

[0302] Sixth Embodiment In the position loss function M according to the fifth embodiment, even when the target part feature extracted by the position estimation unit 110 corresponds to an area that does not include the target part (for example, the pupil center), there is a risk that the target part feature will be included in the position loss.

[0303] The position loss function M according to this embodiment is a function that does not include, in the position loss, a loss related to a position estimated using a position estimation model with a target part feature amount as input, when the target part feature amount input to the position estimation model is different from the part corresponding to the target part.

[0304] Equation (4) is an example of the position loss function M according to this embodiment. The position loss function M according to this embodiment is obtained by adding a mask α to the position loss function M according to embodiment 5. α is 1 if |X - Xg| < 2 is satisfied, and is 0 otherwise.

[0305] So far, the sixth embodiment of this disclosure has been described.

[0306] (Actions and Effects) According to this embodiment, the position loss function M is a function that, when the target part feature amount input to the position estimation model is different from the part corresponding to the target part, does not include, in the position loss, a loss related to the position estimated using the position estimation model with the target part feature amount as input.

[0307] This reduces the possibility that, when a target part feature that is different from the part corresponding to the target part is extracted, the loss related to the estimation result based on the target part feature is included in the position loss. In other words, it is possible to at least reduce learning based on target part feature that corresponds to an area that is significantly displaced from the target part or its vicinity. Therefore, it becomes possible to estimate the position of the target part with even greater accuracy.

[0308] Seventh Embodiment In the fifth embodiment, an example in which the similarity loss and the position loss are integrated has been described, but the similarity loss and the position loss do not have to be integrated.

[0309] In this embodiment, the corrector 510 does not need to include the loss consolidator 511 .

[0310] The feature extraction model correction unit 512 may correct the feature extraction model based on the similarity loss calculated by the similarity loss acquisition unit 501. In detail, for example, the feature extraction model correction unit 512 may correct the feature extraction model by updating parameters included in the feature extraction model so as to reduce the similarity loss.

[0311] The position estimation model correction unit 513 may correct the position estimation model based on the position loss calculated by the position loss acquisition unit 502. In detail, for example, the position estimation model correction unit 513 may correct the position estimation model by updating parameters included in the position estimation model so as to reduce the position loss.

[0312] Except for these, the information processing device according to this embodiment may be configured similarly to the information processing device 500 according to embodiment 5. The information processing device according to this embodiment may be physically configured similarly to the information processing device 100 according to embodiment 1.

[0313] In the learning process according to this embodiment, step S510 does not need to be executed.

[0314] In step S511, the feature extraction model correction unit 512 may correct the feature extraction model based on the similarity loss calculated in step S508. In step S512, the location estimation model correction unit 513 may correct the location estimation model based on the location loss calculated in step S509.

[0315] Except for these, the learning process according to this embodiment may be similar to the learning process according to the fifth embodiment.

[0316] So far, the seventh embodiment of this disclosure has been described.

[0317] (Operations and Effects) According to this embodiment, it is also possible to correct both the feature extraction model and the position estimation model, thereby enabling the position of the target region to be estimated with high accuracy.

[0318] Eighth Embodiment In this embodiment, an example in which the estimated position of the target region is displayed on a display unit will be described. In this embodiment, for the sake of simplicity, differences from the fourth embodiment will be mainly described, and overlapping descriptions will be omitted as appropriate.

[0319] 22 is a diagram showing an example of the functional configuration of an information processing device 800 according to this embodiment. The information processing device 800 includes the same components as those in embodiment 4: a target image acquisition unit 101, a reference image acquisition unit 102, a storage unit 103, multiple position estimation units 110a to 110c, a line-of-sight acquisition unit 401, and an alert unit 402. In addition to these components, the information processing device 800 also includes a display unit 801 and a display control unit 802.

[0320] The display unit 801 displays various types of information. The display unit 801 is configured, for example, with a liquid crystal panel, an organic EL, etc. The display control unit 802 controls the display unit 801 and causes the display unit 801 to display various types of information.

[0321] In detail, for example, the display control unit 802 may superimpose the region position identified by the region identification unit 131 and the position of the target part of the estimation target estimated by the position estimation unit 110 on the estimation target image and display them on the display unit 801. At this time, the display control unit 802 may correct the region position identified by the region identification unit 131 to its position in the estimation target image.

[0322] Furthermore, for example, the display control unit 802 may cause the display unit 801 to display the position of the target part in the estimation object in a more emphasized manner than the corresponding region.

[0323] Furthermore, for example, the display control unit 802 may cause the display unit 801 to display the gaze direction of the estimation target estimated by the gaze acquisition unit 401 superimposed on the estimation target image.

[0324] So far, an example of the functional configuration of the information processing device 800 according to this embodiment has been described. The information processing device 800 according to this embodiment may be physically configured in the same manner as the information processing device 100 according to embodiment 1. From here, an example of the operation of the information processing device 800 according to this embodiment will be described.

[0325] (Example of operation of information processing device 800) Fig. 23 is a flowchart showing an example of display processing according to this embodiment. The display processing is processing for displaying the estimated result on the display unit 801. The display processing may be executed, for example, following the gaze acquisition processing according to embodiment 4. That is, the display processing is started, for example, when a gaze direction is estimated as a result of executing the gaze acquisition processing. Note that if the gaze direction is not to be displayed, the display processing may be executed following the information processing.

[0326] The display control unit 802 creates a display image (step S801).

[0327] 24 is a diagram showing an example of a display image according to this embodiment. In the example shown in the drawing, the display image includes the region position identified by the region identifying unit 131 and the position of the target part of the estimation target estimated by the position estimating unit 110 superimposed on the estimation target image.

[0328] In the example shown in the figure, the region position is indicated by a rectangle (a black square in FIG. 24 ) according to the shape of the estimated corresponding region. The region position may be indicated in an appropriate color such as red or blue. The region position may also be indicated in a light color, which makes it easier to compare with the image of the estimation target included in the estimation target image.

[0329] In the example shown in the figure, the position of the target area is indicated by an X. However, the method for indicating the position of the target area is not limited to this, and it is preferable to indicate it using an appropriate mark or the like.

[0330] In the example shown in the figure, the position of the target part is highlighted by being outlined in white, but the emphasis can be achieved by using a color of appropriate intensity, a mark, or the like.

[0331] In the example shown in the figure, the display image further includes an arrow indicating the gaze direction estimated by the gaze acquisition unit 401 superimposed on the estimation target image.

[0332] 23 again, the display control unit 802 causes the display unit 801 to display the display image created in step S801 (step S802), and ends the display process. This allows the corresponding region, the estimated position of the target part, and the gaze direction to be superimposed on the estimation target image and displayed on the display unit 801.

[0333] So far, the eighth embodiment of this disclosure has been described.

[0334] (Operations and Effects) According to this embodiment, the information processing device 800 further includes a display control unit 802 that causes the display unit 801 to display the position of the target part in the estimation target and the corresponding region superimposed on the estimation target image.

[0335] This allows the positions estimated by different methods (i.e., the positions of the target parts in the estimation target and the corresponding areas) to be overlaid on the estimation target image for comparison, allowing the user to easily grasp the accuracy of the estimation, thereby improving convenience.

[0336] (Operations and Effects) According to this embodiment, the display control unit 802 displays the position of the target portion in the estimation object on the display unit 801 in a more emphasized manner than the corresponding region.

[0337] This allows the user to easily visually grasp the position that is estimated with a higher degree of accuracy than the corresponding region, thereby improving convenience.

[0338] Ninth Embodiment In this embodiment, a configuration example of an information processing system including an information processing device will be described. The information processing device may be any of the other embodiments, but in this embodiment, the information processing device 100 according to the first embodiment will be described as an example.

[0339] 25 is a diagram showing an example of the configuration of an information processing system 900 according to this embodiment. The information processing system 900 includes an image capturing device 901 and the same information processing device 100 as in the first embodiment.

[0340] The image capturing device 901 captures an image of an estimation target. The image capturing device 901 and the information processing device 100 are connected via a communication network N that may be wired, wireless, or a combination of these. This allows the image capturing device 901 and the information processing device 100 to transmit and receive information to and from each other.

[0341] For example, the information processing device 100 acquires an image of the estimation target from the imaging device 901 via the communication network N.

[0342] So far, the ninth embodiment of this disclosure has been described.

[0343] (Operations and Effects) According to this embodiment, the same effects as those of the first embodiment are achieved.

[0344] (Variation 4) The information processing system 900 may include a plurality of image capturing devices 901. The plurality of image capturing devices 901 may include, for example, a first image capturing device for capturing an image of the entire body of the estimation target, and a second image capturing device for capturing an image of the estimation target (image of both eyes).

[0345] Furthermore, for example, there may be a plurality of second image capturing devices for capturing the estimation target image. In this case, the second image capturing devices may be installed at different heights. A control device controlling the plurality of image capturing devices 901 may estimate the height of the eyes of the estimation target based on the whole-body image captured by the first image capturing device. Then, the control device may control the plurality of second image capturing devices so as to capture the eyes of the estimation target using the second image capturing devices installed at positions corresponding to the estimated eye heights.

[0346] According to this embodiment, the same effects as those of the first embodiment are achieved.

[0347] In the above, in this disclosure, the embodiments and modifications have been described with reference to the drawings, but these are merely examples, and various configurations other than those described above can also be adopted.

[0348] In addition, although the flowcharts used in the above description describe multiple steps (processes) in a sequential order, the execution order of the steps in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed to the extent that the content is not affected. Furthermore, the above-described embodiments and variations can be combined to the extent that the content is not contradictory.

[0349] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0350] 1. An information processing device comprising: a target image acquisition means for acquiring an estimation target image including an image of a target region of an estimation target; and a position estimation means for extracting target region features related to a portion corresponding to the target region from features of the estimation target image based on a learning result obtained using a reference image including an image of the target region and a target image including the image of the target region, and estimating the position of the target region based on the target region features. 2. The information processing device described in 1., wherein the position estimation means includes extraction means for generating similarity information indicating the similarity between the features of the estimation target image and reference features which are features of the reference image, and extracting the target region features from the features of the estimation target image based on the similarity information. 3. The information processing device described in 2., wherein the extraction means includes generation means for generating the similarity information based on the features of the estimation target image and the reference features, and feature extraction means for extracting the target region features from the features of the estimation target image based on the similarity information. 4. The information processing device described in any one of 2. to 4., wherein the learning result includes a trained feature extraction model that has been subjected to machine learning to extract feature amounts of the estimation target image, and the extraction means further includes feature acquisition means that acquires feature amounts of the estimation target image using the feature extraction model with the estimation target image as input. 5. The information processing device described in any one of 2. to 4., wherein the learning result includes a trained position estimation model that has been subjected to machine learning to estimate an intra-region position that indicates the position of the target region in a corresponding region that corresponds to the target region feature, and the position estimation means includes estimation means that estimates the intra-region position using the position estimation model with the target region feature as input, and estimates the position of the target region based on the intra-region position. 6. The information processing device described in 5., wherein the estimation means includes area identification means that identifies an area position that indicates the position of the corresponding region, first estimation means that estimates the intra-region position using the position estimation model with the target region feature as input, and second estimation means that estimates the position of the target region based on the area position and the intra-region position.7. The information processing device described in 6., wherein the estimation means further includes conversion means for converting the estimated position of the target region into a position in the estimation target image. 8. The information processing device described in any one of 5. to 7., further comprising: position loss acquisition means for determining, based on a position loss function, a position loss that is a loss related to the position within the region estimated using the position estimation model with the target region feature amount among the feature amounts of the target image as input; and correction means for correcting the position estimation model based on the position loss. 9. The information processing device described in 8., wherein, when the target region feature amount input to the position estimation model differs from the part corresponding to the target region, the position loss is a function that does not include, in the position loss, a loss related to the position estimated using the position estimation model with the target region feature amount as input. 10. The information processing device according to any one of items 8. to 9., wherein the learning result includes a trained feature extraction model that has been subjected to machine learning to extract features of the estimation target image, and further comprises similarity loss acquisition means that calculates a similarity loss that is a loss related to the similarity information based on the similarity information and a similarity loss function, and the correction means further corrects the feature extraction model based on the similarity loss. 11. The information processing device according to item 10., wherein the correction means includes: loss integration means that calculates an integrated loss that integrates the position loss and the similarity loss; feature extraction model correction means that corrects the feature extraction model based on the integrated loss; and position estimation model correction means that corrects the position estimation model based on the integrated loss. 12. The information processing device according to any one of items 5. to 11., further comprises display control means that causes display means to display the position of the target portion in the estimation target and the corresponding region superimposed on the estimation target image. 13. The display control means causes the display means to display the position of the target portion in the estimation target with more emphasis than the corresponding region. The information processing device described in14. The information processing device described in any one of 2. to 13., wherein the estimation target is a person, the target region is a pupil center, the reference image is a one-eye image including only a predetermined eye, the target image is a two-eye image including both eyes, and the similarity is spatial cosine similarity. 15. The information processing device described in 14., wherein there are multiple target regions, and further comprising gaze acquisition means for estimating a gaze direction of the person who is the estimation target based on the positions of each of the multiple target regions. 16. The information processing device described in 15., wherein the multiple target regions further include an outer corner and an inner corner of the eye. 17. The information processing device described in 15. or 16., further comprising alert means for outputting an alert regarding the estimation target based on the gaze direction. 18. The information processing device described in any one of 1. to 17., wherein the target region includes at least one of the pupil center, outer corner and inner corner of the eye. 19. 1. to 18. 20. An information processing system comprising: an information processing device according to any one of the above items; and an imaging device for imaging the estimation target. 20. An information processing method in which one or more computers acquire an estimation target image including an image of a target region of the estimation target, extract target region features related to a portion corresponding to the target region from features of the estimation target image based on a learning result obtained using a reference image including an image of the target region and a target image including an image of the target region, and estimate a position of the target region based on the target region features. 21. The information processing method according to item 20., in which estimating the position of the target region includes generating similarity information indicating a similarity between features of the estimation target image and reference features that are features of the reference image, and extracting the target region features from the features of the estimation target image based on the similarity information. 22. The information processing method according to item 21, in which extracting the target region features includes generating the similarity information based on the features of the estimation target image and the reference features, and extracting the target region features from the features of the estimation target image based on the similarity information.23. The information processing method according to 22., wherein the learning result includes a trained feature extraction model that has been subjected to machine learning to extract feature amounts of the estimation target image, and extracting the target region feature amounts further includes acquiring feature amounts of the estimation target image using the feature extraction model with the estimation target image as input. 24. The information processing method according to any one of 21. to 23., wherein the learning result includes a trained position estimation model that has been subjected to machine learning to estimate an intra-region position that indicates the position of the target region in a corresponding region that corresponds to the target region feature amount, and estimating the position of the target region includes estimating the intra-region position using the position estimation model with the target region feature amount as input, and estimating the position of the target region based on the intra-region position. 25. The information processing method according to 24, wherein estimating the position of the target region based on the intra-region position includes identifying an intra-region position that indicates the position of the corresponding region, estimating the intra-region position using the position estimation model with the target region feature amount as input, and estimating the position of the target region based on the intra-region position. 26. The information processing method according to 25., wherein estimating the position of the target part based on the intra-region position further includes converting the estimated position of the target part into a position in the estimation target image. 27. The information processing method according to any one of 24. to 26., further including determining, based on a position loss function, a position loss that is a loss related to the intra-region position estimated using the position estimation model with the target part feature amount among the feature amounts of the target image as input, and correcting the position estimation model based on the position loss. 28. The information processing method according to 27., wherein, when the target part feature amount input to the position estimation model differs from the part corresponding to the target part, the position loss is not included in the position loss.29. The information processing method according to 27. or 28., wherein the learning result includes a trained feature extraction model that has been subjected to machine learning to extract features of the estimation target image, and further includes calculating a similarity loss that is a loss related to the similarity information based on the similarity information and a similarity loss function, and correcting the position estimation model further includes correcting the feature extraction model based on the similarity loss. 30. The information processing method according to 29., wherein correcting the position estimation model includes calculating an integrated loss that integrates the position loss and the similarity loss, correcting the feature extraction model based on the integrated loss, and correcting the position estimation model based on the integrated loss. 31. The information processing method according to any one of 24. to 30., further including displaying the position of the target portion in the estimation target and the corresponding region on a display means such that they are superimposed on the estimation target image. 32. The information processing method according to 31., wherein displaying on a display means includes displaying on the display means the position of the target portion in the estimation target with greater emphasis than the corresponding region. 33. The information processing method described in any one of 21. to 32., wherein the estimation target is a person, the target region is a pupil center, the reference image is a one-eye image including only a predetermined eye, the target image is a binocular image including both eyes, and the similarity is spatial cosine similarity. 34. The information processing method described in 33., wherein there are multiple target regions, and further comprising estimating a gaze direction of the person who is the estimation target based on the positions of each of the multiple target regions. 35. The information processing method described in 34., wherein the multiple target regions further include an outer corner and an inner corner of the eye. 36. The information processing method described in 34. or 35., further comprising outputting an alert regarding the estimation target based on the gaze direction. 37. The information processing method described in any one of 20. to 36., wherein the target region includes at least one of the pupil center, outer corner, and inner corner of the eye.38. A program causing one or more computers to execute the following steps: acquiring an estimation target image including an image of a target region in an estimation target; extracting target region features related to a portion corresponding to the target region from features of the estimation target image based on a learning result obtained using a reference image including an image of the target region and a target image including an image of the target region; and estimating the position of the target region based on the target region features. 39. The program described in 38., in which estimating the position of the target region includes generating similarity information indicating the similarity between the features of the estimation target image and reference features that are features of the reference image, and extracting the target region features from the features of the estimation target image based on the similarity information. 40. The program described in 39., in which extracting the target region features includes generating the similarity information based on the features of the estimation target image and the reference features, and extracting the target region features from the features of the estimation target image based on the similarity information. 41. The program described in 40., wherein the learning result includes a trained feature extraction model that has been subjected to machine learning for extracting feature amounts of the estimation target image, and extracting the target region feature amounts further includes acquiring feature amounts of the estimation target image using the feature extraction model with the estimation target image as input. 42. The program described in any one of 39. to 41., wherein the learning result includes a trained position estimation model that has been subjected to machine learning for estimating an intra-region position indicating the position of the target region in a corresponding region corresponding to the target region feature amounts, and estimating the position of the target region includes estimating the intra-region position using the position estimation model with the target region feature amounts as input, and estimating the position of the target region based on the intra-region position. 43. The program described in 42, wherein estimating the position of the target region based on the intra-region position includes identifying an intra-region position indicating the position of the corresponding region, estimating the intra-region position using the position estimation model with the target region feature amounts as input, and estimating the position of the target region based on the intra-region position.44. The program described in 43., wherein estimating the position of the target part based on the intra-region position further includes converting the estimated position of the target part to a position in the estimation target image. 45. The program described in any one of 42. to 44., further including determining, based on a position loss function, a position loss that is a loss related to the intra-region position estimated using the position estimation model with the target part feature amount among the feature amounts of the target image as input, and correcting the position estimation model based on the position loss. 46. The program described in 45., wherein, when the target part feature amount input to the position estimation model differs from the part corresponding to the target part, the position loss is not included in the position loss. 47. The program according to 45. or 46., wherein the learning result includes a trained feature extraction model obtained by machine learning to extract features of the estimation target image, and further includes calculating a similarity loss that is a loss related to the similarity information based on the similarity information and a similarity loss function, and correcting the position estimation model further includes correcting the feature extraction model based on the similarity loss. 48. The program according to 47., wherein correcting the position estimation model includes calculating an integrated loss that integrates the position loss and the similarity loss, correcting the feature extraction model based on the integrated loss, and correcting the position estimation model based on the integrated loss. 49. The program according to any one of 42. to 48., further including displaying the position of the target portion in the estimation target and the corresponding region on a display means so as to be superimposed on the estimation target image. 50. The program according to 49., wherein displaying on a display means includes displaying the position of the target portion in the estimation target on the display means with the position of the target portion more emphasized than the corresponding region.51. The program described in any one of 39. to 50., wherein the estimation target is a person, the target region is a pupil center, the reference image is a single-eye image including only a predetermined eye, the target image is a binocular image including both eyes, and the similarity is spatial cosine similarity. 52. The program described in 51., wherein there are multiple target regions, and the program further includes estimating a gaze direction of the person who is the estimation target based on the positions of each of the multiple target regions. 53. The program described in 52., wherein the multiple target regions further include an outer corner and an inner corner of the eye. 54. The program described in 52. or 53., further includes outputting an alert regarding the estimation target based on the gaze direction. 55. The program described in any one of 38. to 54., wherein the target region includes at least one of the pupil center, outer corner and inner corner of the eye. 56. A recording medium having recorded thereon a program for causing one or more computers to acquire an estimation target image including an image of a target portion of an estimation target, extract target portion features for a portion corresponding to the target portion from features of the estimation target image based on a learning result obtained using a reference image including an image of the target portion and a target image including the image of the target portion, and estimate the position of the target portion based on the target portion features. 57. A recording medium having recorded thereon a program described in any one of 39. to 55.

[0351] This application claims priority based on Japanese Patent Application No. 2022-103353, filed on June 28, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0352] 100, 300, 400, 500, 800 Information processing device 101 Target image acquisition unit 102 Reference image acquisition unit 103 Storage unit 110, 110a to 110c Position estimation unit 120 Extraction unit 121 Feature acquisition unit 122 Reference feature acquisition unit 123 Generation unit 124 Feature extraction unit 130 Estimation unit 131 Area identification unit 132 First estimation unit 133 Second estimation unit 134 Conversion unit 401 Gaze acquisition unit 402 Alert unit 501 Similarity loss acquisition unit 502 Position loss acquisition unit 510 Correction unit 511 Loss integration unit 512 Feature extraction model correction unit 513 Position estimation model correction unit 801 Display unit 802 Display control unit 900 Information processing system 901 Imaging device

Claims

1. An object image acquisition means for acquiring an estimated object image including an image of a target part in an estimated object; a position estimation means for extracting a target portion feature amount for a portion corresponding to the target portion from the feature amount of the estimation target image based on a learning result obtained by using a reference image including an image of the target portion and a target image including the image of the target portion, and estimating a position of the target portion based on the target portion feature amount. Information processing device.

2. The position estimation means includes an extraction means for generating similarity information indicating a similarity between a feature amount of the estimation target image and a reference feature amount which is a feature amount of the reference image, and extracting the target portion feature amount from the feature amount of the estimation target image based on the similarity information. The information processing device according to claim 1 .

3. The extraction means includes: a generating means for generating the similarity information based on a feature amount of the estimation target image and the reference feature amount; and a feature extraction means for extracting the target portion feature from the feature of the estimation target image based on the similarity information. The information processing device according to claim 2 .

4. the learning result includes a trained feature extraction model that has been subjected to machine learning for extracting features of the estimation target image, The extraction means includes: The method further includes a feature acquisition means for acquiring a feature of the estimation target image by using the feature extraction model as an input of the estimation target image. The information processing device according to claim 3 .

5. the learning result includes a trained position estimation model that has been subjected to machine learning for estimating an intra-region position indicating a position of the target region in a corresponding region that corresponds to the target region feature amount, The position estimation means includes an estimation means for estimating a position within the region using the position estimation model with the target part feature amount as an input, and estimating a position of the target part based on the position within the region. The information processing device according to claim 2 .

6. The estimation means includes: an area specifying means for specifying an area position indicating the position of the corresponding area; a first estimation means for estimating a position within the region by using the position estimation model with the target portion feature amount as an input; and a second estimation means for estimating a position of the target portion based on the region position and the intra-region position. The information processing device according to claim 5 .

7. The estimation means includes: The method further includes converting the position of the estimated target portion into a position in the estimated target image. The information processing device according to claim 6.

8. a position loss acquisition means for acquiring a position loss, which is a loss related to a position within the region estimated using the position estimation model, by inputting the target portion feature amount among the feature amounts of the target image based on a position loss function; and a correction means for correcting the position estimation model based on the position loss. The information processing device according to claim 5 .

9. One or more computers Acquire an estimation target image including an image of a target part in the estimation target; Based on a learning result obtained by using a reference image including an image of a target part and a target image including an image of the target part, a target part feature amount for a portion corresponding to the target part is extracted from the feature amount of the estimation target image, and a position of the target part is estimated based on the target part feature amount. Information processing methods.

10. On one or more computers, Acquire an estimation target image including an image of a target part in the estimation target; A program for extracting target part features for a portion corresponding to the target part from features of the estimated target image based on a learning result obtained by using a reference image including an image of a target part and a target image including an image of the target part, and estimating a position of the target part based on the target part features.

Images