Program, information processing apparatus, and information processing method

Abstract

To execute estimation processing of a target included in an image with higher accuracy.SOLUTION: A program causes a computer to execute a feature extraction step, a calculation step, and an estimation step. The feature extraction step extracts, from an input image, a first feature map representing a distribution of a plurality of first feature amounts that is a feature for each of a plurality of first positions corresponding to a plurality of positions of the input image. The calculation step calculates, for each of a plurality of second feature amounts that is a feature of a plurality of targets preliminarily registered, a first similarity map representing a distribution of similarities between the second feature amounts, and the plurality of first feature amounts included in the first feature map. The estimation step estimates the targets included in the input image using information based on the plurality of first similarity maps calculated for the plurality of second feature amounts.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present embodiment relates to a program, an information processing device, and an information processing method. [Background technology]

[0002] Multimodal learning, which integrates and learns from different types of information (modals), such as text data, audio data, and image data (including still and video data), is advancing. For example, CLIP (Contrastive Language-Image Pre-training) is a cross-modal model of images and text that can search for highly relevant images simply by inputting text. CLIP is used as a foundational model because it can associate objects that do not exist in the training data by utilizing the rich expressive power of the language model.

[0003] One proposed technology that utilizes CLIP is to extend the task of object detection by adding a network model for object detection after CLIP and classifying object regions in an image using CLIP features. However, technology that classifies regions in an image may mistakenly recognize an object that reacts strongly within the region (e.g., has a higher similarity) as a different object. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Alec Radford,et al.,“Learning Transferable Visual Models From Natural Language Supervision”,ICML2021, 8748-8763. [Non-patent document 2] Weicheng Kuo,et al.,“F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models”, in arXiv:2209.15639v2, 23 Feb 2023. [Non-patent document 3] Rohit Girdhar, et al., "Imagebind: One embedding space to bind them all." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023. [Non-patent document 4] Kaiming He, et al., "Mask r-cnn." Proceedings of the IEEE international conference on computer vision. 2017. Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide a program, an information processing device, and an information processing method that can execute an estimation process of an object included in an image with higher accuracy. [Means for solving the problem]

[0006] A program according to an embodiment causes a computer to execute a feature extraction step, a calculation step, and an estimation step. The feature extraction step extracts, from an input image, a first feature map representing a distribution of a plurality of first feature amounts, which are features for a plurality of first positions corresponding to a plurality of positions in the input image. The calculation step calculates, for each of a plurality of second feature amounts, which are features of a plurality of pre-registered objects, a first similarity map representing a distribution of similarities between the second feature amount and a plurality of first feature amounts included in the first feature map. The estimation step estimates an object included in the input image using information based on the plurality of first similarity maps calculated for the plurality of second feature amounts. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram of an information processing apparatus according to a first embodiment. [Figure 2] 4 is a flowchart of an estimation process according to the first embodiment. [Figure 3] FIG. 10 is a diagram showing an example of a display screen for outputting an estimation result. [Figure 4] 10 is a flowchart of a registration process according to the first embodiment. [Figure 5] FIG. 10 is a block diagram of an information processing apparatus according to a second embodiment. [Figure 6] 10 is a flowchart of an estimation process according to the second embodiment. [Figure 7] FIG. 2 is a hardware configuration diagram of an information processing apparatus according to the first or second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of an information processing apparatus according to the present invention will be described in detail below with reference to the accompanying drawings.

[0009] In the following, an embodiment will be described in which the present invention is applied to object detection (an example of estimation) for detecting a target object (an example of a target) captured in an input image (input image) from the input image. The technology to which the present invention can be applied is not limited to object detection.

[0010] (First embodiment) The information processing device of the first embodiment estimates an object in an input image using a similarity map that represents the distribution of similarities between feature amounts for each of multiple positions (e.g., pixels) in the input image and feature amounts of registered target objects. This improves the accuracy of object detection compared to the above-mentioned technology that classifies regions using a single feature amount (e.g., CLIP feature amount).

[0011] 1 is a block diagram showing an example of the configuration of an information processing device 100 according to the first embodiment. As shown in FIG. 1, the information processing device 100 includes an acquisition unit 101, a feature extraction unit 102, a calculation unit 103, a selection unit 104, an estimation unit 105, an output control unit 106, a registration unit 110, a storage unit 121, and a display unit 122.

[0012] The storage unit 121 stores various information used in the information processing device 100. For example, the storage unit 121 stores data representing a plurality of feature amounts FB (second feature amounts) that are features of a plurality of pre-registered target objects, and that are used for comparison with feature amounts FA (first feature amounts) at each position (pixel, etc.) of the input image. The data representing the feature amounts FB may be data created in advance by an external device or the like, or may be data created by the registration unit 110.

[0013] The feature FB may be extracted from any type (modal) of data, for example, text data, audio data, still image data, and video data representing the target object. That is, the feature FB represents the feature of any of the above types of data representing the target object.

[0014] The storage unit 121 can be configured with any commonly used storage medium, such as a flash memory, a memory card, a RAM (Random Access Memory), a HDD (Hard Disk Drive), an optical disk, etc. Some or all of the data stored in the storage unit 121 may be stored in physically different storage media, or may be stored in different storage areas of the same physically identical storage medium.

[0015] The display unit 122 is a display device such as a liquid crystal display for displaying various types of information.

[0016] The acquisition unit 101 acquires various types of information used in the information processing device 100. For example, the acquisition unit 101 acquires an input image used for object detection. Any method for acquiring information may be used, but methods such as reading information stored in a storage medium and receiving information from an external device via a network can be applied.

[0017] The acquisition unit 101 may also acquire registration data used in the registration process by the registration unit 110. The registration data is data representing a target object whose features are to be registered in advance in the storage unit 121 for comparison with the features of the input image. The registration data may be any type (modal) of data, such as text data, audio data, still image data, and moving image data.

[0018] The feature extraction unit 102 extracts, from the input image, a feature map FMA (first feature map) that represents the distribution of multiple feature quantities FA for multiple positions P1 (first positions) corresponding to multiple positions in the input image. The positions in the input image are specified by, for example, a horizontal coordinate (X coordinate) and a vertical coordinate (Y coordinate). Similarly, the position P1 in the feature map FMA is specified by a horizontal coordinate and a vertical coordinate. The multiple positions in the input image and the multiple positions P1 in the feature map FMA may correspond one-to-one, many-to-one, or one-to-many.

[0019] Any method may be used to extract the feature map FMA from the input image, but for example, a method using a pre-trained extraction model MA may be applied. The extraction model MA is a model that is trained to input an input image and output a feature map FMA. The extraction model MA may use a model that corresponds to the image encoder of a multimodal deep learning model that binds data from different modalities into a common feature space, such as CLIP (e.g., Non-Patent Document 1) and Imagebind (e.g., Non-Patent Document 3).

[0020] The feature amount FA at each position P1 of the feature map FMA and the feature amount FB stored in the storage unit 121 are in a format that allows calculation of the mutual similarity. For example, the feature amount FA and the feature amount FB are expressed in the format of vectors (feature vectors) with the same number of elements (dimensions).

[0021] Therefore, similar to the feature extraction unit 102, the feature FB may be extracted using a model equivalent to an encoder (image encoder, text encoder, etc.) in a multimodal technique linked to a common feature space.

[0022] The calculation unit 103 calculates a similarity map SMA (first similarity map) using the feature amounts FB and the feature map FMA stored in the storage unit 121. For example, the calculation unit 103 calculates, for each of the plurality of feature amounts FB, a similarity map SMA that represents a distribution of similarities between the feature amount FB and the plurality of feature amounts FA included in the feature map FMA. The number of similarity maps SMA calculated is the same as the number of the plurality of feature amounts FB.

[0023] The similarity may be any index that indicates the degree of similarity between multiple feature vectors. For example, the similarity is an index that indicates the distance between the feature vectors FB and FA. The distance is, for example, the cosine similarity between the feature vectors.

[0024] The selection unit 104 selects one or more similarity maps SMA from the plurality of similarity maps SMA corresponding to the plurality of feature amounts FB, and outputs a similarity map SMB (second similarity map) based on the selected similarity map SMA.

[0025] For example, the selection unit 104 selects one or more similarity maps SMA from the multiple similarity maps SMA based on the statistical value of the similarity for each position. The selected one or more similarity maps SMA are, for example, the top N (N is an integer equal to or greater than 1) similarity maps SMA sorted in descending order of statistical value. The statistical value is, for example, the average, median, or maximum value of the similarity at each of the multiple positions.

[0026] When N=1, the selection unit 104 selects, for example, one similarity map SMA with the largest statistical value and outputs the selected similarity map SMA as the similarity map SMB. When N is 2 or more, the selection unit 104 may create and output the similarity map SMB by integrating the selected similarity maps SMA.

[0027] Any method may be used to integrate a plurality of similarity maps SMA into one similarity map SMB, but for example, the following method may be applied. For each corresponding position in the multiple similarity maps SMA, the sum of similarities or a statistical value (such as the mean or median) is calculated and used as the similarity for the corresponding position in the similarity map SMB. Instead of the sum of similarities, a weighted sum of similarities or a sum of similarities after bias adjustment may be used. A single similarity map SMB is created by combining multiple similarity maps SMA. Combining involves, for example, vectorizing each of the multiple similarity maps SMA and concatenating the multiple vectors.

[0028] As described above, the feature FB can be extracted from data of multiple modalities, such as text data, audio data, still image data, and video data. Therefore, the similarity map SMA representing the similarity for the feature FB also corresponds to one of the multiple modalities. The selection unit 104 can select multiple similarity maps SMA corresponding to the multiple modalities and integrate them into the similarity map SMB.

[0029] The similarity map SMB output by the selection unit 104 is used for estimation by the estimation unit 105. The similarity map SMB is an example of information based on a plurality of similarity maps SMA.

[0030] That is, the estimation unit 105 estimates a target object included in an input image using information (similarity map SMB) based on a plurality of similarity maps SMA. Any estimation method using the similarity map SMB may be used, and for example, a method using a pre-trained estimation model MB may be applied. The estimation model MB is a model that is trained to input the similarity map SMB and output an estimation result of the target object. The estimation model MB may be a model of any form, and for example, is a network model such as a convolutional neural network model or a Transformer.

[0031] The estimation result of the target object is, for example, information indicating the class of the target object. The estimation model MB may be a model that outputs, as the estimation result, information indicating the position of the target object in the similarity map SMB together with the target object.

[0032] The output control unit 106 controls the output of various types of information used in the information processing device 100. For example, the output control unit 106 outputs information indicating the target object estimated by the estimation unit 105. The output control unit 106 may output information in any manner, but for example, a method of displaying the information on the display unit 122, or a method of transmitting the information to an external device via a network can be applied.

[0033] The registration unit 110 is a function for registering feature quantities FB of multiple target objects. When feature quantities FB created in advance are used, the registration unit 110 may not be provided. Alternatively, the information processing device 100 may be configured to have only the function for registering feature quantities FB (the registration unit 110).

[0034] The registration unit 110 includes a feature extraction unit 111 and an integration unit 112. Note that the registration data used in the registration process by the registration unit 110 is acquired by, for example, the acquisition unit 101 as described above.

[0035] For example, the acquisition unit 101 acquires a plurality of enrollment data, which are data of a plurality of modalities and represent a plurality of target objects. The type of target object represented by the enrollment data is designated in advance by, for example, a user. The enrollment data may include enrollment data D1 (first enrollment data) and enrollment data D2 (second enrollment data), which are a plurality of enrollment data representing the same target object (first target object) but having different modalities.

[0036] The feature extraction unit 111 extracts, for each of a plurality of registered data, feature values ​​FC that represent the features of the target object represented by the registered data. Any method may be used to extract the feature values ​​FC from the registered data, but for example, a method using a pre-trained extraction model MC may be applied. The extraction model MC is a model that is trained to input the registered data and output the feature values ​​FC. For example, the extraction model MC may use a model that corresponds to the image encoder of a multimodal deep learning model that links data of different modalities into a common feature space, such as CLIP and Imagebind.

[0037] The extracted feature FC is stored in, for example, the storage unit 121, and can be used as a feature FB used to calculate the similarity between the feature FA. Therefore, the feature FC is expressed in the same format (for example, a feature vector) as the feature FA and the feature FB.

[0038] The feature values ​​FC extracted from a plurality of registered data representing the same target object but in different modalities may be integrated and stored. The integration unit 112 is used to integrate the feature values ​​FC in such a case.

[0039] That is, the integration unit 112 integrates the features extracted from the registration data D1 (first registration data) with the features extracted from the registration data D2, which represents the same target object as the registration data D but has a different modality, and outputs the integrated features as the features of the target object.

[0040] As a method for integrating the feature amounts, for example, the following method can be applied. The average value of the feature is calculated as the integrated feature. Multiple features are classified into multiple clusters (clustering), and the representative value of one or more features contained in each cluster is calculated as an integrated feature. The representative value can be, for example, the feature closest to the center of gravity of the cluster, or a statistical value (average, median, etc.) of one or more features.

[0041] In an example using clustering, the number of feature quantities calculated for one target object is equal to the number of clusters. Note that the integration unit 112 may further calculate a statistical value (such as an average or median) of the feature quantities corresponding to each cluster, and calculate the calculated statistical value as one feature quantity for the target object.

[0042] In this way, by registering a feature that integrates multiple feature amounts for the same target object, it becomes possible to perform target estimation processing using the registered feature amounts with higher accuracy.

[0043] At least a part of each of the above units (acquisition unit 101, feature extraction unit 102, calculation unit 103, selection unit 104, estimation unit 105, output control unit 106, and registration unit 110) may be realized by one or more processing units. Each of the above units is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0044] Furthermore, the information processing device 100 may be physically configured as one device, or may be physically configured as multiple devices. For example, the information processing device 100 may be constructed in a cloud environment. Furthermore, each unit in the information processing device 100 may be distributed across multiple devices. For example, the information processing device 100 (information processing system) may be configured to include a device (e.g., a registration device) that has a function for registering registration data (such as the registration unit 110 and the function of the acquisition unit 101 for acquiring registration data) and a device (e.g., an object detection device) that has other functions.

[0045] Next, a description will be given of the estimation process performed by the information processing apparatus 100 according to the first embodiment. Fig. 2 is a flowchart showing an example of the estimation process according to the first embodiment.

[0046] The acquisition unit 101 acquires an input image and, if necessary, converts the input image to the input size of the extraction model MA (step S101). The conversion is, for example, a process of enlarging or reducing the resolution of the input image so that it becomes the resolution defined as the input data of the extraction model MA.

[0047] The feature extraction unit 102 inputs an input image to an extraction model MA and extracts a feature map FMA (step S102). The calculation unit 103 calculates a plurality of similarity maps SMA that represent the distribution of similarities between each feature FA in the extracted feature map FMA and a plurality of feature FB stored in the storage unit 121 (step S103).

[0048] The selection unit 104 selects one or more similarity maps SMA (e.g., the top N similarity maps SMA) from the multiple similarity maps SMA that have high similarity statistical values ​​(e.g., average values), and outputs a similarity map SMB that integrates the selected similarity maps SMA (step S104).

[0049] The estimation unit 105 inputs the similarity map SMB into the estimation model MB to estimate the target object (step S105). The output control unit 106 outputs the estimation result by the estimation unit 105 (step S106), and the estimation process ends.

[0050] Fig. 3 is a diagram showing an example of a display screen for outputting an estimation result. As shown in Fig. 3, the display screen includes an input image 310 including an area 311 indicating the position of the estimated target object, and basis information 321 and 322. The display screen may further include information (such as text data) indicating the class of the target object.

[0051] The basis information 321 and 322 are information indicating the basis for estimating the estimation result. In the example of FIG. 3, the basis information 321 and 322 include registered data used for the estimation and a similarity map corresponding to the registered data. The basis information 321 includes registered data whose modal is text ("tie", "person") and a similarity map calculated using the feature FB extracted from the registered data. The basis information 322 includes registered data whose modal is an image and a similarity map calculated using the feature FB extracted from the registered data. Note that the estimation process uses a similarity map SMB that combines these similarity maps.

[0052] By outputting a display screen such as that shown in Fig. 3, for example, the user can confirm the basis on which the estimation result was obtained. Furthermore, the user can select registered data that is valid for the estimation result. For example, when the user selects valid registered data, the feature values ​​FB corresponding to the unselected registered data may be deleted from the storage unit 121. This allows only the feature values ​​FB that are valid for the estimation process to be used in the estimation process, and the estimation process can be performed with higher accuracy.

[0053] Next, a description will be given of the registration process performed by the information processing apparatus 100 according to the first embodiment. Fig. 4 is a flowchart showing an example of the registration process according to the first embodiment.

[0054] The registration unit 110 acquires registration data via, for example, the acquisition unit 101, and converts the registration data to the input size of the extraction model MC as needed (step S201). The feature extraction unit 111 inputs the registration data to the extraction model MC and extracts feature quantities FC (step S202). When multiple registration data are acquired, the processes of steps S201 and S202 are executed for each of the multiple registration data.

[0055] When multiple registration data of different modals for the same target object are registered, the integration unit 112 integrates multiple feature amounts FC extracted from the multiple registration data for each target object (step S203). The integration unit 112 stores the feature amounts FC (or the integrated feature amounts FC if integrated) for each target object in the storage unit 121 (step S204), and ends the registration process.

[0056] This embodiment can be applied to object detection in an area where target objects change frequently, such as a factory. For example, when it becomes necessary to detect a new target object (hereinafter, a new object), a user instructs a registration process using the registration data of the new object. In accordance with this instruction, the registration unit 110 extracts a feature FB based on the registration data (text data, image data, etc.) of the new object and registers it in the storage unit 121. After this, it becomes possible to detect target objects including the new object (estimation process) using, for example, an image captured by an imaging device (such as a camera) in the area as an input image.

[0057] In addition, the detection results (estimation results) can be output together with information on the basis of the estimation. This allows the user to check the accuracy of the detection of target objects, including new objects. Depending on the confirmation results, the user can select registration data and register new registration data.

[0058] In the technology for classifying regions in an image as described above, for example, one feature representing the characteristics of each region is compared with the feature values ​​of multiple pre-registered target objects. Then, a target object with a high degree of similarity between the feature values ​​is estimated as the target object included in the input image. When multiple target objects are included in a region, one feature value that includes the features of each of the multiple target objects is used. Therefore, for example, the target object that has a higher similarity (responds more strongly) to the feature value of the input image among the multiple target objects is output as the estimation result.

[0059] An example of estimation processing for input image 310 as shown in FIG. 3 will be described. Assume that region 311 of input image 310 is specified as the target of estimation. Assume also that feature quantities extracted from a person ("person") and a tie ("tie") are registered as feature quantities to be matched. The object to be detected from region 311 is the person ("person"), which occupies a large portion of region 311. However, region 311 also contains a tie as a target object. Therefore, the similarity to the feature quantity corresponding to the tie becomes greater, and the tie may be detected as the target object from region 311. In this way, technology that classifies regions in an image may erroneously recognize a region as a different object that reacts strongly (e.g., has a higher similarity).

[0060] In contrast, in this embodiment, an object in an input image is estimated using a similarity map that represents the distribution of similarities between feature amounts for multiple positions in the input image and feature amounts of registered target objects. In other words, estimation is performed using information (similarity map) that includes more detailed information, namely, similarities for multiple positions. This makes it possible to perform estimation processing of objects included in an image with higher accuracy.

[0061] (Second embodiment) The information processing device of the second embodiment uses a feature map to extract one or more candidate areas representing candidates for an area including a target object from an input image, and estimates the target object for each candidate area.

[0062] Fig. 5 is a block diagram showing an example of the configuration of an information processing device 100-2 according to the second embodiment. As shown in Fig. 5, the information processing device 100-2 includes an acquisition unit 101, a feature extraction unit 102, a calculation unit 103-2, a candidate extraction unit 107-2, a selection unit 104-2, an estimation unit 105-2, an output control unit 106, a registration unit 110, a storage unit 121, and a display unit 122.

[0063] The second embodiment differs from the first embodiment in that a candidate extraction unit 107-2 is added and in the functions of a calculation unit 103-2, a selection unit 104-2, and an estimation unit 105-2. The other configurations and functions are the same as those of the information processing device 100 of the first embodiment in FIG. 1, which is a block diagram of the information processing device 100, and therefore the same reference numerals are used and the description thereof will be omitted here.

[0064] The calculation unit 103-2 has the same function as the calculation unit 103 in the first embodiment, that is, a function of calculating the similarity map SMA by using the feature amount FB and the feature map FMA stored in the storage unit 121. The functions added to the calculation unit 103-2 will be described later.

[0065] The candidate extraction unit 107-2 extracts one or more candidate regions from the feature map FMA by using the feature map FMA extracted by the feature extraction unit 102 and the similarity map SMA calculated by the calculation unit 103-2. The candidate regions are regions that represent candidates for the region that includes the target object to be estimated.

[0066] Any method may be used to extract the candidate region, but for example, a method using a pre-trained extraction model MD can be applied. The extraction model MD is a model that is trained to input a feature map FMA and a similarity map SMA and output one or more candidate regions. For example, the extraction model MD can use a model such as Mask R-CNN (e.g., Non-Patent Document 4).

[0067] As described above, multiple similarity maps SMA can be calculated for multiple feature quantities FB. The candidate extraction unit 107-2 may input each of the multiple similarity maps SMA into the extraction model MD to extract multiple candidate regions. The candidate extraction unit 107-2 may input a similarity map obtained by integrating multiple similarity maps SMA into the extraction model MD to extract candidate regions. As a method for integrating multiple similarity maps SMA, a method similar to the method used by the selection unit 104 described above for integrating multiple similarity maps SMA into a similarity map SMB can be applied.

[0068] In this embodiment, the calculation unit 103-2 further has a function of calculating a similarity map SMC (third similarity map) representing the distribution of similarity between the feature amount FB for each of one or more candidate regions. For example, the calculation unit 103-2 extracts, for each of one or more candidate regions, from the feature map FMA a partial feature map representing the distribution of feature amounts for each position included in the candidate region. Any method may be used to extract the partial feature map corresponding to the candidate region, and a method such as ROI Align (e.g., Non-Patent Document 4) can be applied.

[0069] The calculation unit 103-2 calculates a similarity map SMC representing the distribution of similarities between each of the plurality of feature values ​​FB and the plurality of feature values ​​FA included in the partial feature map. The similarity map SMA corresponds to a distribution representing areas with high similarity in the entire input image. On the other hand, the similarity map SMC corresponds to a distribution representing areas with high similarity for each candidate area extracted for each target object.

[0070] The subsequent processing is performed for each candidate region. For example, the selection unit 104-2 performs processing similar to that of the selection unit 104 in the first embodiment, using a similarity map SMC calculated for each candidate region instead of the similarity map SMA. That is, the selection unit 104-2 selects, for each candidate region, one or more similarity maps SMC from the multiple similarity maps SMC corresponding to each of the multiple feature amounts FB, and outputs a similarity map SMB (second similarity map) based on the selected similarity map SMC.

[0071] The estimation unit 105-2 estimates an object included in the input image for each of one or more candidate regions using information based on the plurality of similarity maps SMC (similarity maps SMB).

[0072] Next, the estimation process performed by the information processing device 100-2 according to the second embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the estimation process according to the second embodiment.

[0073] Steps S301 to S303 are the same as steps S101 to S103 in the information processing apparatus 100 of the first embodiment, and therefore a description thereof will be omitted.

[0074] The candidate extraction unit 107-2 inputs the feature map FMA extracted by the feature extraction unit 102 and the similarity map SMA calculated by the calculation unit 103-2 into the extraction model MD, and extracts one or more candidate regions (step S304).

[0075] For each candidate region, the calculation unit 103-2 extracts a partial feature map from the feature map FMA that corresponds to the candidate region, and calculates a similarity map SMC that represents the distribution of similarities between each of the multiple feature values ​​FB and the multiple feature values ​​FA included in the partial feature map (step S305).

[0076] The selection unit 104-2 selects, for each candidate region, one or more similarity maps SMC (e.g., the top N similarity maps SMA) having high similarity statistical values ​​(e.g., average values) from among the multiple similarity maps SMC, and outputs a similarity map SMB that integrates the selected similarity maps SMC (step S306).

[0077] The estimation unit 105-2 inputs the similarity map SMB into the estimation model MB for each candidate region and estimates the target object (step S307). The output control unit 106 outputs the estimation result by the estimation unit 105 (step S308), and the estimation process ends.

[0078] In this way, in the second embodiment, one or more candidate regions including the target object are extracted from the input image using the feature map, and the target object can be estimated for each candidate region.

[0079] As described above, according to the first and second embodiments, it is possible to more accurately execute the estimation process of the object included in the image.

[0080] Next, the hardware configuration of the information processing apparatus according to the first or second embodiment will be described with reference to Fig. 7. Fig. 7 is an explanatory diagram showing an example of the hardware configuration of the information processing apparatus according to the first or second embodiment.

[0081] The information processing device of the first or second embodiment includes a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 or a RAM (Random Access Memory) 53, a communication I / F 54 that connects to a network and communicates, and a bus 61 that connects each part.

[0082] The programs executed by the information processing device of the first or second embodiment are provided in advance in the ROM 52 or the like.

[0083] The program executed by the information processing device of the first or second embodiment may be configured to be provided as a computer program product by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0084] Furthermore, the program executed by the information processing device of the first or second embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program executed by the information processing device of the first or second embodiment may be provided or distributed via a network such as the Internet.

[0085] The program executed by the information processing device of the first or second embodiment can cause the computer to function as each part of the information processing device described above. In this computer, the CPU 51 can read the program from a computer-readable storage medium onto the main storage device and execute it.

[0086] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0087] 100, 100-2 Information processing device 101 Acquisition Department 102 Feature Extraction Unit 103, 103-2 Calculation section 104, 104-2 Selection section 105, 105-2 Estimation part 106 Output control section 107-2 Candidate Extraction Unit 110 Registration Department 111 Feature Extraction Unit 112 Integration Department 121 Storage section 122 Display section

Claims

1. On the computer, a feature extraction step of extracting, from an input image, a first feature map representing a distribution of a plurality of first feature amounts, which are features for a plurality of first positions corresponding to a plurality of positions in the input image; a calculation step of calculating a first similarity map representing a distribution of similarities between a plurality of second feature amounts, which are features of a plurality of pre-registered targets, and the plurality of first feature amounts included in the first feature map; an estimation step of estimating an object included in the input image using information based on the plurality of first similarity maps calculated for the plurality of second feature amounts; A program to execute.

2. The computer, further executing a selection step of selecting one or more of the first similarity maps from the plurality of first similarity maps and outputting a second similarity map based on the selected first similarity map; the estimating step estimates an object included in the input image using the second similarity map; The program according to claim 1.

3. the selecting step includes selecting one or more of the first similarity maps from a plurality of the first similarity maps based on a statistical value of the similarity for each position, and outputting the second similarity map based on the selected first similarity map. The program according to claim 2.

4. each of the plurality of objects is represented by one of a plurality of types of data including text data, audio data, still image data, and moving image data; the selecting step selects a plurality of the first similarity maps calculated using a plurality of the second feature amounts corresponding to a plurality of the targets represented by data of different types from each other, and outputs the second similarity map obtained by integrating the selected plurality of the first similarity maps. The program according to claim 2.

5. The computer, further executing a candidate extraction step of extracting one or more candidate regions in the first feature map, the candidate regions representing candidates for a region including an object to be estimated, using the first feature map and a plurality of the first similarity maps; the calculating step includes extracting, for each of the one or more candidate areas, a partial feature map from the first feature map that represents a distribution of feature amounts for each position included in the candidate area, and calculating, for each of a plurality of second feature amounts, a third similarity map that represents a distribution of similarities between the second feature amount and a plurality of first feature amounts included in the partial feature map; the estimating step estimates an object included in the input image for each of the one or more candidate regions using information based on the plurality of third similarity maps; The program according to claim 1.

6. the calculating step calculates, for each of the plurality of second feature amounts, the first similarity map representing a distribution of the similarities, which are distances between the second feature amount and the plurality of first feature amounts included in the first feature map; The program according to claim 1.

7. the estimation step includes inputting information based on the plurality of first similarity maps and estimating the object included in the input image using an estimation model trained to output an estimation result of the object; The program according to claim 1.

8. each of the plurality of objects is represented by one of a plurality of types of data including text data, audio data, still image data, and moving image data; the plurality of second feature amounts represent features of any of the plurality of types of data; The program according to claim 1.

9. To the computer further executing an output control step of outputting information indicating the estimated target; The program according to claim 1.

10. On the computer, an acquiring step of acquiring a plurality of registered data, each of which is any of a plurality of types of data and represents a plurality of objects, the plurality of registered data including first registered data and second registered data, each of which represents a first object among the plurality of objects and is of a different type from each other; a feature extraction step of extracting, for each of the plurality of registered data, a feature amount representing a feature of the object represented by the registered data; an integration step of integrating the feature amount extracted from the first registered data and the feature amount extracted from the second registered data and outputting the integrated feature amount as the feature amount of the first object; A program to execute.

11. a feature extraction unit that extracts, from an input image, a first feature map that represents a distribution of a plurality of first feature amounts that are characteristics for each of a plurality of positions in the input image; a calculation unit that calculates a first similarity map representing a distribution of similarities between a plurality of second feature amounts, which are features of a plurality of pre-registered targets, and the plurality of first feature amounts included in the first feature map; an estimation unit that estimates an object included in the input image using information based on the plurality of first similarity maps calculated for the plurality of second feature amounts; An information processing device comprising:

12. An information processing method executed by an information processing device, a feature extraction step of extracting, from an input image, a first feature map representing a distribution of a plurality of first feature amounts, which are features for a plurality of first positions corresponding to a plurality of positions in the input image; a calculation step of calculating a first similarity map representing a distribution of similarities between a plurality of second feature amounts, which are features of a plurality of pre-registered targets, and the plurality of first feature amounts included in the first feature map; an estimation step of estimating an object included in the input image using information based on the plurality of first similarity maps calculated for the plurality of second feature amounts; An information processing method including:

Images