Information processing device, information processing method, and program

Abstract

It searches for images corresponding to the entered text with high accuracy. [Solution] In the information processing device, the acquisition means acquires text and an image to be compared. The score calculation means calculates a score indicating the degree of consistency between the text and the image. The correction value calculation means calculates a correction value for the score based on the image using a correction model. The correction means corrects the score using the correction value and outputs the corrected score. The learning means learns the correction model to optimize the correction value.

Description

Technical Field

[0006] , , ,

[0001] This disclosure relates to image search.

Background Art

[0002] Methods for searching for a desired image from a large number of images have been proposed. For example, Patent Document 1 describes a method for searching for a target image that matches a search text based on the search text.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Since the method of Patent Document 1 determines the target image based on the similarity between the search text and a plurality of images to be searched, the accuracy of the search depends on the similarity calculation method. Therefore, when the similarity value changes depending on the characteristics of the images to be searched, for example, the way an object is depicted in the image, there is a risk that the search accuracy will decrease.

[0005] One object of this disclosure is to provide an information processing apparatus capable of highly accurately searching for an image corresponding to an input text.

Means for Solving the Problems

[0006] In one aspect of this disclosure, the information processing apparatus acquisition means for acquiring a text and an image to be compared, score calculation means for calculating a score indicating the degree of matching between the text and the image, corrected value calculation means for calculating a corrected value of the score using a correction model based on the image, A correction means that corrects the score using the correction value and outputs the corrected score, A learning means for learning the correction model to optimize the correction value, It is equipped with.

[0007] From another perspective of this disclosure, computer-based information processing methods are: Obtain the text and the image to be compared, A score indicating the degree of consistency between the text and the image is calculated. Based on the aforementioned image, the correction value of the score is calculated using the correction model. The score is corrected using the correction value, and the corrected score is output. The correction model is trained to optimize the aforementioned correction value.

[0008] In yet another aspect of this disclosure, the program is Obtain the text and the image to be compared, A score indicating the degree of consistency between the text and the image is calculated. Based on the aforementioned image, the correction value of the score is calculated using the correction model. The score is corrected using the correction value, and the corrected score is output. The computer is made to perform a process to learn the correction model so as to optimize the correction value. [Effects of the Invention]

[0009] This disclosure makes it possible to provide an information processing device that can search for images corresponding to input text with high accuracy. [Brief explanation of the drawing]

[0010] [Figure 1] This shows the overall configuration of an image search system related to an example of this disclosure. [Figure 2] This is a block diagram showing the hardware configuration of an information processing device. [Figure 3] This is a block diagram showing the functional configuration of a learning device. [Figure 4] It is a block diagram showing the functional configuration of an information processing apparatus. [Figure 5] It is a block diagram showing the functional configuration of another learning apparatus. [Figure 6] It is a block diagram showing the functional configuration of another information processing apparatus. [Figure 7] It is a block diagram showing the functional configuration of another learning apparatus. [Figure 8] It is a block diagram showing the functional configuration of another information processing apparatus. [Figure 9] It is a flowchart of learning processing. [Figure 10] It is a flowchart of image search processing. [Figure 11] It is a diagram for explaining a calculation example of a matching score. [Figure 12] It is another diagram for explaining a calculation example of a matching score. [Figure 13] It is a block diagram showing the functional configuration of another information processing apparatus. [Figure 14] It is a flowchart of processing by another information processing apparatus.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. <Summary Explanation> A known method for performing image searches using text as input is one that uses the similarity between the input text and the target image. However, depending on the method used to calculate the similarity, the resulting similarity score may be affected by the content and features of the image. For example, when using cosine similarity, even if the object described in the text is present in the image, the resulting similarity score may be low if other objects are also present in the image. Furthermore, even when using similarity methods other than cosine similarity, the resulting similarity score may fluctuate depending on the position, size, and appearance of the object in the image. Therefore, in the following embodiment, the accuracy of image searches is improved by correcting the similarity score based on the features of the image.

[0012] <First Embodiment> [Overall structure] Figure 1 shows the overall configuration of an image search system according to an example of the present disclosure. The image search system 1 searches for images corresponding to text entered by the user. As shown in Figure 1, the image search system 1 comprises an image database (hereinafter, "database" will be referred to as "DB") 2 and an image search device 3. The image search device 3 comprises an information processing device 100 and an output unit 200.

[0013] Image DB2 stores multiple images that are the target of the search. Image DB2 may also store feature quantities extracted from each image (hereinafter referred to as "image features") associated with these multiple images.

[0014] When a user enters text indicating the search target, the information processing device 100 retrieves images from the image database 2 that match the entered text and outputs them as search results. As will be described in detail later, the information processing device 100 calculates a matching (matching) score between the entered text and multiple images stored in the image database 2 and outputs it to the output unit 200. The output unit 200 retrieves a predetermined number of images with high matching scores from the image database 2 and outputs them as search results. For example, the output unit 200 outputs k images with the highest matching scores, sorted by their matching scores, to a display device or the like.

[0015] [Hardware configuration] Figure 2 is a block diagram showing the hardware configuration of the information processing device 100. As shown in the figure, the information processing device 100 comprises a processor 11, an interface (IF) 12, a ROM (Read Only Memory) 13, a RAM (Random Access Memory) 14, a database (DB) 15, and a recording medium 16. Each component is connected to the others, for example, via a bus 18.

[0016] The processor 11 is a computer such as a CPU (Central Processing Unit) and controls the entire information processing device 100 by executing a pre-prepared program. Specifically, the processor 11 can be a CPU, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof.

[0017] Furthermore, the processor 11 loads programs stored in the ROM 13 and recording medium 16 into the RAM 14 and executes each process coded in the program. The processor 11 functions as part or all of the information processing device 100. Specifically, the processor 11 performs the learning process and image retrieval process described later.

[0018] IF12 transmits and receives data to and from external devices. Specifically, the information processing device 100 obtains text entered by the user through IF12. The information processing device 100 also accesses the image DB2 through IF12 to obtain images and image features. Furthermore, the information processing device 100 outputs the image search results to a display device or other external devices through IF12.

[0019] ROM 13 stores various programs executed by processor 11. RAM 14 is used as working memory while processor 11 is executing various processes.

[0020] DB15 stores various algorithms, data, and machine learning models that the information processing device 100 uses when it performs the learning process and image search process described later.

[0021] The recording medium 16 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or semiconductor memory. The recording medium 16 may be configured to be detachable from the information processing device 100. The recording medium 16 stores various programs executed by the processor 11.

[0022] In addition to the above, the information processing device 100 may also be equipped with a display device such as a liquid crystal display, and an input device such as a keyboard or mouse. These display devices and input devices are used, for example, by the operator of the information processing device 100.

[0023] [First Embodiment] (Learning device) Figure 3 is a block diagram showing the functional configuration of a learning device according to the first embodiment. This learning device is for learning a correction model that calculates a correction value for the matching score. As shown in the figure, the learning device 10a comprises a score calculation unit 112, a correction value calculation unit 113, a score correction unit 114, and a correction model learning unit 115.

[0024] Training data is input to the learning device 10a. The training data includes feature quantities corresponding to text (hereinafter referred to as "text features"), image features corresponding to images, and correct labels corresponding to pairs of text features and image features (hereinafter also referred to as "text-image pairs"). Specifically, in a given text-image pair, if the object indicated by the text is depicted in the image, the text and image are considered to be in agreement (matches), and the text-image pair is called a "positive example pair," and a value indicating a positive example pair (for example, "1") is assigned as the correct label. On the other hand, in a given text-image pair, if the object indicated by the text is not depicted in the image, the text and image are considered to be in agreement, and the text-image pair is called a "negative example pair," and a value indicating a negative example pair (for example, "0" or "-1") is assigned as the correct label.

[0025] During training, the training data described above is input to the training device 10a. Specifically, text features T are input to the score calculation unit 112, and image features I are input to both the score calculation unit 112 and the correction value calculation unit 113.

[0026] The score calculation unit 112 calculates a matching score s between the image feature I and the text feature T and outputs it to the score correction unit 114. The matching score s is a score that indicates the degree of consistency between the image feature I and the text feature T. Basically, if the object indicated by the text feature T is visible in the image, the matching score s will be high, and if it is not visible, the matching score s will be low. For example, the score calculation unit 112 calculates the cosine similarity between the image feature I and the text feature T as the matching score s. The score calculation unit 112 may also calculate a similarity other than cosine similarity as the matching score s.

[0027] The correction value calculation unit 113 calculates a correction value c for the matching score s based on the input image features I and outputs it to the score correction unit 114. Specifically, the correction value calculation unit 113 calculates the correction value c using a machine learning model, the correction model M1. The correction value c is a value that reduces the influence of the surrounding environment of the search target in the image on the matching score. For example, the correction model M1 is composed of a neural network and is expressed as follows. NeuralNetA1(I)=c (1)

[0028] The score correction unit 114 corrects the matching score s using the correction value c. Specifically, the score correction unit 114 corrects the matching score s using the following correction formula and calculates the corrected matching score s'. s / c=s' (2)

[0029] Thus, since the score correction unit 114 uses a correction formula that requires less computation, the computational load required to correct the matching score s can be reduced. The score correction unit 114 outputs the corrected matching score s' to the correction model learning unit 115. The corrected matching score s' is a score in which the influence of the surrounding environment of the search target in the image on the matching score has been reduced. In the following explanation, the matching score s before correction may be referred to as the "uncorrected matching score s" to distinguish it from the corrected matching score s'.

[0030] The correction model learning unit 115 optimizes the correction model M1 using the corrected matching score s' and the aforementioned ground truth labels. Specifically, the correction model learning unit 115 updates the correction model M1 using gradient descent to minimize the error between the corrected matching score s' and the ground truth labels.

[0031] In this way, the correction model M1 is trained using pre-prepared training data. When the predetermined training termination conditions are met, training is completed and the trained correction model M1 is obtained.

[0032] (Information processing device) Next, we will describe an information processing device that performs inference using the correction model M1 learned by the learning device 10a described above. Here, inference refers to calculating a corrected matching score between the text input by the user and the image. Figure 4 is a block diagram showing the functional configuration of the information processing device 100a. As shown in the figure, the information processing device 100a comprises an encoder 111, a score calculation unit 112, a correction value calculation unit 113, and a score correction unit 114. Here, the score calculation unit 112 and the score correction unit 114 are the same as those of the learning device 10a shown in Figure 3. The correction value calculation unit 113 uses the correction model M1 learned by the learning device 10a.

[0033] During image search, the text entered by the user is input to the encoder 111. Additionally, image features of the images to be compared, obtained from the image DB2, are input to the score calculation unit 112 and the correction value calculation unit 113. The encoder 111 converts the input text into text features T and outputs them to the score calculation unit 112. The score calculation unit 112 calculates the cosine similarity between the text features T and the image features I as a matching score s and outputs it to the score correction unit 114.

[0034] Meanwhile, the correction value calculation unit 113 calculates a correction value c from the image features I using the trained correction model M1 and outputs it to the score correction unit 114. The score correction unit 114 corrects the matching score s using the correction value c according to the correction formula (2) above and outputs the corrected matching score s'. In this way, the corrected matching score s' between the text entered by the user and one image is obtained. The information processing device 100a performs this process for multiple images and outputs the corrected matching score s' for each image.

[0035] Furthermore, since the images targeted for image retrieval are predetermined images, such as those stored in the image DB2, a correction value c corresponding to each image can be calculated using the trained correction model M1 before starting the actual inference process, and stored in memory or elsewhere in association with the image or image features. In this way, during the actual inference process, the correction value calculation unit 113 only needs to retrieve the pre-calculated correction value c from memory instead of calculating the correction value c for each image, thereby shortening the time required for the actual image retrieval.

[0036] As described above, according to the first embodiment, a correction model M1 is trained using training data that includes positive example pairs and negative example pairs, and a correction value c is calculated using the trained correction model M1. Therefore, a corrected matching score s' corresponding to the text entered by the user can be calculated with high accuracy.

[0037] [Second Example] (Learning device) Figure 5 is a block diagram showing the functional configuration of a learning device according to the second embodiment. This learning device is for learning a correction model that calculates a correction value for the matching score. As shown in the figure, the learning device 10b comprises a score calculation unit 112, a correction value calculation unit 123, and a correction model learning unit 125.

[0038] In the second embodiment, the correction model calculates a predicted matching score based solely on image features. Therefore, in the second embodiment, positive example pairs, i.e., pairs of positive example image features and text features, are used as training data. Furthermore, in the second embodiment, the training data does not need to include correct labels as in the first embodiment.

[0039] Text features T included in the training data are input to the score calculation unit 112, and image features I are input to both the score calculation unit 112 and the correction value calculation unit 123. The score calculation unit 112 is basically the same as in the first embodiment, and calculates the matching score s between the image features I and the text features T, and outputs it to the correction model learning unit 125.

[0040] The correction value calculation unit 123 calculates a correction value c of the matching score s based on the input image features I and outputs it to the correction model learning unit 125. Specifically, the correction value calculation unit 113 calculates the correction value c using a machine learning model, the correction model M2. Here, unlike the first embodiment, the correction model M2 is learned to output a predicted value of the matching score s output by the score calculation unit 112 based only on the input image features I. In other words, the correction model M2 is learned to predict and output the trend of the magnitude of the matching score caused by the image. From this point of view, in the second embodiment, the correction value c corresponds to a predicted value of the matching score, and is hereafter referred to as the "predicted matching score c". For example, the correction model M2 is composed of a neural network and is represented as follows. NeuralNetA2(I)=c (3)

[0041] The correction model learning unit 125 optimizes the correction model M2 using the matching score s input from the score calculation unit 112 and the predicted matching score c input from the correction value calculation unit 123. Specifically, the correction model learning unit 125 updates the correction model M2 using gradient descent to minimize the error between the matching score s and the predicted matching score c.

[0042] In this way, the corrected model M2 is trained using pre-prepared training data. When the predetermined training termination conditions are met, training is completed and the trained corrected model M2 is obtained.

[0043] (Information processing device) Next, an information processing device that performs inference using the correction model M2 learned by the learning device 10b described above will be explained. Here, inference refers to calculating a corrected matching score between the text input by the user and the image. Figure 6 is a block diagram showing the functional configuration of the information processing device 100b. As shown in the figure, the information processing device 100b comprises an encoder 111, a score calculation unit 112, a correction value calculation unit 123, and a score correction unit 114. Here, the encoder 111, the score calculation unit 112, and the score correction unit 114 are the same as those in the information processing device 100a of the first embodiment shown in Figure 4. The correction value calculation unit 123 uses the correction model M2 learned by the learning device 10b.

[0044] During image search, the text entered by the user is input to the encoder 111. Additionally, image features of the images to be compared, obtained from the image DB2, are input to the score calculation unit 112 and the correction value calculation unit 123. The encoder 111 converts the input text into text features T and outputs them to the score calculation unit 112. The score calculation unit 112 calculates the cosine similarity between the text features T and the image features I as a matching score s and outputs it to the score correction unit 114.

[0045] Meanwhile, the correction value calculation unit 123 calculates a predicted matching score c (corrected value c) from the image features I using the trained correction model M2 and outputs it to the score correction unit 114. The score correction unit 114 corrects the matching score s using the predicted matching score c according to the correction formula (2) and outputs the corrected matching score s'.

[0046] In the second embodiment, the matching score s is corrected using the predicted matching score c output by the correction model M2. Therefore, the score correction unit 114 increases the matching score s when the predicted matching score c is small, and decreases the matching score s when the predicted matching score c is large. This makes it possible to suppress the influence of the image-dependent trend of the matching score on the final output matching score.

[0047] In this way, a corrected matching score s' is obtained between the text entered by the user and one image. The information processing device 100b performs this process on multiple images and outputs a corrected matching score s' for each image.

[0048] In the second embodiment as well, since the images to be searched are predetermined images, such as images stored in the image DB2, the trained correction model M2 can be used to calculate the predicted matching score c corresponding to each image before starting the actual inference process, and this score can be stored in memory or elsewhere in association with the image or image features. In this way, during the actual inference process, the correction value calculation unit 123 only needs to retrieve the pre-calculated predicted matching score c from memory instead of calculating the predicted matching score c for each image, thereby shortening the time required for the actual image search.

[0049] As described above, according to the second embodiment, a correction model M2 is trained using training data corresponding to positive example pairs, and the predicted matching score c is calculated using the trained correction model M2. Therefore, the corrected matching score s' corresponding to the text entered by the user can be calculated with high accuracy.

[0050] [Third Embodiment] (Learning device) Figure 7 is a block diagram showing the functional configuration of the learning device according to the third embodiment. The third embodiment corresponds to a modification of the second embodiment. As shown in the figure, the learning device 10c comprises a score calculation unit 112, a correction value calculation unit 123, a correction model learning unit 135, and a prediction error absorption unit 136.

[0051] In the third embodiment, the matching score is corrected using the correction model M2, similar to the second embodiment. In addition, in the third embodiment, a prediction error absorption unit 136 is added to absorb the prediction error that occurs in the correction model M2 due to the input text. The prediction error absorption unit 136 uses the correction model M3. The correction model M3 has the role of correcting the prediction matching score c output by the correction model M2 based on the text features T. The learning data used by the learning device 10c in the third embodiment is basically the same as the learning data used by the learning device 10b in the second embodiment.

[0052] Text features T included in the training data are input to the score calculation unit 112, and image features I are input to both the score calculation unit 112 and the correction value calculation unit 123. The score calculation unit 112 calculates a matching score s between the image features I and the text features T and outputs it to the correction model learning unit 135. The correction value calculation unit 123 calculates a predicted matching score c using the correction model M2 based on the input image features I and outputs it to the correction model learning unit 135 and the prediction error absorption unit 136.

[0053] The prediction error absorption unit 136 has the role of absorbing the prediction error, i.e., variability, of the prediction matching score c caused by the text features T. Even when the same image is input, if the complexity of the input text differs, variability will occur in the matching score s, which is the learning target of the correction model M2. Therefore, the prediction error absorption unit 136 corrects the prediction matching score c output by the correction model M2 based on the text features T using the correction model M3, and outputs the corrected prediction matching score c' to the correction model learning unit 135. The correction model M3 used by the prediction error absorption unit 136 is composed of a neural network and is expressed as follows. NeuralNetB(T)=c (4) Furthermore, the corrected prediction matching score c' output by the prediction error absorption unit 136 is expressed by the following formula. c × NeuralNetB(T) = c' (5)

[0054] The correction model learning unit 135 optimizes the correction models M2 and M3 using the matching score s input from the score calculation unit 112, the predicted matching score c input from the correction value calculation unit 123, and the corrected predicted matching score c' input from the prediction error absorption unit 136. Specifically, the correction model learning unit 135 updates the correction models M2 and M3 by gradient descent to minimize the weighted sum of the first error between the matching score s and the predicted matching score c, and the second error between the matching score s and the corrected predicted matching score c'.

[0055] In this way, the correction models M2 and M3 are trained using pre-prepared training data. When the predetermined training termination conditions are met, training is completed, and the trained correction models M2 and M3 are obtained.

[0056] (Information processing device) Next, we will describe an information processing device that performs inference using the correction models M2 and M3 learned by the learning device 10c described above. Here, inference refers to calculating the corrected matching score between the text input by the user and the image. Two possible configurations for the information processing device of the third embodiment are as follows.

[0057] The first configuration example uses only the trained model M2. In the learning device 10c described above, the corrected model M2 is trained using the corrected prediction matching score c' output from the prediction error absorption unit 136. Therefore, in the first configuration example, inference is performed using only the corrected model M2. In this case, the configuration of the information processing device is the same as that of the information processing device 100b shown in Figure 6. However, the trained corrected model M2 used by the correction value calculation unit 123 is the one trained by the learning device 10c shown in Figure 7.

[0058] The second configuration example uses both the learned correction models M2 and M3. Figure 8 is a block diagram showing the functional configuration of the information processing device 100c according to the second configuration example of the third embodiment. The information processing device 100c uses the correction models M2 and M3 learned by the learning device 10c described above.

[0059] As shown in the figure, the information processing device 100c comprises an encoder 111, a score calculation unit 112, a correction value calculation unit 123, a score correction unit 124, and a prediction error absorption unit 136. Here, the encoder 111 and the score calculation unit 112 are the same as those in the information processing device 100a of the first embodiment shown in Figure 4. The correction value calculation unit 123 uses a correction model M2 learned by the learning device 10c. The prediction error absorption unit 136 uses a correction model M3 learned by the learning device 10c.

[0060] During image search, the text entered by the user is input to the encoder 111. Additionally, image features of the images to be compared, obtained from the image DB2, are input to the score calculation unit 112 and the correction value calculation unit 123. The encoder 111 converts the input text into text features T and outputs them to the score calculation unit 112 and the prediction error absorption unit 136. The score calculation unit 112 calculates the cosine similarity between the text features T and the image features I as a matching score s and outputs it to the score correction unit 124.

[0061] Meanwhile, the correction value calculation unit 123 uses the trained correction model M2 to calculate the predicted matching score c from the image features I and outputs it to the prediction error absorption unit 136. The prediction error absorption unit 136 uses the trained correction model M3 to modify the predicted matching score c based on the text features T to calculate the modified predicted matching score c' and outputs it to the score correction unit 114.

[0062] The score correction unit 124 corrects the matching score s using the corrected predicted matching score c' according to the correction formula below, and outputs the corrected matching score s'. s / c'=s' (5)

[0063] In this way, a corrected matching score s' is obtained between the text entered by the user and one image. The information processing device 100b performs this process on multiple images and outputs a corrected matching score s' for each image.

[0064] In the third embodiment, the matching score s is corrected using the corrected predicted matching score c' output by the correction model M3, thereby absorbing errors and variability in the matching score caused by the text. Therefore, the corrected matching score s' corresponding to the text entered by the user can be calculated with high accuracy.

[0065] [Learning Process] Next, the learning process by the learning devices of the first to third embodiments described above will be explained. Figure 9 is a flowchart of the learning process by the learning devices 10a to 10c. This process is realized by the processor 11 shown in Figure 2 executing a pre-prepared program and operating as the elements shown in Figure 3, Figure 5, or Figure 7. In the following explanation, when the learning devices 10a to 10c are not distinguished, they will be collectively referred to as "learning device 10".

[0066] First, the learning device 10 acquires text features included in the training data (step S11) and image features corresponding to those text features (step S12). Next, the learning device 10 calculates a matching score s from the text features and image features (step S13). Next, the learning device 10 calculates a correction value c from the image features (step S14). In this case, in the first embodiment, the learning device 10a calculates the correction value c. In the second and third embodiments, the learning devices 10b and 10c calculate the predicted matching score c.

[0067] Next, the learning device 10 updates the correction model based on the correction value (step S15). In the first embodiment, the learning device 10a updates the correction model M1. In the second embodiment, the learning device 10b updates the correction model M2. In the third embodiment, the learning device 10c updates the correction models M2 and M3.

[0068] Next, the learning device 10 determines whether a predetermined learning termination condition has been met (step S16). The predetermined learning termination condition is, for example, that learning has been performed using all of the pre-prepared learning data. If the learning termination condition is not met (step S16: No), the process returns to step S12, and steps S12 to S15 are executed for the next learning data. On the other hand, if the learning termination condition is met (step S16: Yes), the learning process ends.

[0069] [Image search processing] Next, the image search process performed by the image search device 3 equipped with the information processing devices of the first to third embodiments described above will be explained. Figure 10 is a flowchart of the image search process performed by the image search device 3 equipped with information processing devices 100a to 100c. This process is realized by the processor 11 shown in Figure 2 executing a pre-prepared program and operating as the elements shown in Figures 4, 6, or 8. In the following explanation, when the information processing devices 100a to 100c are not distinguished, they will be collectively referred to as "information processing device 100".

[0070] First, the information processing device 100 acquires the text entered by the user (step S21) and generates text features from that text (step S22). Next, the information processing device 100 acquires one image feature to be compared (step S23). Next, the information processing device 100 calculates a matching score s from the text features and image features (step S24).

[0071] Next, the information processing device 100 calculates a correction value c from the image features (step S25). In the first embodiment, the information processing device 100a calculates the correction value c using correction model M1. In the second embodiment, the information processing device 100b calculates the predicted matching score c using correction model M2. In the third embodiment, the information processing device 100c either calculates the predicted matching score c using correction model M2, or calculates the corrected predicted matching score c' using correction models M2 and M3.

[0072] Next, the information processing device 100 corrects the matching score s with a correction value and calculates the corrected matching score s' (step S26). In this case, in the first and second embodiments, the information processing devices 100a and 100b correct the matching score s using a correction value c or a predicted matching score c. In the third embodiment, the information processing device 100c corrects the matching score s using a predicted matching score c or a corrected predicted matching score c'.

[0073] Next, the information processing device 100 determines whether or not it has processed the image features of all images to be searched (step S27). If not all image features have been processed (step S27: No), the process returns to step S23, and steps S23 to S26 are executed for the next image feature. On the other hand, if all image features have been processed (step S27: Yes), the information processing device 100 outputs the corrected matching score s' calculated for all image features to the output unit 200 of the image search device 3. The output unit 200 outputs the top k images with the highest corrected matching score s', along with their corrected matching score s', to a display device or external device (step S28). The image search process then ends.

[0074] [Example of Matching Score Calculation] Next, we will explain an example of calculating the matching score. Let's assume that the images to be compared are, as shown in Figure 11(A), image P1 which shows only an apple, image P2 which shows only an orange, and image P3 which shows an apple and a car. When the text "apple" is entered, the uncorrected matching score s calculated by the score calculation unit 112 is assumed to be "0.8" for image P1, "0.7" for image P2, and "0.6" for image P3. Note that for image P2, the uncorrected matching score s is 0.7 because the text is "apple," but if the text is "orange," that is, if the text and image are a positive example pair, the uncorrected matching score s will be "0.8."

[0075] In Figure 11(A), the uncorrected matching score s for the text "apple" is image P1 > image P2 > image P3. In this example, image P2 does not contain an apple, while image P3 does. However, since image P3 contains not only an apple but also a car, the score calculation unit 112 calculates a higher matching score s for image P2, which does not contain an apple, than for image P3, which does contain an apple. Thus, when a matching score is calculated based on similarity, the accuracy of the matching score may decrease depending on the content of the image, etc.

[0076] Figure 12 illustrates an example of matching score correction in this case. As shown in the figure, consider the case where the matching score is corrected for images P1 to P3 using the information processing device 100a or 100b described above.

[0077] Image P1 contains only apples, and the text entered by the user is "apple". In this case, the uncorrected matching score s = 0.8, and the corrected value (predicted matching score) c = 0.8. Therefore, the corrected matching score s' = 1.0.

[0078] Assume that image P2 contains only oranges, and the text entered by the user is "apple". In this case, the uncorrected matching score s = 0.7. Also, as shown in Figure 11(A), the correction value calculation unit 113 or 123 calculates a correction value (predicted matching score) c = 0.8 based only on image P2. Therefore, the corrected matching score s' = 0.875.

[0079] Image P3 contains an apple and a car, and the text entered by the user is "apple". In this case, the uncorrected matching score s = 0.6, and the corrected value (predicted matching score) c = 0.6. Therefore, the corrected matching score s' = 1.0.

[0080] As shown in Figure 11(B), the corrected matching score s' obtained by the information processing device 100 is such that image P1 ≥ image P3 > image P2, and the corrected matching score s' is higher for image P3, which includes apples and a car, than for image P2, which includes only oranges. Thus, the information processing device 100 makes it possible to suppress the decrease in the matching score caused by factors such as how objects are depicted in the image.

[0081] [Examples of application] The image search method disclosed herein can be used, for example, to understand the situation during a disaster. Specifically, to understand the situation at a disaster site, by entering text such as "houses have collapsed" or "roads are impassable" and searching for images, it is possible to collect images of locations that are in such a state.

[0082] Furthermore, the image search method disclosed herein can be used, for example, to assist police investigations. Specifically, by specifying the color of a vehicle seen at a crime scene and entering "red vehicle," or by specifying the appearance of a person seen at a crime scene and entering "wearing a gray sweat suit," images of vehicles or people related to the crime can be found.

[0083] Furthermore, the image search method disclosed herein can be used, for example, by media outlets that handle large amounts of video and image data when collecting specific images.

[0084] <Second Embodiment> Figure 13 is a block diagram showing the functional configuration of an information processing apparatus according to the second embodiment of this disclosure. The information processing apparatus 70 of the second embodiment includes an acquisition means 71, a score calculation means 72, a correction value calculation means 73, a correction means 74, and a learning means 75.

[0085] Figure 14 is a flowchart of the processing performed by the information processing device of the second embodiment. The acquisition means 71 acquires text and the image to be compared (step S71). The score calculation means 72 calculates a score indicating the degree of consistency between the text and the image (step S72). The correction value calculation means 73 calculates a correction value for the score based on the image using a correction model (step S73). The correction value for the score is a value that reduces the influence of the surrounding environment of the search target in the image on the matching score. The correction means 74 corrects the score using the correction value and outputs the corrected score (step S74). The corrected score is a score in which the influence of the surrounding environment of the search target in the image on the score has been reduced. The learning means 75 learns the correction model to optimize the correction value (step S75).

[0086] According to the information processing device 70 described above, it becomes possible to search for images corresponding to input text with high accuracy.

[0087] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0088] (Note 1) A means for obtaining text and an image to be compared, A score calculation means for calculating a score indicating the degree of consistency between the text and the image, A correction value calculation means that calculates a correction value of the score using a correction model based on the aforementioned image, A correction means that corrects the score using the correction value and outputs the corrected score, A learning means for learning the correction model to optimize the correction value, An information processing device equipped with the following features.

[0089] (Note 2) The information processing device described in Appendix 1, wherein the learning means learns the correction model to minimize the error between the corrected score and the correct label using learning data which includes text, images, and correct labels indicating the degree of consistency between the text and the images.

[0090] (Note 3) The learning means is an information processing device according to Appendix 1, which uses pairs of text and images consistent with the text as training data to train the correction model so as to minimize the error between the score before correction and the correction value.

[0091] (Note 4) The information processing apparatus according to Appendix 3, comprising correction value correction means for correcting the correction value based on the characteristics of the text.

[0092] (Note 5) The correction means is an information processing device according to Appendix 4, which corrects the score using the corrected correction value.

[0093] (Note 6) The information processing device according to Appendix 4, wherein the learning means uses pairs of text and images consistent with the text as training data to train the correction model to minimize a first error between the score before correction and the correction value, and a second error between the score before correction and the corrected correction value.

[0094] (Note 7) The information processing device according to any one of the appendices 1 to 6, wherein the score is the similarity between the text and the image.

[0095] (Note 8) The acquisition means acquires multiple images to be compared, The correction means outputs a predetermined number of images from the plurality of images, starting with the one with the highest corrected score, as images corresponding to the text, according to any one of the appendices 1 to 7.

[0096] (Note 9) A method of information processing performed by a computer, Obtain the text and the image to be compared, A score indicating the degree of consistency between the text and the image is calculated. Based on the aforementioned image, the correction value of the score is calculated using the correction model. The score is corrected using the correction value, and the corrected score is output. An information processing method for learning the correction model to optimize the correction value.

[0097] (Note 10) Obtain the text and the image to be compared, A score indicating the degree of consistency between the text and the image is calculated. Based on the aforementioned image, the correction value of the score is calculated using the correction model. The score is corrected using the correction value, and the corrected score is output. A program that causes a computer to perform a process of learning the correction model in order to optimize the correction value.

[0098] Furthermore, some or all of the configurations described in Appendices 2 to 8, which are subordinate to Appendice 1 above, may also be subordinate to Appendices 9 and 10 in the same way as those described in Appendices 2 to 8. Moreover, not limited to Appendices 1, 9, and 10, some or all of the configurations described as appendices may also be subordinate to various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above.

[0099] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications to the structure and details of the present disclosure can be understood by those skilled in the art within the scope of the present disclosure. [Explanation of symbols]

[0100] 1. Image search system 2 Image DB 3. Image search device 11 processors 111 encoder 112 Score Calculation Unit 113, 123 Correction Value Calculation Unit 114, 124 Score Correction Section 115, 125, 135 Correction Model Learning Unit 136 Prediction Error Absorption Unit 100 Information Processing Devices 200 Output section

Claims

1. A means for obtaining text and an image to be compared, A score calculation means for calculating a score indicating the degree of consistency between the text and the image, A correction value calculation means that calculates a correction value of the score using a correction model based on the aforementioned image, A correction means that corrects the score using the correction value and outputs the corrected score, A learning means for learning the correction model to optimize the correction value, An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, wherein the learning means learns the correction model using learning data which includes text, images, and correct labels indicating the degree of consistency between the text and the images, so as to minimize the error between the corrected score and the correct labels.

3. The information processing apparatus according to claim 1, wherein the learning means uses pairs of text and images consistent with the text as training data to train the correction model so as to minimize the error between the score before correction and the correction value.

4. The information processing apparatus according to claim 3, further comprising a correction value correction means for correcting the correction value based on the characteristics of the text.

5. The information processing apparatus according to claim 4, wherein the correction means corrects the score using the corrected correction value.

6. The information processing apparatus according to claim 4, wherein the learning means uses pairs of text and images consistent with the text as learning data to learn the correction model so as to minimize a first error between the score before correction and the correction value, and a second error between the score before correction and the corrected correction value.

7. The information processing apparatus according to claim 1, wherein the score is the similarity between the text and the image.

8. The acquisition means acquires multiple images to be compared, The information processing apparatus according to claim 1, wherein the correction means outputs a predetermined number of images from the plurality of images, starting with the one with the highest corrected score, as images corresponding to the text.

9. A method of information processing performed by a computer, Obtain the text and the image to be compared, A score indicating the degree of consistency between the text and the image is calculated. Based on the aforementioned image, the correction value of the score is calculated using the correction model. The score is corrected using the correction value, and the corrected score is output. An information processing method for learning the correction model to optimize the correction value.

10. Obtain the text and the image to be compared, A score indicating the degree of consistency between the text and the image is calculated. Based on the aforementioned image, the correction value of the score is calculated using the correction model. The score is corrected using the correction value, and the corrected score is output. A program that causes a computer to perform a process of learning the correction model in order to optimize the correction value.

Images