Image processing device, image processing method, and program

Abstract

The purpose of the present disclosure is to extract a multimodal feature amount.　To this end, an image processing device according to the present disclosure is for extracting a feature amount of image data and comprises: an image understanding unit 41 for vectorizing an image pattern in image data and extracting an image feature amount; a text understanding unit 43 for vectorizing a text pattern in accompanying text data that accompanies the image data and extracting a text feature amount; and a feature amount combining unit 44 for generating a combined feature amount as a feature amount by projecting the image feature amount extracted by the image understanding unit 41 and the text feature amount extracted by the text understanding unit 43 onto the same vector space and combining the image feature amount and the text feature amount.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In recent years, the accuracy of image classification has improved due to the advancement of AI (Artificial Intelligence). Image classification is, for example, classifying an image (medium) as to whether the image or a specific object in the image is a pigeon or a swallow.

[0003] Conventionally, a technique has been proposed for improving the accuracy of image classification by using not only comparison of features between image data but also comparison results between text data that accompanies the image data or that is input by a user (see Non-Patent Document 1). In this case, for example, an image of a pigeon and text data that is a sentence describing the pigeon in the image are used. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Shaping Visual Representations with Language for Few-Shot Classification Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, the comparison of features between image data and the comparison of features between text data are performed independently, and therefore a problem occurs in that multimodal features cannot be extracted.

[0006] The present invention has been made in consideration of the above points, and has an object to extract features that are more multimodal than conventional techniques. [Means for solving the problem]

[0007] In order to solve the above problem, the invention of claim 1 is an image processing device that extracts features of image data, comprising: an image understanding unit that vectorizes an image pattern of the image data to extract image features; a text understanding unit that vectorizes a text pattern of associated text data accompanying the image data to extract text features; and a feature mixing unit that projects the image features extracted by the image understanding unit and the text features extracted by the text understanding unit into the same vector space and mixes the image features and the text features to generate mixed features as the features. Effect of the Invention

[0008] As described above, the present invention has an effect of being able to extract multimodal features compared to the conventional art. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic diagram of a communication system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a hardware configuration diagram of an image classification device and a communication terminal. [Diagram 3] 1 is a functional configuration diagram of an image classification device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a detailed functional configuration diagram of a feature extraction unit in the image classification device. [Diagram 5] FIG. 2 is a detailed functional configuration diagram of a text generation unit in the feature extraction unit. [Figure 6] 1 is a flowchart showing the processing performed by the image classification device in the training (learning) phase. [Figure 7]13 is a flowchart showing detailed processing executed by a feature extraction unit. [Figure 8] 13 is a flowchart showing a process performed by the image classification device in the inference phase. [Figure 9] FIG. 9 shows the experimental results. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [System configuration of the embodiment] First, an outline of the configuration of a communication system 1 according to the present embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of a communication system according to an embodiment of the present invention.

[0012] 1, a communication system 1 of this embodiment is constructed by an image classification device 3 and a communication terminal 5. The communication terminal 5 is managed and used by a user Y.

[0013] Furthermore, the image classification device 3 and the communication terminal 5 can communicate with each other via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

[0014] The image classification device 3 is composed of one or more computers. When the image classification device 3 is composed of multiple computers, it may be referred to as an "image classification device" or an "image classification system."

[0015] The image classification device 3 is a device that performs image classification using AI (Artificial Intelligence). For example, this image classification is classifying an image (medium) into a class as to whether the image or a specific object in the image is a pigeon or a swallow. The image classification device 3 then outputs classification result data that is the result of the image classification. Examples of output methods include transmitting the classification result data to the communication terminal 5, thereby displaying or printing a graph or the like related to the classification result data on the communication terminal 5 side, displaying the graph or the like on a display connected to the image classification device 3, and printing the graph or the like on a printer or the like connected to the image classification device 3.

[0016] The communication terminal 5 is a computer, and in Fig. 1, a notebook computer is shown as an example, but is not limited to a notebook type, and may be a desktop computer. The communication terminal may also be a smartphone or a tablet terminal. In Fig. 1, a user Y operates the communication terminal 5.

[0017] [Hardware configuration of image classification device and communication terminal] Next, the hardware configuration of the image classification device 3 and the communication terminal 5 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the hardware configuration of the image classification device and the communication terminal.

[0018] 2, the image classification device 3 includes a processor 301, a memory 302, an auxiliary storage device 303, a connection device 304, a communication device 305, and a drive device 306. The hardware components constituting the image classification device 3 are connected to each other via a bus 307.

[0019] The processor 301 serves as a control unit that controls the entire image classification device 3, and includes various arithmetic devices such as a CPU (Central Processing Unit). The processor 301 reads various programs onto the memory 302 and executes them. The processor 301 may include a GPGPU (General-purpose computing on graphics processing units).

[0020] The memory 302 has a main storage device such as a read only memory (ROM) and a random access memory (RAM). The processor 301 and the memory 302 form a so-called computer, and the processor 301 executes various programs read onto the memory 302, causing the computer to realize various functions.

[0021] The auxiliary storage device 303 stores various programs and various information used when the various programs are executed by the processor 301 .

[0022] The connection device 304 is a connection device that connects an external device (eg, a display device 310, an operation device 311) and the image classification device 3.

[0023] The communication unit 305 is a communication device for transmitting and receiving various types of information to and from other units.

[0024] The drive device 306 is a device for setting the recording medium 330. The recording medium 330 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM (Compact Disc Read-Only Memory), a flexible disk, a magneto-optical disk, etc. The recording medium 330 may also include semiconductor memories that record information electrically, such as a ROM (Read Only Memory), a flash memory, etc.

[0025] The various programs to be installed in the auxiliary storage device 303 are installed, for example, by setting the distributed recording medium 330 in the drive device 306 and reading out the various programs recorded in the recording medium 330 by the drive device 306. Alternatively, the various programs to be installed in the auxiliary storage device 303 may be installed by being downloaded from a network via the communication device 305.

[0026] FIG. 2 also shows the hardware configuration of the communication terminal 5. However, the only difference is that the reference numerals have changed from the 300s to the 500s, and the configuration is the same, so the description thereof will be omitted.

[0027] [Functional configuration of the image classification device] Next, the functional configuration of the image classification device will be described with reference to Fig. 3. Fig. 3 is a diagram showing the functional configuration of the image classification device according to the embodiment of the present invention.

[0028] 3, the image classification device 3 includes an input unit 30, a readout unit 31, a selection unit 32, a feature extraction unit 33, a similarity calculation unit 34, a loss calculation unit 35, a parameter update unit 36, and an output unit 39. Each of these units is a function realized by an instruction from the processor 301 in FIG. 2 based on a program.

[0029] Furthermore, learning models A and B are stored in the memory 302 or the auxiliary storage device 303 in FIG. 2. Learning model A is constructed by a large number of image similarity parameters described later. Learning model B is constructed by a large number of text generation probability parameters described later. Furthermore, a large number of image data, which are candidates for support data as teacher data, are stored in the memory 302 or the auxiliary storage device 303 in FIG. 2. Furthermore, each of the large number of image data is accompanied by text data indicating the contents of the image. That is, one pair of support data is composed of image data and accompanying text data, and a large number of pairs of support data are stored in the memory 302 or the auxiliary storage device 303 in FIG. 2. For example, one pair of support data includes image data of a pigeon and text data that is a sentence that accompanies this image data and describes the pigeon that appears in the image. Hereinafter, the text data that accompanies this image data will be referred to as "accompanying text data." Note that "accompanying" includes cases where text data is added to image data and cases where text data and image data are input or output separately and associated with each other. Note that there are also cases where text data accompanying image data is generated by the image classification device 3 based on the image data (generated text data) and added to this image data.

[0030] The input unit 30 inputs image data, which is query data as classification target (evaluation target) data for training or inference. For example, the input unit 30 inputs query data transmitted from the communication terminal 5 by the user Y to the image classification device 3 to the image classification device. The image data, which is the query data, is accompanied by accompanying text data. That is, one pair of query data is composed of the image data and the accompanying text data. Note that, in the training phase, the accompanying text data is always accompanied, but in the inference phase, the accompanying text data may not be accompanied. As a method of accompanying the accompanying text data, the image data may be captioned or may be manually input by the user Y. In many machine learning models, humans cannot intervene in the inference of image classification, but by enabling the user Y to input text data, the user Y can intervene in the inference of image classification.

[0031] The reading unit 31 reads out a candidate group of support data (M types, and j pairs for each type) to be compared with the query data from the memory 302 or the auxiliary storage device 303 in Fig. 2. For example, M is 100 and j is 60. In this case, a total of 6000 pairs are read out. Note that M being 100 and j being 60 is just an example, and M may be greater than or less than 100, and j may be greater than or less than 60.

[0032] The selection unit 32 randomly selects N types of k pairs of support data from the candidate group of support data to be compared with the query data. Here, the following description will be given assuming that N is 5 types and k is 1 pair (total of 5 pairs) of support data is randomly selected. This method of selecting 1 pair of support data for 5 types is a commonly performed process, but the selection unit 32 does not necessarily need to select 1 pair of support data for 5 types. For example, 2 pairs of 10 types (total of 20 pairs) may be used. In addition to the image data and the accompanying text data, the training support data is provided with information indicating the type of subject (also called "class") appearing in the image of the image data. For example, if the image is an image of a bird, the class indicates the type of bird, such as "pigeon," "hawk," or "swallow."

[0033] The feature extraction unit 33 extracts image features from the image data in one pair, and also extracts text features from the text data in the same pair. Furthermore, the feature extraction unit 33 generates a mixed feature by mixing the image features and the text features. The feature extraction unit 33 also generates text data from the image features. Hereinafter, the text data generated from the image features will be referred to as "generated text data." In other words, the generated text data is text data derived from an image, and is a different type from the associated text data derived from text.

[0034] <Feature extraction section> Here, the feature extraction unit in the image classification device will be described in detail with reference to FIG. 4. FIG. 4 is a detailed functional configuration diagram of the feature extraction unit in the image classification device.

[0035] 4, the feature extraction unit 33 has an image understanding unit 41, a text generation unit 42, a text understanding unit 43, and a feature amount mixing unit 44. Any neural network can be used for the image understanding unit 41, the text generation unit 42, the feature amount mixing unit 44, and the similarity calculation unit 34. For example, a four-layer CNN (Convolutional Neural Network) is used for the image understanding unit 41. By performing pre-learning for the text generation unit 42 and the text understanding unit 43, the text generation ability and the text understanding ability are improved.

[0036] Among these, the image understanding unit 41 acquires image data (an example of first image data) of the query data from the input unit 30, and also acquires image data (an example of second image data) of a specific pair of support data of one pair of five types from the selection unit 32. The image understanding unit 41 then vectorizes the image pattern of the image data of the query data to extract image features for the query, and also vectorizes the image pattern of the image data of the support data to extract image features for the support. The image features are vectors, and the text generation unit 42 can use any neural network, and generally uses an RNN (Recurrent neural network) or Transformer that uses the image features as initial values.

[0037] The text generation unit 42 generates generated text data for an image-derived query by projecting the image features for the query extracted by the image understanding unit 41 onto the vector space of the text data and decoding it, and also generates generated text data for image-derived support by projecting the supporting image features extracted by the image understanding unit 41 onto the vector space of the text data and decoding it.

[0038] (Text Generation) The text generator 42 will now be described in further detail with reference to Fig. 5. Fig. 5 is a detailed functional block diagram of the text generator.

[0039] 5, the text generator 42 has a linear transformation layer 421 and a decoder 422. Furthermore, the linear transformation layer 421 holds linear transformation layer parameters 421p, and the decoder 422 holds decoder parameters 422p. The linear transformation layer parameters 421p and the decoder parameters 422p are included in the learning model B shown in FIG.

[0040] The linear transformation layer 421 uses the linear transformation layer parameters 421p to project the image features acquired from the image understanding unit 41 onto the vector space of the associated text data, thereby extracting features derived from the image.

[0041] The decoder 422 uses the decoder parameters 422p to generate generated text data derived from the image from the feature amount acquired from the linear transformation layer 421.

[0042] Here, by using an existing pre-trained language model for the text generation unit 42 and the text understanding unit 43, it can be considered that pre-training has been performed for the text generation unit 42 and the text understanding unit 43. However, an existing language model cannot be used as is for the text generation unit 42. This is because existing language models capable of generating text have an Encoder-Decoder type structure.

[0043] A language model having an Encoder-Decoder type structure is disclosed in, for example, Reference 1. <Reference 1>Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer The Encoder-Decoder type structure refers to a structure in which text is first given as input, converted into features by an Encoder, the features are input to a Decoder, and the Decoder generates text. In this embodiment, since image features are input to the text generation unit 62, the Encoder of the existing language model in Reference 1 is not used, and instead, an arbitrary neural network such as a linear transformation layer is added before the Decoder. With this configuration, it becomes possible to convert image features into features suitable for the language model, input them to the Decoder, and generate text.

[0044] 4, the text understanding unit 43 acquires the associated text data of the query data from the input unit 30, and also acquires the associated text data of a specific pair of support data out of the five types of pairs from the selection unit 32. Then, the text understanding unit 43 vectorizes the text pattern of the associated text data of the query data to extract text features for the query, and also vectorizes the text pattern of the associated text data of the support data to extract text features for support.

[0045] For example, the text understanding unit 43 converts the text data into vectors using an existing language model such as Bidirectional Encoder Representations from Transformers (BERT).

[0046] As described above, while the image data is accompanied by the accompanying text data in the training phase, there are cases where the image data is not accompanied by the accompanying text data in the inference phase. In such cases, the text understanding unit 43 extracts the feature amount of the text data for the query, which is derived from the image, by treating (considering) the generated text data for the query derived from the image generated by the text generating unit 42 as the accompanying text data.

[0047] Next, the feature mixing unit 44 projects the image feature for query extracted by the image understanding unit 41 and the text feature for query extracted by the text understanding unit 43 onto the same vector space, and mixes the image feature for query and the text feature for query, thereby generating a mixed feature as a feature for query. Similarly, the feature mixing unit 44 projects the support image feature extracted by the image understanding unit 41 and the support text feature extracted by the text understanding unit 43 onto the same vector space, and mixes the support image feature and the support text feature, thereby generating a mixed feature as a feature for support. Note that the process of mixing the image feature and the text feature may project one feature onto the vector space of the other feature, or may project the two features onto a third vector space different from each other.

[0048] For example, the feature mixing unit 44 can reflect both image features and text features in the similarity calculation. The feature mixing unit 44 can use any neural network that accepts both image features and text features as inputs.

[0049] Here, the feature mixer 44 will be described in more detail.

[0050] The following model is used as the feature mixing unit 44. The image feature is ximage, and the text feature output by the text understanding unit 43 is x Lang Let MLP (Multilayer Perceptron) be a three-layer neural network. Let Linear be a linear transformation layer to two dimensions. Let [;] be an operation to vertically connect vectors. In this case, the vector h output by the feature mixing unit 44 is expressed as follows by (Equation 1), (Equation 2), and (Equation 3).

[0051]

number

[0052]

number

[0053]

number

[0054] Next, the feature mixing unit 44 calculates the importance of the image feature and the text feature by λ using (Equation 2). image and λ Lang It is dynamically determined by λ image ,λ Lang The softmax operation ensures that λ is a non-negative number that sums to 1. For example, if the original resolution of the image data is low (if the target object in the image is extremely small and blurry), λ is sorted to give more weight to the classification results when the accompanying text data is added to the image data. image and λ Lang is dynamically determined. Also, the user can set λ image and λ Lang It is also possible to manually change the degree to which the user-entered text is reflected in the classification results by adjusting in the range from 0 to 1. Note that Linear is an operation in which a weight matrix is ​​multiplied from the left and a bias vector is added. The weight matrix and bias vector during the Linear operation are included in the image similarity parameters of learning model A and the text generation probability parameters of learning model B.

[0055] Finally, the feature mixing unit 44 uses (Equation 3) to determine the feature to be output by a weighted sum according to the importance.

[0056] 4, the image similarity parameters of learning model A are used when the image understanding unit 41, the text understanding unit 43, and the feature mixing unit 44 execute each process. The text generation probability parameters of learning model B are used when the image understanding unit 41 and the text generation unit 42 execute each process. However, in the case of the inference phase, when the image data is accompanied by accompanying text data, the text generation probability parameters of learning model B are not used.

[0057] In the training phase, even when image data is accompanied by accompanying text data, the text generation probability parameters of the learning model B are used and updated through training (learning). This is done so that the text generator 42 can generate generated text data even when image data is not accompanied by accompanying text data in the inference phase. In addition, training (learning) the learning model B has a positive effect of improving the understanding ability of the image comprehension unit 41 that uses the text generation probability parameters.

[0058] 3, the similarity calculation unit 34 compares the query mixed feature with the support mixed feature to calculate the image similarity. In the inference phase, the image similarity is output to the output unit 39 and used as the classification result data for image classification. On the other hand, in the training phase, the image similarity is output to the loss calculation unit 35.

[0059] For example, the similarity calculation unit 34 is a bilinear layer. Here, consider N-way k-shot image classification. In the similarity calculation unit 34, k support features (vectors) are first given for each class. The vector obtained by averaging these is set as the class feature. X is a matrix in which N class features (vectors) are arranged. y is the feature of the query data, and W is a learnable parameter. In this case, the classification score for each class of the query data is expressed as follows:

[0060]

number

[0061] The loss calculation unit 35 calculates a loss function value from the image similarity. In addition, the loss calculation unit 35 calculates a loss function value from generated text data of the query data and support data, a generation probability distribution of the query data and support data, and accompanying text data of the query data and support data.

[0062] For example, the loss function calculated by the loss calculation unit 35 can use the classification score of the similarity calculation unit 34 or any loss related to text generation. Typically, a cross-entropy loss or a negative log-likelihood function is used.

[0063] The parameter update unit 36 ​​updates the image similarity parameters of the learning model A of the neural network that constitutes the feature extraction unit 33 and the similarity calculation unit 34, based on the loss function value calculated by the loss calculation unit 35 from the image similarity calculated by the similarity calculation unit 34. In this case, the loss calculation unit 35 performs learning so that the similarity between the image data of the support data and the image data of the query data is reduced, and further so that the similarity with the incorrect image is increased.

[0064] Furthermore, the parameter update unit 36 ​​updates the text generation probability parameter of the learning model B of the neural network that constitutes the feature extraction unit 33 and the similarity calculation unit 34, based on the loss function value calculated by the loss calculation unit 35. In this case, the loss calculation unit 35 performs learning so as to increase the probability that the generated text data is similar to the associated text data.

[0065] For example, the parameter update unit 56 calculates the gradient of the loss based on the loss calculated by the loss calculation unit 35, and updates the parameters.

[0066] [Processing or Operation of the Embodiment] Next, the processing or operation of this embodiment will be described in detail with reference to Fig. 6 to Fig. 8. The description will be divided into a training (learning) phase and an inference phase.

[0067] <Training Phase> First, the training phase will be described with reference to Figures 6 and 7. Figure 6 is a flowchart showing the processing executed by the image classification device in the training (learning) phase.

[0068] First, the input unit 30 inputs teacher data (query data) for training (S10). The readout unit 31 reads out a candidate group of teacher data (support data) for training (S11). The selector 32 randomly selects five types of support data (one pair (image data and associated text data)) as teacher data from the candidate group (S12). The selector 32 also selects any number of pairs from the same five types as query data. At this time, the selector 32 defines support data of the same type as a correct answer for each selected query data, and defines support data of a different type as an incorrect answer for the query data, thereby adding data defining a correct answer or an incorrect answer to the support data. For example, when the query data indicates "pigeon", of the five types of support data, the support data indicating "pigeon" is defined as a correct answer, and the support data indicating other types (classes) is defined as an incorrect answer. The definition of a correct answer or an incorrect answer may be performed by the readout unit 31.

[0069] Next, the feature extraction unit 33 generates a mixed feature for a query based on the query data acquired from the input unit 30, and generates a mixed feature for support based on a predetermined one of the five types and one pair of support data (a total of five pairs) selected by the selection unit 32 (S13). At this time, the feature extraction unit 33 receives set data in which correct answers or incorrect answers have been defined (query data, support data, and data defining correct answers or incorrect answers), calculates mixed features of the query data and support data included in this set data, and outputs them to the similarity calculation unit. At this time, if the number k of pairs of support data is 2 or more, a vector obtained by averaging the image features of the image data of each pair may be set as the image feature of that support data.

[0070] Here, the detailed process executed by the feature extraction unit will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the detailed process executed by the feature extraction unit.

[0071] As shown in FIG. 7, the image understanding unit 41 extracts each image feature (query image feature, support image feature) based on each image data of the query data and the support data (S131). The text generation unit 42 generates each generated text data based on each image feature (S132). In the training phase, steps S133 and S135 described below are not executed, and the text understanding unit 43 subsequently extracts each text feature (query text feature, text text feature) based on each associated text data of the query data and the support data (S134). The feature mixing unit 44 mixes the query image feature and the query text feature to generate a query mixed feature, and mixes the support image feature and the support text feature to generate a support mixed feature (S136).

[0072] 6, the similarity calculation unit 34 compares the mixed feature for query (an example of the first mixed feature) with the mixed feature for support (an example of the second mixed feature) to calculate the image similarity (S14). At this time, the similarity calculation unit 34 calculates the similarity of each pair of query data and support data included in the set data, and passes it to the loss calculation unit.

[0073] Then, the feature extraction unit 33 judges whether or not the calculation of similarity for all five pairs of the five types and one pair (total of five pairs) of support data selected by the selection unit 32 has been completed (S15). If the feature extraction unit 33 judges that the calculation of similarity for all five pairs of support data has not been completed (S15; NO), the process returns to step S13, and steps S13 and onward are performed for the support data for which the calculation of similarity has not been completed. Note that, for the query data acquired from the input unit 30, the mixed feature has already been generated, so reprocessing from step S13 onward is not performed.

[0074] On the other hand, in step S15, if the feature extraction unit 33 determines that the calculation of similarity for all five pairs has been completed (S15; YES), the loss calculation unit 35 calculates the loss (S16). At this time, the loss calculation unit 35 calculates the loss based on each similarity of the pairs of query data and support data included in each set data, and the definition data of correct or incorrect answer for each pair of support data for query data. Note that this similarity includes the similarity between images and the similarity between associated texts.

[0075] <Supplement A> In the training phase, when the accompanying text data in the teacher data can be used, the accompanying text data in the teacher data is input to the text understanding unit 43. However, in the inference phase, generated text data generated by the text generation unit 42 may be input, so a discrepancy occurs between the training phase and the inference phase. Therefore, in this embodiment, the cross-entropy loss L calculated from the image feature amount and the accompanying text data is class,gold In addition, the cross-entropy loss L calculated from the image feature and the generated text data generated by the text generation unit 42 is class,gen This process makes it possible to reduce the discrepancy between the training phase and the inference phase.

[0076] <Supplement B> In addition, the loss of contrastive learning shown in (Equation 4) L cntr By using this, the model can capture subtle differences in text data and acquire features.

[0077]

number

[0078]

number

[0079]

number

[0080]

number

[0081]

number

[0082] As described above, the loss function L is the sum of the four Ls mentioned above according to (Equation 5).

[0083]

number

[0084]

number

[0085] Next, the parameter update unit 36 ​​calculates the gradient of the loss, and updates (trains) the image similarity parameter of the learning model A and the text generation probability parameter of the learning model B (S17). At this time, the parameter update unit 36 ​​updates the parameters so as to minimize the loss.

[0086] Next, the selection unit 32 determines whether the selection has been performed a prescribed number of times (e.g., 20 times) (S18). For example, if the selection unit 32 performs selection 20 times as the prescribed number of times, 5 pairs of support data are selected in one selection, resulting in a total of 100 pairs of support data being selected. However, since the selection unit 32 randomly selects 5 types and 1 pair of support data (a total of 5 pairs) from the candidate group, there are cases in which the same support data is selected multiple times.

[0087] Then, in step S18 above, if the selection unit 32 determines that the specified number of selections has not been completed (S18; NO), the process returns to step S12, and the selection unit 32 randomly selects new support data of 5 types and 1 pair (5 pairs in total) from the candidate group, and then the process from step S13 onwards is performed.

[0088] On the other hand, in step S18, if the selection unit 32 determines that the specified number of selections have been completed (S18; YES), the process of the training phase shown in FIG. 6 ends.

[0089] <Inference Phase> Next, the training phase will be described with reference to Figures 7 and 8. Figure 8 is a flowchart showing the processing executed by the image classification device in the inference phase.

[0090] First, the input unit 30 inputs query data, which is data to be classified for inference (S30). The reading unit 31 reads support data for inference (S31).

[0091] Next, the feature extraction unit 33 generates a mixed feature for a query based on the query data, which is data to be classified, acquired from the input unit 30, and also generates a mixed feature for support based on a predetermined one of the five types and one pair of support data (a total of five pairs) selected by the selection unit 32 (S32). Here, the detailed processing executed by the feature extraction unit will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the detailed processing executed by the feature extraction unit.

[0092] 7, the image understanding unit 41 extracts image features (query image features, support image features) based on each image data of the query data and the support data (S131). The text generation unit 42 generates generated text data based on each image feature (S132). In the inference phase, steps S133 and S135 described below are executed.

[0093] <Supplement C> In the inference phase, the generated text data is generated using beam-search. Therefore, for example, if the beam width is set to 10, the top 10 tokens at each time can be selected as candidates for generation. However, beam-search requires a heavy amount of calculation, so it cannot be used in the training phase. Therefore, in order to suppress the discrepancy between the training phase and the inference phase, the text generation unit 42 performs a "language generation" process.

[0094] The "language generation" process is a process of generating generated text data by using a greedy method and random sampling in the training phase. In the greedy method, the text generation unit 42 selects the most significant token at each time and generates generated text data. In the random sampling, the text generation unit 42 generates generated text data by selecting tokens by sampling from a predetermined number of tokens (e.g., top 20) that are the most significant at each time. In addition, in the test, if no accompanying text data is provided from the input unit 30, the text generation unit 42 passes the generated text data to the text comprehension unit 43. At this time, the text generation unit 42 performs a beam search with a length penalty of 0.5 and a beam width of 5, generates generated text data with the maximum classification score for the query data being each class, and passes the generated text data to the text comprehension unit 43.

[0095] Therefore, the text understanding unit 43 judges whether the query data and the support data each contain accompanying text data, i.e., whether the image data of the query data and the image data of the support data each contain accompanying text data (S133). If the text understanding unit 43 judges that the query data and the support data each contain accompanying text data, i.e., the image data of the query data and the image data of the support data each contain accompanying text data (S133; YES), the text understanding unit 43 extracts each text feature (text feature for query, text feature for text) based on each accompanying text data of the query data and the support data (S134).

[0096] On the other hand, in step S133, if the text understanding unit 43 determines that neither the query data nor the support data contains accompanying text data, that is, if it determines that neither the image data of the query data nor the image data of the support data is accompanied by accompanying text data (S133; NO), the text understanding unit 43 performs the following processing.

[0097] That is, in the above case (S133; NO), when the support data does not include accompanying data, the text understanding unit 43 extracts text features based on the accompanying text of the query data, and extracts text features based on the generated text of the support data (S135). In a similar case (S133; NO), when the query data does not include accompanying data, the text understanding unit 43 extracts text features based on the accompanying text of the support data, and extracts text features based on the generated text of the query data (S135). In a similar case (S133; NO), when neither the query data nor the support data includes accompanying text data, the text understanding unit 43 extracts each text feature based on the generated text of the query data and the support data (S135).

[0098] <Supplement D> In addition, the text generation unit 42 uses L calculated by teacher-forcing and cross-entropy loss as the loss for generating the generated text data. text If learning is done only with the text generator 42, the purpose of learning is to "reproduce the accompanying text data in the training data" and it does not take into account "generating generated text data that contributes to image classification". This is because the gradient graph is destroyed by the discrete processing of the text generator 42. In other words, the loss (L class,gold ,L class,gen ) is not propagated to the text generation unit 42 by the error backpropagation method. To improve this, the text understanding unit 43 performs the following process.

[0099] That is, in this embodiment, the text understanding unit 43 first calculates x image is mapped (projected) into the same vector space as the text features.

[0100]

number

[0101]

number

[0102] Here, the sequence of hidden states in the final layer of the text understanding unit 43 is H BERT If the token is a stop word, wj=0, otherwise wj=1. If the associated text data is input by the input unit 30 instead of the text generation unit 42, pj=1 is set for all tokens.

[0103]

number

[0104] Then, after step S134 or S135, the feature mixing unit 44 mixes the query image features and the query text features to generate query mixed features, and mixes the supporting image features and the supporting text features to generate supporting mixed features (S136).

[0105] Next, returning to FIG. 8, the similarity calculation unit 34 compares the query mixed feature (an example of the first mixed feature) with the support mixed feature (an example of the second mixed feature) to calculate the image similarity (S33).

[0106] Next, the feature extraction unit 33 judges whether or not the comparison of all five pairs of support data among the five types and one pair (a total of five pairs) of support data selected by the selection unit 32 has been completed (S34). If the feature extraction unit 33 judges that the comparison of all five pairs of support data has not been completed (S35; NO), the process returns to step S32, and steps S32 and onward are performed on the support data among the five types and one pair (a total of five pairs) of support data for which comparison has not been completed. Note that, for the query data, which is the data to be classified acquired from the input unit 30, the mixed feature has already been generated, so reprocessing from step S32 onward is not performed.

[0107] On the other hand, in step S34, if the feature extraction unit 33 determines that comparison of all five pairs of support data has been completed (S34; YES), the output unit 39 outputs classification result data indicating the classification result based on the comparison results up to this point (S35). The classification result data indicates, for example, that the image related to the data to be classified is an image of a pigeon, that there is a 90% possibility that it is an image of a pigeon, and that there is a 10% possibility that it is an image of another bird.

[0108] [Experimental Results] Next, the experimental setup and results will be described.

[0109] <Experimental Setup> We used the Caltech-UCSD Birds (CUB) dataset (References 3, 4) as a 5-way 1-shot classification problem. This data contains 200 bird breed classes, with 40-60 images per breed. Of the 200 images, 100 are used for training, 50 for development, and 50 for testing. <Reference 3> Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. <References 4>Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine-grained visual descriptions. In CVPR, pp. 49-58, 2016. <Experimental Results> FIG. 9 is a diagram showing experimental results. In FIG. 9, the result of executing all the proposed methods according to this embodiment is the LIDE value in the first row of the first row. From there, the experimental results when the above-mentioned <Supplementary A>, <Supplementary B>, <Supplementary C>, and <Supplementary D> are not performed are shown in the first row of the second row, the second row, the fourth row of the third row, and the third row of the third row. In addition, the experimental results when all of the above-mentioned <Supplementary A>, <Supplementary B>, <Supplementary C>, and <Supplementary D> are not performed are shown in the fifth row of the second row. As a result, it was confirmed that all four elements of the proposed method according to <Supplementary A>, <Supplementary B>, <Supplementary C>, and <Supplementary D> contribute to performance improvement.

[0110] [Main Effects of the Embodiments] As described above, according to this embodiment, the image classification device 3 generates a mixed feature by mixing the image feature of image data and the text feature of associated text data accompanying the image data. This allows the image classification device 3, as a feature extraction device, to extract multimodal features compared to simple comparisons between features of image data and between text data. In addition, the image classification device 3 also provides the effect of being able to perform more accurate image classification by extracting features related to image data with higher accuracy.

[0111] Furthermore, according to this embodiment, the following process enables images to be classified with high accuracy, and has the effect of allowing the user to influence the classification results by inputting accompanying text data in the inference phase.

[0112] <Supplementary A> When text data is used to supplement information in a small number of case image classification tasks, the discrepancy in the text data available in the training phase and inference phase is reduced by using an image classification loss function in combination.

[0113] Furthermore, according to <Supplementary B>, the text understanding unit 43 learns through contrastive learning so as to output features that capture subtle differences in the text data.

[0114] Furthermore, <Supplementary C> reduces the discrepancy between text generation methods in the training and inference phases by generating generated text data through random sampling.

[0115] Furthermore, according to <Supplementary D>, the text generation unit 42 performs learning that takes into consideration the improvement of image classification performance by pooling using the text generation score (generation score for each token).

[0116] [Overall supplement] The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations).

[0117] The image classification device 3 can be realized by a computer and a program, and this program can be recorded on a (non-transitory) recording medium or provided via the communication network 100.

[0118] In the above embodiment, the image classification device 3 is shown, but when specialized for the feature extraction unit 33, it can be expressed as a feature extraction device. In addition, both the image classification device 3 and the feature extraction device can be expressed as an image processing device.

[0119] In addition to the above embodiment, any process used in neural network learning can be added to the above embodiment. For example, the number of data can be inflated by applying rule-based paraphrasing to the input accompanying text data. As an example of paraphrasing, "big" in "This bird is big" can be paraphrased to "This bird is large."

[0120] [Additional notes] The above-described embodiment can also be expressed as the following invention.

[0121] [Additional Note 1] An image processing device that extracts a feature amount of image data, an image understanding step of vectorizing an image pattern of the image data and extracting image features; A text understanding step of vectorizing a text pattern of accompanying text data accompanying the image data and extracting text features; a feature mixing step of projecting the image features extracted by the image understanding step and the text features extracted by the text understanding step onto the same vector space and mixing the image features and the text features to generate mixed features as the features; An image processing device that executes the above.

[0122] [Additional note 2] The image processing device according to claim 1, wherein the image understanding step, the text understanding step, and the feature mixing step are each realized by a neural network, and the image understanding step, the text understanding step, and the feature mixing step perform processing based on model parameters of the neural network.

[0123] [Additional note 3] Item 2: The image processing device according to claim 1, The processor, a similarity calculation step of calculating an image similarity between a first mixed feature related to the first image data generated by the feature mixing step and a second mixed feature related to the second image data generated by the feature mixing step; a parameter updating step of updating an image similarity parameter included in the model parameters based on the image similarity calculated by the similarity calculation step; An image processing device that executes the above.

[0124] [Additional Note 4] Item 2: The image processing device according to claim 1, The processor, a text generation step of generating generated text data by projecting the image feature quantity extracted by the image understanding step onto a vector space of the associated text data; a parameter updating step of updating a text generation probability parameter included in the model parameters based on the generated text data and the associated text data generated by the text generating step; An image processing device that executes the above.

[0125] [Additional Note 5] Item 1 is an image processing device according to the present invention; The processor, A text generation step is performed to generate generated text data by projecting the image feature quantity extracted by the image understanding step onto a vector space of the associated text data; An image processing device, comprising: an image comprehension step for extracting text features by treating the generated text data generated by the text generation step as the associated text data when the image data is not accompanied by the associated text data;

[0126] [Additional Note 6] The image processing device described in Appendix 4, wherein the parameter update step includes a process of updating the text generation probability parameter based on a loss based on the image features and the associated text data, and a loss based on the image features and the generated text data.

[0127] [Additional Note 7] The image processing device described in Appendix 4, wherein the parameter updating step includes a process of updating the text generation probability parameter so that the text features of the associated text data and the text features of the generated text data, which are related to image data of the same class, approach each other, and updating the text generation probability parameter so that the text features of the associated text data and the text features of the generated text data, which are related to image data of a different class, move apart.

[0128] [Additional Note 8] The image processing device described in appendix 4, wherein the text generation step includes a process of combining generation of the generated text data by random sampling from a predetermined number of tokens that are ranked top at each time point and generation of the generated text data under normal circumstances.

[0129] [Additional Note 9] The text understanding step includes a process of extracting the text feature by performing weighted pooling using a probability indicating a likelihood of a predetermined number of tokens related to the generated text data. 5. The image processing device according to claim 4.

[0130] [Additional Note 10] An image processing method executed by an image processing device that extracts a feature amount of image data, comprising: The image processing device includes: an image understanding step of vectorizing an image pattern of the image data and extracting image features; A text understanding step of vectorizing a text pattern of accompanying text data accompanying the image data and extracting text features; a feature mixing step of projecting the image features extracted by the image understanding step and the text features extracted by the text understanding step onto the same vector space and mixing the image features and the text features to generate mixed features as the features; An image processing method that performs

[0131] [Additional Note 11] A non-transitory recording medium having a program recorded thereon for causing a computer to execute the method described in appended claim 10.

[0132] 〔others〕 This patent application claims priority to international application PCT / JP2021 / 041801 filed on November 12, 2021, the entire contents of which are incorporated herein by reference. [Explanation of symbols]

[0133] 1. Communication Systems 3. Image classification device (an example of an image processing device) 5. Communication terminals 30 Input section 31 Reading section 32 Selection section 33 Feature Extraction Unit 34 Similarity calculation part 35 Loss calculation section 36 Parameter Update Section 39 Output section 41 Image Understanding Department 42 Text Generation Unit 43 Text Comprehension Department 44 Feature Mixing Unit 422 Decoder

Claims

1. An image processing device that extracts a feature amount of image data, an image understanding unit that vectorizes an image pattern of the image data and extracts image features; a text understanding unit that vectorizes a text pattern of accompanying text data that accompanies the image data and extracts text features; a feature mixing unit that projects the image features extracted by the image understanding unit and the text features extracted by the text understanding unit onto the same vector space and mixes the image features and the text features to generate a mixed feature as the feature; and An image processing device comprising:

2. 2. The image processing device according to claim 1, An image processing device having a similarity calculation unit that calculates image similarity between a first mixed feature related to first image data generated by the feature mixing unit and a second mixed feature related to second image data generated by the feature mixing unit.

3. 2. The image processing device according to claim 1, an image processing device comprising: a text generating unit that generates generated text data by projecting the image feature amount extracted by the image understanding unit onto a vector space of the associated text data;

4. 2. The image processing device according to claim 1, a text generation unit that generates generated text data by projecting the image feature quantity extracted by the image understanding unit onto a vector space of the associated text data; An image processing device in which, when the image data is not accompanied by the associated text data, the text understanding unit extracts the text features by using the generated text data generated by the text generation unit as the associated text data.

5. The image processing device according to claim 3 , wherein the text generating unit generates the generated text data by random sampling from a predetermined number of tokens that are top ranking at each time point, and generates the generated text data in a normal state in combination.

6. The text understanding unit extracts the text features by performing weighted pooling using a probability indicating a likelihood of a predetermined token related to the generated text data. The image processing device according to claim 3 .

7. The image processing device according to claim 2 ; a communication terminal that transmits the image data to the image processing device via a communication network and receives classification result data of the image based on the previous image similarity from the image processing device via the communication network; A communication system having the above configuration.

8. An image processing method executed by an image processing device that extracts a feature amount of image data, comprising: The image processing device includes: an image understanding step of vectorizing an image pattern of the image data and extracting image features; A text understanding step of vectorizing a text pattern of accompanying text data accompanying the image data and extracting text features; a feature mixing step of projecting the image features extracted by the image understanding step and the text features extracted by the text understanding step onto the same vector space and mixing the image features and the text features to generate mixed features as the features; An image processing method that performs

9. A program causing a computer to execute the method according to claim 8.

Images