A multi-modal composite coding image retrieval method and system

Abstract

The present invention provides a multi-modal composite coded image retrieval method and system, the method comprising: obtaining information to be retrieved, and determining the data type of the information to be retrieved; selecting coarse-grained retrieval or fine-grained retrieval according to the data type of the information to be retrieved, the fine-grained retrieval comprising when the input information to be retrieved includes both image and text description data, fusing the two features to obtain composite features, and then performing data retrieval through the composite features; the present invention can select coarse-grained retrieval or fine-grained retrieval according to the data type, can obtain composite features by fusing the two features, and then perform fine-grained retrieval through the composite features; the present invention can improve the efficiency of database construction, increase the flexibility of retrieval methods, reduce retrieval difficulty, and provide a solution for image retrieval that unifies accuracy and generalization performance, and is more flexible in switching between fuzzy retrieval and precise retrieval.

Description

Technical Field

[0001] The present invention relates to the field of computer applications, and in particular to a multi-modal composite coded image retrieval method and system. Background Art

[0002] Image retrieval technology has been widely used in various fields, such as product search, video content understanding, etc. However, due to the diversity of images and the complexity of image content, the research on efficient and accurate image retrieval methods has always been an important research topic in the field of machine vision.

[0003] At present, the technical process of image retrieval mainly consists of three steps: feature extraction, feature encoding and database indexing. According to the different features, image retrieval is generally divided into text-based retrieval and image content-based retrieval. These two technologies have also been widely used and studied. Text-based retrieval generally marks the attributes of each dimension of the image. During retrieval, only images with the same attributes can be matched. It has high accuracy but poor generalization performance. In addition, a large number of images need to be manually annotated in advance, which is time-consuming. Image content-based retrieval uses image semantics as clues for processing, and completes the extraction of low- and high-dimensional features of the image based on image processing technology, and matches them. It can retrieve images with the same or similar characteristics, has certain generalization performance, and requires less manual participation; however, due to the diversity and complexity of image features, there may be a large number of mismatches. For example, when there are images of the same category but not the same target in the image, it is difficult to distinguish. In addition, by annotating the image with text semantics in advance, the text description of the image content is completed. The application of annotation costs is high and the maintenance is difficult. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present invention provides a multi-modal composite coded image retrieval method and system to solve the above-mentioned technical problems.

[0005] The multimodal composite coded image retrieval method provided by the present invention comprises:

[0006] Obtaining information to be retrieved and determining the data type of the information to be retrieved;

[0007] Coarse-grained retrieval or fine-grained retrieval is selected according to the data type of the information to be retrieved. The fine-grained retrieval includes when the input information to be retrieved includes both image and text description data, by fusing the two features to obtain composite features, and then performing data retrieval through the composite features to obtain target data; the coarse-grained retrieval includes when the input information to be retrieved includes one of image or text description data, performing data retrieval according to the corresponding features to obtain target data.

[0008] In one embodiment of the present invention, a recognition model is pre-established and trained, wherein the recognition model includes a first model for image description generation, a second model for text processing, and a third model for multimodal feature extraction;

[0009] Processing the input image data through the first model to obtain a first output result, where the first output result includes an image feature code and a text description of the image content;

[0010] Processing the input text data and / or the text description of the image content output by the first model through the second model to obtain a second output result;

[0011] When the input information to be searched is a text description, the coarse-grained search is performed through the output result of the second model;

[0012] When the input information to be retrieved is an image, the coarse-grained retrieval is performed through the image feature coding in the first output result, or the text description of the image content in the first output result is input into the second model, and the fine-grained retrieval is performed through the image feature coding in the first output result and the second output result;

[0013] When an image and a text description are input simultaneously, the image is input into the first model, the text description is input into the second model, and the respective output results are simultaneously input into the third model, and the fine-grained retrieval is performed according to the output results of the third model.

[0014] In one embodiment of the present invention, the second output result includes a text encoding of text features. When the input text data and / or the text description of the image content output by the first model includes multiple sentences, the text encoding is a mixed encoding including global features and local features of the sentences.

[0015] In one embodiment of the present invention, processing the input text data and / or the text description of the image content output by the first model through the second model to obtain a text encoding containing text features includes:

[0016] Obtaining a sentence vector, and using the sentence vector as a sentence encoding part of the hybrid encoding;

[0017] Perform part-of-speech tagging on the multiple sentences and perform named entity recognition to obtain words and word vectors for search-related parts of speech, and use the word vectors as word encoding parts of mixed encoding;

[0018] Obtaining a mixed code including global features and local features of the sentence through the sentence coding part and the word coding part;

[0019] The sentence encoding part and the word encoding part are both used as retrieval features.

[0020] In one embodiment of the present invention, the step of fusing the two features to obtain a composite feature includes:

[0021] According to the output results of the first model and the second model, the image features and the text features are spliced ​​to obtain the feature matrix and complete the preliminary fusion;

[0022] Performing attention mechanism processing on the feature matrix through a special convolution to obtain a new feature matrix;

[0023] The final fusion is completed by performing channel fusion on the new feature matrix and the original feature matrix.

[0024] In one embodiment of the present invention, the feature matrix is ​​processed by an attention mechanism through a special convolution to obtain a new feature matrix that fuses the graph vector with all text features, and the length of the special convolution is the vector length of the feature matrix;

[0025] The new feature matrix is ​​channel-fused with the original feature matrix to form a 2-channel feature map, completing the final fusion.

[0026] In one embodiment of the present invention, the preliminary fusion includes:

[0027] Get the graph vector, all the sentence vectors corresponding to the graph vector, and the word vectors corresponding to all the sentences;

[0028] All word vectors in the image are concatenated in sequence according to the order of the image vector, the first sentence vector corresponding to the image, and the first word vector corresponding to the first sentence to obtain the feature matrix containing the image features and the text features.

[0029] The present invention also provides a multi-modal composite coded image retrieval system, comprising:

[0030] The acquisition module is used to obtain the information to be retrieved and determine the data type of the information to be retrieved;

[0031] A processing module is used to select coarse-grained retrieval or fine-grained retrieval according to the data type of the information to be retrieved. The fine-grained retrieval includes when the input information to be retrieved includes both image and text description data, by fusing the two features to obtain a composite feature, and then performing data retrieval through the composite feature to obtain target data; the coarse-grained retrieval includes when the input information to be retrieved includes one of image or text description data, performing data retrieval according to the corresponding feature to obtain target data.

[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any of the above methods when executed by a processor.

[0033] The present invention also provides an electronic terminal, comprising: a processor and a memory;

[0034] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods described above.

[0035] The beneficial effects of the present invention are as follows: the multimodal composite coded image retrieval method and system of the present invention can select coarse-grained retrieval or fine-grained retrieval according to the data type, can obtain composite features by fusing two features, and then perform fine-grained retrieval of data through the composite features; the present invention can improve the efficiency of database construction, increase the flexibility of retrieval methods, reduce retrieval difficulty, and provide a solution for image retrieval that unifies accuracy and generalization performance, and is more flexible in switching between fuzzy retrieval and precise retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of a multi-modal composite coded image retrieval method in an embodiment of the present invention.

[0037] Figure 2 It is a flowchart of sentence encoding of the multimodal composite coding image retrieval method in an embodiment of the present invention.

[0038] Figure 3 It is a flowchart of word encoding in the multimodal composite encoding image retrieval method in an embodiment of the present invention.

[0039] Figure 4 It is a schematic diagram of the multimodal feature conformance encoding process of the multimodal composite encoding image retrieval method in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0041] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0042] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0043] like Figure 1 As shown, the multimodal composite coded image retrieval method in this embodiment includes:

[0044] S1. Obtain the information to be retrieved and determine the data type of the information to be retrieved;

[0045] S2. According to the data type of the information to be retrieved, coarse-grained retrieval or fine-grained retrieval is selected. The fine-grained retrieval includes when the input information to be retrieved includes both image and text description data, by fusing the two features to obtain composite features, and then performing data retrieval through the composite features to obtain target data; the coarse-grained retrieval includes when the input information to be retrieved includes one of image or text description data, performing data retrieval according to the corresponding features to obtain target data.

[0046] In this embodiment, the information to be retrieved is first obtained, and the data type of the information to be retrieved is determined. The data type includes images and texts. In this embodiment, multimodal composite coding is performed based on image processing technology and natural language processing technology. It can provide richer and multi-dimensional feature information for back-end image retrieval, and can automatically extract text information from image content, solving the requirements of image retrieval on annotation difficulty and workload, and can simultaneously meet the two retrieval methods of fuzzy retrieval and precise retrieval, reducing the difficulty of building an image database and improving image retrieval efficiency.

[0047] In this embodiment, when image and text description data are input at the same time, the image description generation algorithm is used to extract image features and text description for the input image data, and the text processing algorithm is used to extract text features for the input text data. Then the image features and text features are input to complete the extraction of composite features that integrate the two features, and the input features are searched in the database to obtain the target data. When only image data is input, the image description generation algorithm is used to extract image features and obtain text description, and the text description is output and then the text processing algorithm is used to extract text features, and the input features are searched in the database to obtain the target data. When only text description data is input, the text processing algorithm is directly used to extract text features, and the input features are searched in the database to obtain the target data. In this way, the retrieval of image data and text data, image and text data input at the same time can be satisfied. The fusion of image and text features increases the feature dimension, enriches the feature information, and can improve the matching accuracy. The combination of multiple features satisfies both fuzzy search and precise search. At the same time, the text description is directly generated by the image description generation algorithm, which can reduce the degree of manual participation, reduce costs, and improve efficiency.

[0048] In this embodiment, a recognition model is pre-established and trained, and the recognition model includes a first model for generating image descriptions, a second model for text processing, and a third model for extracting multimodal features; the input image data is processed by the first model to obtain a first output result, and the first output result includes an image feature code and a text description of the image content; the input text data and / or the text description of the image content output by the first model is processed by the second model to obtain a second output result; when the input information to be retrieved is a text description, a coarse-grained search is performed through the output result of the second model; when the input information to be retrieved is an image, the coarse-grained search is performed through the image feature code in the first output result, or the text description of the image content in the first output result is input into the second model, and a fine-grained search is performed through the image feature code in the first output result and the second output result; when an image and a text description are input at the same time, the image is input into the first model, the text description is input into the second model, and then the respective output results are simultaneously input into the third model, and the fine-grained search is performed according to the output result of the third model.

[0049] In this embodiment, the first model is an image description generation algorithm module, and its main function is to process the input image data, obtain low-dimensional image encoding and text description of the image content. The image description generation (Image Caption) in this embodiment can adopt mature solutions in the prior art, such as the model of the NIC (Neural Image Caption) structure class of the encoder-decoder framework, regard the image description generation as a translation task, extract features of the image through CNN, obtain image features, and use the CNN output results as the input of RNN, and introduce the attention (attention mechanism) structure to generate description text, and finally output image feature encoding and text description at the same time as the input of subsequent modules. The first model in this embodiment can be based on the existing model and user private data for transfer learning, and construct a model that conforms to its own data set, which will not be repeated here. In the database construction stage, the image feature encoding and text description generated by the first model are both persisted in the database as the attribute encoding of the image as a retrieval feature.

[0050] In this embodiment, the second model is a text processing algorithm module, and its main function is to process the text description generated by the image description generation algorithm or the manually input text description, and obtain a mixed code containing the global features and local features of the sentence. Since the text description may contain multiple sentences, all sentences can be processed and encoded. First, the sentence vector can be obtained from the sentence through the pre-trained model as the sentence encoding part of the mixed encoding; then the sentence is tagged with parts of speech and named entities are identified to obtain words with parts of speech that are helpful for retrieval, such as nouns and special proper nouns, and their word vectors are obtained after deduplication. Through the processing of the text processing algorithm module, text feature information including word vectors and sentence vectors is obtained. Optionally, in this embodiment, the sentence vector can be obtained from the sentence through the Albert pre-trained model as the sentence encoding part of the mixed encoding, and then the sentence is tagged with parts of speech and identified with named entities using the named entity recognition practice (Albert+CRF). In the database construction stage, the sentence encoding and word encoding generated by the second model are both persisted in the database as retrieval features as attribute encodings of the image, such as Figure 2 , Figure 3 shown.

[0051] In this embodiment, the third model is a multimodal feature composite coding algorithm module. In order to complete the fusion of the image features and text features extracted by the first model and the second model, high-dimensional composite information containing global and local features of the image and text is obtained, and based on the multimodal feature composite coding, the fusion of several different dimensional features such as image features, sentence features and vocabulary features is realized, and higher dimensional attributes containing composite information are provided as retrieval attributes. In this embodiment, sentence vectors are obtained, and the sentence vectors are used as the sentence coding part of the mixed coding; part-of-speech tagging is performed on multiple sentences, and named entity recognition is performed to obtain words and word vectors for the parts of speech related to the retrieval, and the word vectors are used as the word coding part of the mixed coding; through the sentence coding part and the word coding part, a mixed coding containing global and local features of the sentence is obtained. According to the output results of the first model and the second model, the image features and text features are spliced ​​to obtain a feature matrix to complete the preliminary fusion; then the feature matrix is ​​processed by an attention mechanism through a special convolution to obtain a new feature matrix; the new feature matrix is ​​channel-fused with the original feature matrix to complete the final fusion. Specifically:

[0052] First, the image features generated by the image description generation module and the text features generated by the text processing module are spliced ​​and feature fusion is performed. The image vector, sentence 1 vector, sentence 1 vocabulary vector, sentence 2, etc. are spliced ​​in the order of image vector, sentence 1 vector, sentence 2, etc. to form a feature matrix containing image features and text features to achieve preliminary feature fusion. Then, the deep fusion of the two modal features is completed to obtain the final multimodal composite feature encoding for retrieval matching. Based on the fact that different text descriptions contain different image contents, in order to fuse image features with different content descriptions, a special convolution can be used before the conventional text-cnn convolution to complete the attention operation on the feature matrix, and then the text-cnn is used to complete the encoding. Optionally, the feature matrix dimension is w*h, where w is the width, i.e., the vector length, and h is the height, i.e., the number of features. The height of the special convolution is 2 and the length is w, in which the row of the image vector is fixedly involved in the calculation, i.e., the image vector is fused with all text features to obtain a new matrix of w*h. Then, the obtained new matrix is ​​channel-fused with the original feature matrix through Text-cnn to form a 2-channel feature map, which is input into the conventional text-cnn model to complete the feature encoding. Complete the fusion between image features and text features, and various dimensions within text features to obtain richer image information. Finally, since a large number of parameters need to be trained and acquired during feature extraction, the image similarity training method can be used to complete the calculation of feature extraction parameters. During actual retrieval, the training phase does not participate in the workflow. Through the feature extraction and fusion in the above embodiments, the final composite feature encoding completes multimodal fusion, extracts multi-dimensional information such as the image's optical information, content information, and content-related information, and provides more accurate image encoding. The process is as follows: Figure 4 shown.

[0053] In this embodiment, the final retrieval solution is provided by image retrieval based on multimodal composite coding. The first model and the second model can be trained based on the existing model and user data, and the third model needs to be trained based on user data and needs to complete the annotation of similar images. In actual retrieval, multiple retrieval methods can be provided according to different user inputs:

[0054] Fine-grained retrieval: input the image and text description at the same time, the present invention generates the above-mentioned multiple codes to complete accurate retrieval; only input the image, generate image and text codes, and complete accurate retrieval, but this method is slightly lower than inputting the image and text at the same time because the description is completed by the machine, and there may be mismatches.

[0055] Coarse-grained retrieval: ① Input only the image, generate the image feature code, and match only the image feature code; ② Input only the text, generate the text feature code, and complete the matching.

[0056] The following is a specific example for explanation:

[0057] Example 1

[0058] S101. Collection of image data

[0059] Collect image data based on user needs.

[0060] S102. Construction of image description generation model

[0061] For the collected image data, the existing image description generation model is used to generate text descriptions. Since the existing model is not trained for user data, the generated text description is manually modified, and transfer learning is performed on the model to complete the training of the image description generation model.

[0062] S103. Training of text processing models

[0063] For the text description generated in S102, the existing text processing model is also used to complete the generation of sentence vectors and word vectors, and manual modification and transfer learning are performed to complete the training of the text processing model.

[0064] S104. Training of multimodal composite coding model

[0065] The image description generation model, text processing model and image data collected by users are used to complete the training of the multimodal composite coding algorithm.

[0066] S105. Establishment of multi-modal composite coding image retrieval database

[0067] The image description generation model, text processing model and multimodal composite coding model generated by all the above steps are used to process the user image data, generate image features, sentence features, word features and composite coding features as image attributes and store them in the database.

[0068] S106. Image Retrieval

[0069] Construct retrieval interfaces with different inputs to complete the user's retrieval output of specified feature data.

[0070] Accordingly, this embodiment also provides a multi-modal composite coded image retrieval system, including:

[0071] The acquisition module is used to obtain the information to be retrieved and determine the data type of the information to be retrieved;

[0072] A processing module is used to select coarse-grained retrieval or fine-grained retrieval according to the data type of the information to be retrieved. The fine-grained retrieval includes when the input information to be retrieved includes both image and text description data, by fusing the two features to obtain a composite feature, and then performing data retrieval through the composite feature to obtain target data; the coarse-grained retrieval includes when the input information to be retrieved includes one of image or text description data, performing data retrieval according to the corresponding feature to obtain target data.

[0073] The multimodal composite coded image retrieval system in the present invention can automatically complete the understanding and generation of image content through the method in the above embodiment, reduce manual participation, meet more application scenarios, and reduce costs and increase efficiency.

[0074] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0075] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0076] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0077] The computer-readable storage medium in this embodiment can be understood by ordinary technicians in this field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0078] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes each step of the above method.

[0079] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0080] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0081] In the above embodiments, unless otherwise specified, the use of serial numbers such as "first", "second", etc. to describe common objects only indicates that they refer to different instances of the same object, rather than indicating that the objects being described must be in a given order, whether temporally, spatially, sequentially, or in any other way. In the above embodiments, the description's reference to "this embodiment", "one embodiment", "another embodiment", or "other embodiments" indicates that the specific features, structures, or characteristics described in conjunction with the embodiments are included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment", "one embodiment", "another embodiment" do not necessarily all refer to the same embodiment.

[0082] In the above-described embodiments, although the invention has been described in conjunction with specific embodiments of the invention, many substitutions, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed. Embodiments of the invention are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims.

[0083] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0084] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0085] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0086] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A multimodal composite coded image retrieval method, characterized in that: include: Obtaining information to be retrieved and determining the data type of the information to be retrieved; According to the data type of the information to be retrieved, coarse-grained retrieval or fine-grained retrieval is selected. The fine-grained retrieval includes when the input information to be retrieved includes both image and text description data, by fusing the two features to obtain a composite feature, and then performing data retrieval through the composite feature to obtain target data; the coarse-grained retrieval includes when the input information to be retrieved includes one of image or text description data, performing data retrieval according to the corresponding feature to obtain target data; wherein, a recognition model is pre-established and trained, and the recognition model includes a first model for image description generation, a second model for text processing, and a third model for multimodal feature extraction; the input image data is processed by the first model to obtain a first output result, and the first output result includes image feature encoding and image content. The method comprises the steps of: processing the input text data and / or the text description of the image content output by the first model through the second model to obtain a second output result; when the input information to be retrieved is a text description, performing the coarse-grained search through the output result of the second model; when the input information to be retrieved is an image, performing the coarse-grained search through the image feature coding in the first output result, or inputting the text description of the image content in the first output result into the second model, performing a fine-grained search through the image feature coding in the first output result and the second output result; when the image and text description are input at the same time, inputting the image into the first model, inputting the text description into the second model, and then inputting the respective output results into the third model at the same time, and performing the fine-grained search according to the output result of the third model.

2. The multimodal composite coded image retrieval method according to claim 1, characterized in that: The second output result includes a text encoding of text features. When the input text data and / or the text description of the image content output by the first model includes multiple sentences, the text encoding is a mixed encoding including global features and local features of the sentences.

3. The multimodal composite coded image retrieval method according to claim 2, characterized in that: Processing the input text data and / or the text description of the image content output by the first model through the second model to obtain a text encoding containing text features includes: Obtaining a sentence vector, and using the sentence vector as a sentence encoding part of the hybrid encoding; Perform part-of-speech tagging on the multiple sentences and perform named entity recognition to obtain words and word vectors for search-related parts of speech, and use the word vectors as word encoding parts of mixed encoding; Obtaining a mixed code including global features and local features of the sentence through the sentence coding part and the word coding part; The sentence encoding part and the word encoding part are both used as retrieval features.

4. The multimodal composite coded image retrieval method according to claim 1, characterized in that: The composite feature obtained by fusing the two features includes: According to the output results of the first model and the second model, the image features and the text features are spliced ​​to obtain the feature matrix and complete the preliminary fusion; Performing attention mechanism processing on the feature matrix through a special convolution to obtain a new feature matrix; The final fusion is completed by performing channel fusion on the new feature matrix and the original feature matrix.

5. The multimodal composite coded image retrieval method according to claim 4, characterized in that: Performing attention mechanism processing on the feature matrix through a special convolution to obtain a new feature matrix that fuses the graph vector with all text features, wherein the length of the special convolution is the vector length of the feature matrix; The new feature matrix is ​​channel-fused with the original feature matrix to form a 2-channel feature map, completing the final fusion.

6. The multimodal composite coded image retrieval method according to claim 4, characterized in that: The initial integration includes: Get the graph vector, all the sentence vectors corresponding to the graph vector, and the word vectors corresponding to all the sentences; All word vectors in the image are concatenated in sequence according to the order of the image vector, the first sentence vector corresponding to the image, and the first word vector corresponding to the first sentence to obtain the feature matrix containing the image features and the text features.

7. A multimodal composite coded image retrieval system, characterized in that: include: The acquisition module is used to obtain the information to be retrieved and determine the data type of the information to be retrieved; A processing module, for selecting coarse-grained retrieval or fine-grained retrieval according to the data type of the information to be retrieved, wherein the fine-grained retrieval includes, when the input information to be retrieved includes both image and text description data, fusing the two features to obtain a composite feature, and then performing data retrieval through the composite feature to obtain target data; the coarse-grained retrieval includes, when the input information to be retrieved includes either image or text description data, performing data retrieval according to the corresponding feature to obtain target data; wherein a recognition model is pre-established and trained, the recognition model includes a first model for image description generation, a second model for text processing, and a third model for multimodal feature extraction; the input image data is processed by the first model to obtain a first output result, the first output result including image feature encoding and A text description of the image content; the input text data and / or the text description of the image content output by the first model are processed by the second model to obtain a second output result; when the input information to be retrieved is a text description, the coarse-grained search is performed through the output result of the second model; when the input information to be retrieved is an image, the coarse-grained search is performed through the image feature coding in the first output result, or the text description of the image content in the first output result is input to the second model, and a fine-grained search is performed through the image feature coding in the first output result and the second output result; when an image and a text description are input at the same time, the image is input to the first model, the text description is input to the second model, and then the respective output results are simultaneously input to the third model, and the fine-grained search is performed according to the output result of the third model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic terminal, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes the method according to any one of claims 1 to 6.

Images