A multimodal search method and device

Abstract

The present invention provides a multimodal retrieval method and device, which relate to the field of artificial intelligence technology. The method includes: obtaining query information and a preset data set; in a text space, based on the matching degree between the query information and each preset data, determining multiple first candidate data from the preset data set; in a multimodal space, based on the similarity between the query information and each preset data on a coding unit, determining multiple second candidate data from the preset data set; fusing multiple first candidate data and multiple second candidate data to obtain a query result corresponding to the query information. The present invention maps multimodal data to a text space, and uses text for retrieval, which can improve the text processing capability in multimodal retrieval, and uses the similarity of data on coding units for searching in a multimodal space, which can better understand data of different modalities, thereby effectively improving the accuracy of multimodal retrieval.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal retrieval method and device. Background Art

[0002] Multimodal retrieval is an information retrieval technology that can retrieve information from data of multiple different modalities, for example, text search for images, image search for text, etc. In multimodal retrieval, users can enter query data containing at least one modality, and the retrieval system can recall data related to the query data from data of different modalities to provide richer and more accurate retrieval results.

[0003] Currently, most multimodal retrieval relies on multimodal models. However, multimodal models have input limitations. The effective input usually does not exceed 20 tokens, and the processing capacity of long texts is insufficient. In addition, the goal of multimodal model training is to align text and images, resulting in weak text retrieval capabilities. When learning the association between text and images, multimodal models focus on using overall semantics for matching, which makes it difficult to capture detailed semantics, resulting in low accuracy of multimodal retrieval. Summary of the invention

[0004] In view of the above problems, an object of the present invention is to provide a multimodal retrieval method and device to improve the accuracy of multimodal retrieval.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] On the one hand, an embodiment of the present invention provides a multimodal retrieval method, comprising:

[0007] Acquire query information and a preset data set, wherein the preset data set includes preset data of multiple modalities;

[0008] In the text space, based on the matching degree between the query information and each of the preset data, determining a plurality of first candidate data matching the query information from the preset data set;

[0009] In the multimodal space, based on the similarity between the query information and each of the preset data in the encoding unit, determining a plurality of second candidate data matching the query information from the preset data set;

[0010] A plurality of the first candidate data and a plurality of the second candidate data are merged to obtain a query result corresponding to the query information.

[0011] Optionally, in the text space, based on the matching degree between the query information and each of the preset data, determining a plurality of first candidate data matching the query information from the preset data set includes:

[0012] Obtain a preset description corresponding to each of the preset data, where the preset description is a text description of the preset data in a text space;

[0013] Based on the modality type of the query information, a text description of the query information in the text space is generated to obtain a query description, wherein the query description is generated in the same manner as the preset description;

[0014] Based on the query description and each of the preset descriptions, calculating a matching degree between the query information and each of the preset data;

[0015] A plurality of first candidate data matching the query information is determined from the plurality of preset data using the matching degree.

[0016] Optionally, generating a text description of the query information in the text space based on the modality type of the query information to obtain the query description includes:

[0017] If the modality type of the query information is a text type, the query information is used as a query description;

[0018] If the modality type of the query information is an image type, a text description of the query information at each preset level is generated based on the level prompt words corresponding to the preset levels to obtain a query description, wherein the preset levels include an overall image level, an image region level, and an object level within an image.

[0019] Optionally, the calculating the matching degree between the query information and each preset data based on the query description and each preset description includes:

[0020] For each of the preset descriptions, calculating the semantic matching degree between the query description and the preset description;

[0021] Calculating the text matching degree between the query description and the preset description;

[0022] The semantic matching degree and the text matching degree are integrated to obtain the matching degree between the query information and the preset data.

[0023] Optionally, determining, in the multimodal space, a plurality of second candidate data matching the query information from the preset data set based on the similarity between the query information and each of the preset data in the encoding unit comprises:

[0024] Based on the modality type of the query information, encode the query information to obtain a query vector, where the query vector includes a unit vector corresponding to each encoding unit in the query information;

[0025] Acquire a preset vector corresponding to each of the preset data, wherein the preset vector includes a unit vector corresponding to each encoding unit in the preset data;

[0026] Calculating, according to each of the preset vectors and the query vector, a similarity between the query information and each of the preset data on a coding unit;

[0027] Using the similarity, a plurality of second candidate data matching the query information is determined from the plurality of preset data.

[0028] Optionally, encoding the query information based on the modality type of the query information to obtain a query vector includes:

[0029] Obtaining the modality type of the query information, and determining the encoding unit and encoding method corresponding to the modality type;

[0030] Using the encoding method corresponding to the modality type, encoding the query information according to the encoding unit to obtain an encoding vector;

[0031] Perform dimensionality reduction processing on the encoding vector to obtain a query vector corresponding to the query information.

[0032] Optionally, the using the encoding method corresponding to the modality type to encode the query information according to the encoding unit to obtain the encoding vector includes:

[0033] If the modality type of the query information is a text type, using the first prompt word template and the query information to obtain a query input corresponding to the query information;

[0034] If the modality type of the query information is an image type, segmenting the query information according to the coding unit to obtain a plurality of sub-query information, and combining the plurality of sub-query information with a second prompt word template to obtain a query input corresponding to the query information;

[0035] The query input is segmented according to encoding units by an encoder corresponding to the modality type, and encoding processing is performed based on a large language model to obtain an encoding vector.

[0036] Optionally, the calculating, according to each of the preset vectors and the query vector, a similarity between the query information and each of the preset data on a coding unit includes:

[0037] For each unit vector in the query vector, calculating the cosine similarity between the unit vector in the query vector and each unit vector in the preset vector;

[0038] Determine the maximum value of the cosine similarities as the maximum similarity of the unit vector;

[0039] The average value of the maximum similarities of all unit vectors is calculated to obtain the similarity between the query information and each of the preset data in the encoding unit.

[0040] Optionally, the fusing the plurality of the first candidate data and the plurality of the second candidate data to obtain a query result corresponding to the query information includes:

[0041] Acquire a first candidate sequence composed of the plurality of first candidate data, wherein the plurality of first candidate data in the first candidate sequence are sorted according to the matching degree;

[0042] Acquire a second candidate sequence composed of the plurality of second candidate data, wherein the plurality of second candidate data in the second candidate sequence are sorted according to the similarity;

[0043] For each candidate data, obtaining a first ranking of the candidate data in the first candidate sequence and a second ranking of the candidate data in the second candidate sequence, the candidate data including the first candidate data and the second candidate data;

[0044] Calculating a target score for the candidate data using the first ranking and the second ranking;

[0045] The query result corresponding to the query information is determined from the multiple candidate data according to the target score.

[0046] On the other hand, an embodiment of the present invention further provides a multimodal retrieval device, used to implement any of the above methods, the device comprising:

[0047] An acquisition module, used to acquire query information and a preset data set, wherein the preset data set includes preset data of multiple modes;

[0048] A first determination module, configured to determine, in a text space, a plurality of first candidate data matching the query information from the preset data set based on a matching degree between the query information and each of the preset data;

[0049] A second determination module, configured to determine, in a multimodal space, a plurality of second candidate data matching the query information from the preset data set based on a similarity between the query information and each of the preset data in a coding unit;

[0050] A fusion module is used to fuse multiple first candidate data and multiple second candidate data to obtain a query result corresponding to the query information.

[0051] On the other hand, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in any multimodal retrieval method provided by the embodiment of the present invention.

[0052] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for a processor to load to execute the steps in any one of the multimodal retrieval methods provided in the embodiments of the present invention.

[0053] On the other hand, an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in any multimodal retrieval method provided by the embodiment of the present invention.

[0054] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0055] The embodiment of the present invention can obtain query information and multiple preset data, calculate the matching degree between the query information and each preset data in the text space, so as to determine multiple first candidate data, and determine multiple second candidate data in the multimodal space based on the similarity between the query information and each preset data in the coding unit, and finally fuse the first candidate data and the second candidate data to obtain the final query result. By translating multimodal data into a unified text space, the text can be used for retrieval to improve the pure text retrieval and long text processing capabilities in multimodal retrieval. In the multimodal space, based on the similarity in the coding unit, the information difference between different modes can be further reduced to achieve better fusion. Finally, the results of the two retrieval methods are merged as the final query information, which is more accurate, thereby effectively improving the accuracy of multimodal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 is a schematic diagram of an application scenario of the multimodal retrieval method provided by an embodiment of the present invention;

[0058] Figure 2 is a flowchart of a multimodal retrieval method provided by an embodiment of the present invention;

[0059] Figure 3is a schematic diagram of a process for determining first candidate data provided by an embodiment of the present invention;

[0060] Figure 4 is a schematic diagram of a process for determining second candidate data provided by an embodiment of the present invention;

[0061] Figure 5 is a schematic diagram of the structure of a multimodal retrieval model provided by an embodiment of the present invention;

[0062] Figure 6 is a schematic diagram of the structure of a multimodal retrieval device provided by an embodiment of the present invention;

[0063] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0065] It is understandable that in the specific implementation of the present invention, data related to user information, etc., needs to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0066] The embodiment of the present invention proposes a multimodal retrieval method to improve the accuracy of multimodal retrieval. Figure 1 , showing a schematic diagram of an application scenario of a multimodal retrieval method. The application scenario may include a terminal 101, a server 102, and a database 103. The terminal 101, the server 102, and the database 103 may exchange data through a network, and a multimodal retrieval platform may be run on the terminal 101. The terminal 101 may be a mobile phone, a tablet computer, a smart Bluetooth device, a computer, an intelligent robot, a large screen, and the like; the server 102 may be a single server or a server cluster consisting of multiple servers.

[0067] The user can input query information through the multimodal retrieval platform on the terminal 101, and the query information can be forwarded by the terminal 101 to the server 102, and the server 102 can obtain the query information and obtain the preset data set from the database 103. Then, the server 102 can determine multiple first candidate data matching the query information from the preset data set based on the matching degree between the query information and each preset data in the text space; in the multimodal space, based on the similarity between the query information and each preset data in the coding unit, determine the second candidate data matching the query information from the preset data set; finally, multiple first candidate data and multiple second candidate data are merged to obtain the query result corresponding to the query information. The server 102 forwards the query result to the terminal 101 so as to display the query result to the user.

[0068] In this embodiment, a multimodal retrieval method is provided, such as Figure 2 As shown, the specific process of the multimodal retrieval method can be as follows:

[0069] S110: Obtain query information and a preset data set.

[0070] Query information is information that users actively input for query. Query information can be unimodal or multimodal, that is, query information can include at least one of images and texts. A preset data set is a set of data in a database, which can include preset data of multiple modalities. Modality refers to different forms of data or information, which can include text modality, image modality, video modality, audio modality, etc.

[0071] S120: In the text space, based on the matching degree between the query information and each of the preset data, determine a plurality of first candidate data from the preset data set.

[0072] As mentioned above, both query information and preset data may contain data of multiple modalities. The representation of data of different modalities in the feature space is inconsistent, resulting in significant differences in the representation of data of different modalities.

[0073] The text space refers to the feature space corresponding to the data in the text mode or the data space under the text mode. By mapping the query information and the preset data into the text space, the matching degree at the text level can be calculated, and then multiple first candidate data can be determined based on the matching degree.

[0074] In some implementations, when determining the first candidate data, the preset description corresponding to each preset data may be obtained, the preset description being a text description of the preset data in a text space; based on the modality type of the query information, a text description of the query information in the text space is generated to obtain a query description, the query description and the preset description are generated in the same manner; based on the query description and each preset description, a matching degree between the query information and each preset data is calculated; and the first candidate data is determined from the multiple preset data using the matching degree. Figure 3 , showing a schematic diagram of the process of determining the first candidate data, the following will be combined with Figure 3 Provide detailed explanation.

[0075] The preset description is a text description of the preset data in the text space. The preset description corresponding to the preset data can be generated and stored in advance, and can be directly obtained when needed.

[0076] The query information may contain information of multiple modalities, and the modality of the information in the query information can be identified to obtain the modality type corresponding to the query information. The query information of different modality types generates the corresponding query description in different ways, and the text description corresponding to the query information can be generated in a way that is adapted to the modality type to obtain the query description. In the embodiment of the present invention, the modality type includes text type and image type as an example for detailed description.

[0077] Optionally, when generating a query description based on the modality type, if the modality type of the query information is a text type, the query information is used as a query description; if the modality type of the query information is an image type, a text description of the query information at each preset level is generated based on the level prompt words corresponding to the preset levels to obtain a query description, wherein the preset levels include an overall image level, an image area level, and an object level within the image.

[0078] If the modality type of the query information is text type, the query information can be directly used as the query description. If the modality type of the query information is image type, the image can be understood to generate a description of the image content to obtain the query description. For the convenience of subsequent description, the query information of text type is called query text, and the query information of image type is called query image.

[0079] In order to further improve the comprehensibility of query information and facilitate accurate understanding of query information in subsequent processing, the query information of the text type input by the user, that is, the query text, can be rewritten and knowledge enhanced to obtain new query information. For example, the query text can be rewritten using a large language model, mainly to correct grammatical errors in the query text, understand the understanding ability of the large language model to supplement the missing information in the query text, and convert the colloquial expressions in the query text into more professional expressions. By rewriting the query text, the query text can be made clearer, more standardized, and easier to understand.

[0080] The rewritten query text can then be enhanced with knowledge, for example, by using the knowledge base on the Internet to provide background knowledge related to the query text, and integrating the background knowledge with the rewritten query text to provide the query text with richer context information for easier understanding.

[0081] If the modality type of the query information is an image type, that is, a query image, the query image can be directly processed using a visual language model so that the visual language model describes the overall content in the query image to obtain a text description of the query information.

[0082] In order to further improve the accuracy of generating text descriptions of query images, a visual language model can be used to generate text descriptions at multiple preset levels to comprehensively and accurately describe the image content and improve the accuracy of the text description. In an embodiment of the present invention, the preset levels may include an overall image level, an image region level, and an image object level. The overall image level refers to the description from the overall content and scene of the image; the image region level refers to the description of the image content and scene according to different regions; the image object level refers to the description of the attributes, positions, and relationships of each object in the image.

[0083] For each preset level, a corresponding level prompt word can be pre-set, wherein the level prompt word corresponding to the overall image level can be: prompt word 1 (prompt1) = describes the overall content and scene of the image, for example, "This is a photo of a busy city street, the weather is gloomy." The level prompt word corresponding to the image region level can be: prompt word 2 (prompt2) = describes the characteristics and semantic information of different areas in the image. For example, "There are high-rise buildings on the left side of the picture, a park on the right side, and a street in the middle." The level prompt word corresponding to the object level in the image can be: prompt word 3 (prompt3) = describes the attributes, positions and relationships of each object in the image. For example, "There are many cars, pedestrians and buildings on the street. A red car is parked on the side of the road, and a pedestrian in blue clothes is crossing the road."

[0084] The hierarchical prompt words and query information are input into the visual language model together. After analysis and reasoning by the visual language model, the text description corresponding to the preset hierarchy can be output, and all the text descriptions of the preset hierarchy are used as the query description corresponding to the query information.

[0085] It should be noted that the generation method of the preset description corresponding to the preset data is consistent with the generation method of the query description. The preset description corresponding to the preset data can be generated and stored in advance. In the actual retrieval process, the preset description corresponding to the preset data can be directly obtained.

[0086] For each preset data, the query description and the corresponding preset description may be used to calculate the matching degree between the preset data and the query information, so as to determine a plurality of first candidate data matching the query information from the plurality of preset data.

[0087] As an implementation method, when calculating the matching degree between the preset data and the query information, it can be that for each of the preset descriptions, the semantic matching degree between the query description and the preset description is calculated; the text matching degree between the query description and the preset description is calculated; and the semantic matching degree and the text matching degree are combined to obtain the matching degree between the query information and the preset description.

[0088] The semantic matching degree can be measured by the semantic similarity between the query description and the preset description. Optionally, the query description can be encoded to obtain a query semantic vector corresponding to the query description; a preset semantic vector corresponding to the preset description is obtained, and the generation method of the preset semantic vector is consistent with the query semantic vector. For each preset data, the cosine similarity between the preset semantic vector and the query semantic vector can be calculated to obtain the semantic matching degree. Among them, the larger the cosine similarity, the more similar the semantics between the preset data and the query information are, and the higher the semantic matching degree is.

[0089] When calculating the text matching degree, the query description and the preset description may be subjected to preprocessing operations such as word segmentation and stop word removal; for each word segment in the query description, the word frequency of the word segment in the query description and the word frequency of the word segment in each preset description are calculated; then the inverse document frequency of each word segment in the query description is calculated; and the text matching degree is calculated using the word frequency and the inverse document frequency.

[0090] Specifically, the text matching degree can be calculated by the following formula:

[0091] ;

[0092] Among them, Score(D,Query) represents the text matching degree between the query information and the preset data; n represents the number of words in the query description; IDF(q i ) represents the participle q iThe inverse document frequency of f(q i ,D) represents the participle q i The word frequency in the preset data; k1 and b are adjustable parameters; |D| represents the length of the preset data; avgdl represents the average length of all preset data.

[0093] According to the above method, for each preset data, the semantic matching degree and the text matching degree can be calculated, and the semantic matching degree and the text matching degree can be merged to obtain the matching degree between the query information and the preset data. As an implementation method, a first weight can be set in advance for the semantic matching degree, and a second weight can be set for the text matching degree. The first weight can control the influence of the semantic matching degree on the final matching degree, and the second weight can control the influence of the text matching degree on the final matching degree. The specific values ​​of the first weight and the second weight can be set according to actual needs and are not specifically limited here.

[0094] The semantic matching degree is weighted using the first weight to obtain a weighted semantic matching degree; the text matching degree is weighted using the second weight to obtain a weighted text matching degree; the weighted semantic matching degree and the weighted text matching degree are summed to obtain a final matching degree.

[0095] The preset data are sorted in order of matching degree from large to small to obtain a first sequence, and the first N preset data in the first sequence are used as first candidate data, where N is a positive integer and can be set according to actual needs. Multiple first candidate data can maintain the arrangement order in the first sequence to form a first candidate sequence. The image modality data is mapped to the text space through a multi-level description, which is more comprehensive and accurate. Then, the text can be used for retrieval to accurately recall multiple first candidate data matching the query information.

[0096] S130. In a multimodal space, based on a similarity between the query information and each of the preset data in a coding unit, determine a plurality of second candidate data from the preset data set.

[0097] Multimodal space refers to the common feature space of multiple modalities. In multimodal space, data features of multiple modalities can be mapped into the same vector space for joint analysis, processing or fusion. In this space, data features of different modalities are represented uniformly, thereby realizing cross-modal information interaction and fusion.

[0098] A coding unit usually refers to the smallest unit of data. The coding units of data in different modalities are different. For example, the coding unit of data in text modality can be a word or a phrase; the coding unit of data in image modality can be a pixel or an area in the image.

[0099] Both the query information and the preset data can be divided according to the coding unit to obtain multiple sub-query information corresponding to the query information and multiple sub-preset data corresponding to the preset data. In the multimodal space, the query information and the preset data can be encoded into corresponding feature vectors, which can contain the unit vector corresponding to each coding unit. The similarity between the query information and the preset data in the coding unit is calculated, which can better capture the detailed information to improve the accuracy of the retrieval.

[0100] In some embodiments, when determining the second candidate data, the query information may be encoded based on the modality type of the query information to obtain a query vector, the query vector including a unit vector corresponding to each encoding unit in the query information; a preset vector corresponding to each preset data is obtained, the preset vector including a unit vector corresponding to each encoding unit in the preset data; based on each preset vector and the query vector, the similarity between the query information and each preset data in the encoding unit is calculated; and using the similarity, a plurality of second candidate data matching the query information is determined from the plurality of preset data. Figure 4 , shows a schematic diagram of the process of determining the second candidate data, and the following will be combined with Figure 4 Provide detailed explanation.

[0101] In order to calculate the similarity between the query information and each preset data in the coding unit, the feature vectors of the query information and each preset data in the multimodal space can be obtained. Different encoding methods correspond to query information of different modalities. Therefore, it is necessary to encode the query information based on the modality type of the query information to obtain the query vector corresponding to the query information. The query vector contains the unit vector corresponding to each coding unit in the query information, for example Figure 4 In, the encoding units of the query text are T1, T2...Tn, and the encoding units of the query image are I1, I2, I3...In. The encoded query vector also includes the unit vectors corresponding to each encoding unit.

[0102] Optionally, when obtaining a query vector corresponding to the query information, the modal type of the query information can be obtained, and a coding unit and a coding method corresponding to the modal type can be determined; using the coding method corresponding to the modal type, the query information is encoded according to the coding unit to obtain a coding vector; and the coding vector is subjected to dimensionality reduction processing to obtain a query vector corresponding to the query information.

[0103] After obtaining the query information, the modality type corresponding to the query information can be obtained, and then the encoding unit and the encoding method can be determined using the modality type. The association relationship between the preset modality type, the preset encoding unit, and the preset encoding method can be pre-stored. After obtaining the modality type corresponding to the query information, the preset encoding unit and the preset encoding method associated with the modality type can be determined by querying the association relationship, and used as the encoding unit and the encoding method of the query information.

[0104] Then, the query information can be encoded according to the encoding unit using the encoding method corresponding to the modality type to obtain an encoding vector. Specifically, if the modality type of the query information is a text type, the query input corresponding to the query information is obtained by using the first prompt word template and the query information; if the modality type of the query information is an image type, the query information is segmented according to the encoding unit to obtain multiple sub-query information, and the multiple sub-query information is combined with the second prompt word template to obtain the query input corresponding to the query information; the query input is segmented according to the encoding unit by the encoder corresponding to the modality type, and encoding is performed based on the large language model to obtain an encoding vector.

[0105] The model used for encoding is a multimodal retrieval model, which may include an input layer, an image encoder, a connector, a large language model, and an aligned output layer. For example, see Figure 5 , showing a schematic structural diagram of the multimodal retrieval model.

[0106] If the modal type of the query information is text type, the first prompt word template and the query information can be used to obtain the query input corresponding to the query information. The first prompt word template is pre-set and can be: prompt = "Query: {txt}" + " <pad>” ; "Query:" is used to indicate that the current task scenario is to generate a query vector, <pad>" is a special token in the multimodal model, used to indicate image slices. Since there is only text input currently, in order to maintain the consistency of the input format of the multimodal model, a certain number of " <pad>" is used as a placeholder; {text} is the query information.

[0107] That is, by filling the query information into the first prompt word template, a query input corresponding to the text type query information can be obtained, and the query input can be input into the input layer.

[0108] If the modality type of the query information is an image type, the query image may be segmented according to the coding unit. In the embodiment of the present invention, the coding unit corresponding to the query image may be a coding unit with a resolution of That is, the complete query information can be obtained with a resolution of The encoding unit divides the query information into multiple sub-query information. Then, the multiple sub-query information can be combined with the second prompt word template to obtain the corresponding query input. The second prompt word template can be pre-set, and the second prompt word template can be: Prompt = Describe the image in detail; is a placeholder for the query image. The second prompt word template and multiple sub-query information are taken as query input, and the query input can be input into the input layer.

[0109] In some implementations, before the query image is divided according to the coding units, the query image may be preprocessed to improve the subsequent processing. For example, the query image may be scaled to obtain a resized image, the size of which is a multiple of the resize factor; the resized image may be pixel adjusted to obtain a pixel adjusted image, the pixel of which is between a first pixel and a second pixel, the second pixel being larger than the first pixel; and the pixel adjusted image may be used as the query image to be used.

[0110] The resizing factor can be pre-set. Specifically, when adjusting the size of the query image, the width and height of the image can be obtained first. For the width or height, the width or height is divided by the resizing factor, the result is rounded off, and then multiplied by the resizing factor. The process can be expressed as the following formula: ; Where width is the height or width of the query image, and factor is the resizing factor.

[0111] According to the above method, the new width and height of the query image can be calculated to obtain the image after the size change, and then the pixel adjustment can be performed on this basis. The first pixel and the second pixel are both preset, and the first pixel is smaller than the second pixel. Get the current pixel of the image after the size change; if the current pixel is smaller than the first pixel, divide the first pixel by the current pixel to obtain the magnification ratio, multiply the magnification ratio and the current pixel to obtain the pixel-adjusted image. If the current pixel is larger than the second pixel, divide the second pixel by the current pixel to obtain the reduction ratio, multiply the reduction ratio and the current pixel to obtain the pixel-adjusted image, and the pixel of the pixel-adjusted image is between the first pixel and the second pixel.

[0112] After obtaining the query input, the query input can be encoded according to the encoding unit based on the encoder corresponding to the modality type to obtain an encoding vector. Among them, the encoder of the text type is a large language model; the encoder of the image type can be recorded as an image encoder. The image encoder and the large language model are both in the multimodal model.

[0113] Specifically, the query text can first pass through the word segmenter, be divided into text units according to the coding units, and then be input into the large language model to obtain its last layer of latent vectors and obtain the final coding vector. The query input corresponding to the query image can be input into the image encoder and connector for encoding processing to obtain the vector corresponding to each coding unit in the query image, and then the vector is input into the large language model, and the last layer of latent vectors is taken to obtain the final coding vector. The dimension of the coding vector obtained at this time is [1, len1, hidden_size], where len1 represents the number of coding units of the query information, which is a non-fixed value; hidden_size represents the dimension of the latent vector, which is a fixed value, such as 3584.

[0114] In order to improve the efficiency of subsequent similarity calculation, the encoding vector can be reduced in dimension. Specifically, the alignment output layer can be used to reduce the dimension of the encoding vector to a specified dimension. The specified dimension can be set according to actual needs. For example, in an embodiment of the present invention, the specified dimension can be 256 dimensions. After the dimensionality reduction process, the query vector corresponding to the query information can be obtained. At the same time, the alignment output layer can further change the query vector so that the features of the image and text are aligned on the encoding unit, which can be achieved by calculating the similarity.

[0115] The query vector includes the unit vectors corresponding to each coding unit in the query information, and the preset vector also includes the unit vectors corresponding to each coding unit in the preset data, and the similarity between the query information and any preset data in the coding unit is calculated using the unit vectors. Optionally, for each unit vector in the query vector, the cosine similarity between the unit vector in the query vector and each unit vector in the preset vector is calculated; the maximum value of the cosine similarities is determined as the maximum similarity of the unit vector; the average value of the maximum similarities of all unit vectors is calculated to obtain the similarity between the query vector and the candidate vector.

[0116] For each unit vector in the query vector, the cosine similarity between the unit vector and each unit vector in the preset vector can be calculated, thereby obtaining multiple cosine similarities corresponding to the unit vector. Then the maximum value of the multiple cosine similarities can be obtained, and the maximum value is used as the maximum similarity of the unit vector. In this way, each unit vector in the query vector can calculate its corresponding maximum similarity. Based on the maximum similarity of each unit vector and the number of unit vectors, the average value of the maximum similarities of all unit vectors is calculated to obtain the similarity between the query information and each preset data on the coding unit.

[0117] The calculation process of the similarity can refer to the following formula:

[0118] ;

[0119] in, Represents the unit vector corresponding to the i-th encoding unit in the query vector; represents the unit vector corresponding to the jth coding unit in the preset vector; len1 represents the total number of coding units in the query vector; len2 represents the total number of coding units in the preset vector.

[0120] According to the above method, the similarity between the query information and each preset data can be calculated, and the preset data can be sorted in descending order according to the similarity to obtain a second sequence, and then the first M preset data in the second sequence are taken as the second candidate data, where M is a positive integer and can be set according to actual needs. Multiple second candidate numbers can maintain their arrangement order in the second sequence to form a second candidate sequence.

[0121] It should be noted that the multimodal retrieval model needs to be pre-trained before use. When training multimodal retrieval, it is first necessary to construct training data. The training data can use a public multi-domain visual question-answering dataset. Each piece of training data contains a query and a corresponding image. The loss function can use a contrast loss function, that is, in a training batch, the similarity between the query and its corresponding image is very high, while the similarity with other images in the training batch is very low. The multimodal retrieval model can be modified based on the visual language model. When training the multimodal retrieval model, the image encoder and connector can be frozen, and the lora method can be used to memorize efficient parameters for fine-tuning until the loss function converges to obtain the multimodal retrieval model used.

[0122] In some embodiments, in order to improve the retrieval transparency and credibility of the multimodal retrieval model, the attention score between the image and the text can be calculated, and a heat map of the attention score can be drawn to facilitate the user to understand the retrieval process of the second candidate data. When the retrieval result does not meet expectations, analysis can also be performed based on the heat map. For example, taking the query information as the query text and the preset data as the preset image as an example, the attention score of each coding unit in the preset image, that is, the sub-image, can be calculated by the following formula:

[0123] ;

[0124] in, Represents the attention score of a subgraph; len1 represents the number of encoding units in the query information; Represents the unit vector corresponding to the i-th coding unit in the query information; is the unit vector corresponding to the subgraph.

[0125] S140: Merge multiple first candidate data and multiple second candidate data to obtain a query result corresponding to the query information.

[0126] After obtaining a plurality of first candidate data and a plurality of second candidate data, the first candidate data and the second candidate data may be fused to obtain a query result corresponding to the query information.

[0127] As an implementation mode, when determining the query result, a first candidate sequence consisting of the multiple first candidate data may be obtained, and the multiple first candidate data in the first candidate sequence may be sorted according to the matching degree; a second candidate sequence consisting of the multiple second candidate data may be obtained, and the multiple second candidate data in the second candidate sequence may be sorted according to the similarity; for each candidate data, a first ranking of the candidate data in the first candidate sequence and a second ranking in the second candidate sequence may be obtained, and the candidate data may include the first candidate data and the second candidate data; a target score of the candidate data may be calculated using the first ranking and the second ranking; and a query result corresponding to the query information may be determined from the multiple candidate data using the target score.

[0128] As mentioned above, when determining multiple first candidate data, they are sorted by matching degree. Thus, the sequence obtained after sorting the multiple first candidate data from large to small based on matching degree can be recorded as the first candidate sequence. Similarly, the sequence obtained after sorting the second candidate data from large to small based on similarity can be recorded as the second candidate sequence.

[0129] The data in the first candidate sequence and the second candidate sequence may be referred to as candidate data. For each candidate data, the first ranking corresponding to the candidate data in the first candidate sequence and the second ranking corresponding to the candidate data in the second candidate sequence may be determined. Then the reciprocal of the first ranking and the reciprocal of the second ranking are calculated, and the two reciprocals are combined to obtain the target score of the candidate data, and then the multiple candidate data are sorted according to the target score, so as to intercept the first preset number of candidate data with the highest ranking as the final query result, wherein the preset number can be set according to actual needs and is not specifically limited here.

[0130] After determining the query results, the corresponding query results can be directly output so that the user can obtain the query results corresponding to the query information. It should be noted that if the query information contains both text modality data and image modality data, the corresponding modality data is searched in the aforementioned manner to obtain the retrieval results corresponding to each modality, and the retrieval results under each modality are merged to obtain the final retrieval results for output. Similarly, when merging the retrieval results corresponding to each modality, the method of fusing the first candidate data and the second candidate data can also be referred to, and no specific limitation is made here.

[0131] The multimodal retrieval solution provided by the embodiment of the present invention can be applied in various retrieval scenarios, such as book search, photo search, video search, Internet search, intelligent question answering, etc. The solution provided by the embodiment of the present invention supports input of images, texts or graphics during retrieval, and accurately recalls relevant images, texts or graphics from the database based on the input content.

[0132] The method provided by the embodiment of the present invention can introduce text descriptions at multiple preset levels, translate images into a unified text space, and use text retrieval technology to achieve efficient retrieval, which can enhance the long text processing capability and pure text retrieval capability. Feature alignment of query information and preset data on the coding unit can further reduce the data difference between different modal data, and more accurately understand the semantics of query information and preset data. Finally, the first candidate data and the second candidate data recalled under the two retrieval methods are integrated, and the final retrieval result is obtained after rearrangement, which can effectively improve the accuracy of multimodal retrieval.

[0133] In order to better implement the above method, the embodiment of the present invention also provides a multimodal retrieval device, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0134] For example, in this embodiment, the method of the embodiment of the present invention is described in detail by taking the multimodal retrieval device being specifically integrated in a server as an example.

[0135] For example, Figure 6 As shown, the multimodal retrieval device 200 may include an acquisition module 210 , a first determination module 220 , a second determination module 230 and a fusion module 240 .

[0136] An acquisition module 210, configured to acquire query information and a preset data set, wherein the preset data set includes preset data of multiple modalities;

[0137] A first determination module 220, configured to determine, in a text space, a plurality of first candidate data matching the query information from the preset data set based on a matching degree between the query information and each of the preset data;

[0138] A second determination module 230 is used to determine, in a multimodal space, a plurality of second candidate data matching the query information from the preset data set based on the similarity between the query information and each of the preset data in the coding unit;

[0139] The fusion module 240 is used to fuse multiple first candidate data and multiple second candidate data to obtain a query result corresponding to the query information.

[0140] In some embodiments, the first determining module 220 is specifically configured to:

[0141] Obtain a preset description corresponding to each of the preset data, where the preset description is a text description of the preset data in a text space;

[0142] Based on the modality type of the query information, a text description of the query information in the text space is generated to obtain a query description, wherein the query description is generated in the same manner as the preset description;

[0143] Based on the query description and each of the preset descriptions, calculating a matching degree between the query information and each of the preset data;

[0144] A plurality of first candidate data matching the query information is determined from the plurality of preset data using the matching degree.

[0145] In some embodiments, the first determining module 220 is specifically configured to:

[0146] If the modality type of the query information is a text type, the query information is used as a query description;

[0147] If the modality type of the query information is an image type, a text description of the query information at each preset level is generated based on the level prompt words corresponding to the preset levels to obtain a query description, wherein the preset levels include an overall image level, an image region level, and an object level within an image.

[0148] In some embodiments, the first determining module 220 is specifically configured to:

[0149] For each of the preset descriptions, calculating the semantic matching degree between the query description and the preset description;

[0150] Calculating the text matching degree between the query description and the preset description;

[0151] The semantic matching degree and the text matching degree are integrated to obtain the matching degree between the query information and the preset data.

[0152] In some embodiments, the second determining module 230 is specifically configured to:

[0153] Based on the modality type of the query information, encode the query information to obtain a query vector, where the query vector includes a unit vector corresponding to each encoding unit in the query information;

[0154] Acquire a preset vector corresponding to each of the preset data, wherein the preset vector includes a unit vector corresponding to each encoding unit in the preset data;

[0155] Calculating, according to each of the preset vectors and the query vector, a similarity between the query information and each of the preset data on a coding unit;

[0156] Using the similarity, a plurality of second candidate data matching the query information is determined from the plurality of preset data.

[0157] In some embodiments, the second determining module 230 is specifically configured to:

[0158] Obtaining the modality type of the query information, and determining the encoding unit and encoding method corresponding to the modality type;

[0159] Using the encoding method corresponding to the modality type, encoding the query information according to the encoding unit to obtain an encoding vector;

[0160] Perform dimensionality reduction processing on the encoding vector to obtain a query vector corresponding to the query information.

[0161] In some embodiments, the second determining module 230 is specifically configured to:

[0162] If the modality type of the query information is a text type, using the first prompt word template and the query information to obtain a query input corresponding to the query information;

[0163] If the modality type of the query information is an image type, segmenting the query information according to the coding unit to obtain a plurality of sub-query information, and combining the plurality of sub-query information with a second prompt word template to obtain a query input corresponding to the query information;

[0164] The query input is segmented according to encoding units by an encoder corresponding to the modality type, and encoding processing is performed based on a large language model to obtain an encoding vector.

[0165] In some embodiments, the second determining module 230 is specifically configured to:

[0166] For each unit vector in the query vector, calculating the cosine similarity between the unit vector in the query vector and each unit vector in the preset vector;

[0167] Determine the maximum value of the cosine similarities as the maximum similarity of the unit vector;

[0168] The average value of the maximum similarities of all unit vectors is calculated to obtain the similarity between the query information and each of the preset data in the encoding unit.

[0169] In some embodiments, the fusion module 240 is specifically used to:

[0170] Acquire a first candidate sequence composed of the plurality of first candidate data, wherein the plurality of first candidate data in the first candidate sequence are sorted according to the matching degree;

[0171] Acquire a second candidate sequence composed of the plurality of second candidate data, wherein the plurality of second candidate data in the second candidate sequence are sorted according to the similarity;

[0172] For each candidate data, obtaining a first ranking of the candidate data in the first candidate sequence and a second ranking of the candidate data in the second candidate sequence, the candidate data including the first candidate data and the second candidate data;

[0173] Calculating a target score for the candidate data using the first ranking and the second ranking;

[0174] The query result corresponding to the query information is determined from the multiple candidate data according to the target score.

[0175] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0176] As can be seen from the above, the multimodal retrieval device of this embodiment can obtain query information and multiple preset data, calculate the matching degree between the query information and each preset data in the text space, so as to determine multiple first candidate data, and in the multimodal space, determine multiple second candidate data based on the similarity between the query information and each preset data in the coding unit, and finally fuse the first candidate data and the second candidate data to obtain the final query result. By translating multimodal data into a unified text space, text can be used for retrieval to improve the pure text retrieval and long text processing capabilities in multimodal retrieval. In the multimodal space, based on the similarity in the coding unit, the information difference between different modes can be further reduced to achieve better fusion. Finally, the results of the two retrieval methods are combined as the final query information, which is more accurate, thereby effectively improving the accuracy of multimodal retrieval.

[0177] The embodiment of the present invention further provides an electronic device, which may be a terminal, a server, or the like. The terminal may be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer, or the like; the server may be a single server or a server cluster composed of multiple servers, or the like.

[0178] In some embodiments, the multimodal retrieval device may also be integrated into multiple electronic devices. For example, the multimodal retrieval device may be integrated into multiple servers, and the multimodal retrieval method of the present invention may be implemented by multiple servers.

[0179] In this embodiment, the electronic device of this embodiment is a server as an example for detailed description, for example, Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present invention, specifically:

[0180] The electronic device may include components such as a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, a power supply 330, an input module 340, and a communication module 350. Those skilled in the art will appreciate that Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0181] The processor 310 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 320, and calling data stored in the memory 320. In some embodiments, the processor 310 may include one or more processing cores; in some embodiments, the processor 310 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 310.

[0182] The memory 320 can be used to store software programs and modules. The processor 310 executes various functional applications and data processing by running the software programs and modules stored in the memory 320. The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 320 may also include a memory controller to provide the processor 310 with access to the memory 320.

[0183] The electronic device also includes a power supply 330 for supplying power to various components. In some embodiments, the power supply 330 can be logically connected to the processor 310 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 330 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0184] The electronic device may further include an input module 340, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0185] The electronic device may further include a communication module 350. In some embodiments, the communication module 350 may include a wireless module. The electronic device may perform short-range wireless transmission through the wireless module of the communication module 350, thereby providing the user with wireless broadband Internet access. For example, the communication module 350 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0186] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 310 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 320 according to the following instructions, and the processor 310 will run the application programs stored in the memory 320, thereby implementing the steps in the methods of the embodiments of the present invention.

[0187] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0188] As can be seen from the above, the electronic device of the embodiment of the present invention can obtain query information and multiple preset data, calculate the matching degree between the query information and each preset data in the text space, so as to determine multiple first candidate data, and in the multimodal space, determine multiple second candidate data based on the similarity between the query information and each preset data in the coding unit, and finally fuse the first candidate data and the second candidate data to obtain the final query result. By translating multimodal data into a unified text space, the text can be used for retrieval to improve the pure text retrieval and long text processing capabilities in multimodal retrieval. In the multimodal space, based on the similarity in the coding unit, the information difference between different modes can be further reduced to achieve better fusion. Finally, the results of the two retrieval methods are combined as the final query information, which is more accurate, thereby effectively improving the accuracy of multimodal retrieval.

[0189] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0190] To this end, an embodiment of the present invention provides a computer-readable storage medium, in which a plurality of instructions are stored. The instructions can be loaded by a processor to execute the steps in any one of the multimodal retrieval methods provided by the embodiments of the present invention.

[0191] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0192] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program comprising a computer program / instruction, the computer program / instruction being stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instruction from the computer-readable storage medium, and the processor executes the computer program / instruction, so that the electronic device executes the method provided in various optional implementations of the multimodal retrieval aspect provided in the above-mentioned embodiments.

[0193] Since the instructions stored in the storage medium can execute the steps in any multimodal retrieval method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any multimodal retrieval method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0194] The above is a detailed introduction to a multimodal retrieval method and device provided in an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.< / pad> < / pad> < / pad>

Claims

1. A multimodal retrieval method, characterized in that: The method comprises: Acquire query information and a preset data set, wherein the preset data set includes preset data of multiple modalities; In a text space, based on the matching degree between the query information and each of the preset data, a plurality of first candidate data matching the query information is determined from the preset data set, wherein the text space is a feature space corresponding to the data in the text modality; In a multimodal space, based on the similarity between the query information and each of the preset data in the encoding unit, a plurality of second candidate data matching the query information is determined from the preset data set, wherein if the modality type of the query information is a text type, the encoding unit is a word, and if the modality type of the query information is an image type, the encoding unit is an area in the image, and the multimodal space is a unified representation space of data features of multiple modalities; Merging a plurality of the first candidate data and a plurality of the second candidate data to obtain a query result corresponding to the query information; Wherein, in the text space, based on the matching degree between the query information and each of the preset data, determining a plurality of first candidate data matching the query information from the preset data set comprises: Obtain a preset description corresponding to each of the preset data, wherein the preset description is a text description of the preset data in a text space; if the modality type of the query information is a text type, use the query information as a query description; if the modality type of the query information is an image type, generate a text description of the query information at each preset level based on the level prompt words corresponding to the preset level to obtain a query description, wherein the preset level includes an image overall level, an image region level, and an image object level, wherein the query description and the preset description are generated in the same manner; based on the query description and each of the preset descriptions, calculate a matching degree between the query information and each of the preset data; and determine a plurality of first candidate data matching the query information from a plurality of preset data using the matching degree; The determining, in the multimodal space, a plurality of second candidate data matching the query information from the preset data set based on the similarity between the query information and each of the preset data in the encoding unit, comprises: The method comprises: obtaining a modality type of the query information, and determining a coding unit and a coding method corresponding to the modality type; if the modality type of the query information is a text type, using a first prompt word template and the query information to obtain a query input corresponding to the query information; if the modality type of the query information is an image type, segmenting the query information according to the coding unit to obtain a plurality of sub-query information, and combining the plurality of sub-query information with a second prompt word template to obtain a query input corresponding to the query information; segmenting the query input according to the coding unit by an encoder corresponding to the modality type, and performing encoding processing based on a large language model to obtain an encoding vector; performing dimensionality reduction processing on the encoding vector to obtain a query vector corresponding to the query information, the query vector including a unit vector corresponding to each encoding unit in the query information; obtaining a preset vector corresponding to each of the preset data, the preset vector including a unit vector corresponding to each encoding unit in the preset data; calculating the similarity between the query information and each of the preset data in the encoding unit according to each of the preset vectors and the query vector; and determining a plurality of second candidate data matching the query information from a plurality of preset data using the similarity.

2. The method according to claim 1, characterized in that The calculating the matching degree between the query information and each preset data based on the query description and each preset description includes: For each of the preset descriptions, calculating the semantic matching degree between the query description and the preset description; Calculating the text matching degree between the query description and the preset description; The semantic matching degree and the text matching degree are integrated to obtain the matching degree between the query information and the preset data.

3. The method according to claim 1, characterized in that: The calculating, according to each of the preset vectors and the query vector, the similarity between the query information and each of the preset data on the encoding unit includes: For each unit vector in the query vector, calculating the cosine similarity between the unit vector in the query vector and each unit vector in the preset vector; Determine the maximum value of the cosine similarities as the maximum similarity of the unit vector; The average value of the maximum similarities of all unit vectors is calculated to obtain the similarity between the query information and each of the preset data in the encoding unit.

4. The method according to claim 1, characterized in that The fusing the plurality of the first candidate data and the plurality of the second candidate data to obtain a query result corresponding to the query information includes: Acquire a first candidate sequence composed of the plurality of first candidate data, wherein the plurality of first candidate data in the first candidate sequence are sorted according to the matching degree; Acquire a second candidate sequence composed of the plurality of second candidate data, wherein the plurality of second candidate data in the second candidate sequence are sorted according to the similarity; For each candidate data, obtaining a first ranking of the candidate data in the first candidate sequence and a second ranking of the candidate data in the second candidate sequence, the candidate data including the first candidate data and the second candidate data; Calculating a target score for the candidate data using the first ranking and the second ranking; The query result corresponding to the query information is determined from multiple candidate data according to the target score.

5. A multimodal retrieval device, the device being used to implement the method according to any one of claims 1 to 4, characterized in that: The device comprises: An acquisition module, used to acquire query information and a preset data set, wherein the preset data set includes preset data of multiple modes; A first determination module is used to determine, in a text space, a plurality of first candidate data matching the query information from the preset data set based on a matching degree between the query information and each of the preset data, wherein the text space is a feature space corresponding to data in a text modality; A second determination module is used to determine, in a multimodal space, a plurality of second candidate data matching the query information from the preset data set based on the similarity between the query information and each of the preset data in the encoding unit, wherein if the modality type of the query information is a text type, the encoding unit is a word, and if the modality type of the query information is an image type, the encoding unit is a region in the image, and the multimodal space is a unified representation space of data features of multiple modalities; A fusion module, used for fusing a plurality of the first candidate data and a plurality of the second candidate data to obtain a query result corresponding to the query information; Wherein, in the text space, based on the matching degree between the query information and each of the preset data, determining a plurality of first candidate data matching the query information from the preset data set comprises: Obtain a preset description corresponding to each of the preset data, wherein the preset description is a text description of the preset data in a text space; if the modality type of the query information is a text type, use the query information as a query description; if the modality type of the query information is an image type, generate a text description of the query information at each preset level based on the level prompt words corresponding to the preset level to obtain a query description, wherein the preset level includes an image overall level, an image region level, and an image object level, wherein the query description and the preset description are generated in the same manner; based on the query description and each of the preset descriptions, calculate a matching degree between the query information and each of the preset data; and determine a plurality of first candidate data matching the query information from a plurality of preset data using the matching degree; The determining, in the multimodal space, a plurality of second candidate data matching the query information from the preset data set based on the similarity between the query information and each of the preset data in the encoding unit, comprises: The method comprises: obtaining a modality type of the query information, and determining a coding unit and a coding method corresponding to the modality type; if the modality type of the query information is a text type, using a first prompt word template and the query information to obtain a query input corresponding to the query information; if the modality type of the query information is an image type, segmenting the query information according to the coding unit to obtain a plurality of sub-query information, and combining the plurality of sub-query information with a second prompt word template to obtain a query input corresponding to the query information; segmenting the query input according to the coding unit by an encoder corresponding to the modality type, and performing encoding processing based on a large language model to obtain an encoding vector; performing dimensionality reduction processing on the encoding vector to obtain a query vector corresponding to the query information, the query vector including a unit vector corresponding to each encoding unit in the query information; obtaining a preset vector corresponding to each of the preset data, the preset vector including a unit vector corresponding to each encoding unit in the preset data; calculating the similarity between the query information and each of the preset data in the encoding unit according to each of the preset vectors and the query vector; and determining a plurality of second candidate data matching the query information from a plurality of preset data using the similarity.

Images