Cross-modal retrieval method and device

Abstract

The invention provides a cross-modal retrieval method and device, and relates to the technical field of cross-modal retrieval and artificial intelligence. The method comprises the steps that a to-be-queried sample, a pre-trained multi-modal large language model and an identifier index database are obtained, the identifier index database is constructed by using a sample data set of multiple preset modals and the multi-modal large language model, and the to-be-queried sample has at least one of the multiple preset modals; inputting a to-be-queried sample into the multi-modal large language model, and outputting a structured semantic identifier corresponding to the to-be-queried sample; and performing matching in an identifier index database based on the semantic identifier, and determining a query result corresponding to the to-be-queried sample according to a matching result. According to the method, the structured semantic identifier is generated through the multi-modal large language model, and the constructed identifier index database is matched with the semantic identifier, so that efficient and accurate cross-modal retrieval is realized.

Description

Technical Field

[0001] This application relates to the fields of cross-modal retrieval and artificial intelligence technology, and more specifically, to a cross-modal retrieval method and apparatus. Background Technology

[0002] Cross-modal retrieval aims to process data from different modalities, such as images and text, and to achieve semantic association and retrieval matching between these modalities. A cross-modal retrieval system needs to possess multimodal semantic encoding capabilities to map images and text to a unified semantic representation space, thereby achieving cross-modal semantic alignment. This allows users to query images using text or retrieve related text content using images.

[0003] Currently, cross-modal retrieval employs generative cross-modal retrieval technology. This technology leverages the generative capabilities of multimodal large language models to directly generate identifiers corresponding to samples during the inference phase, thus completing the end-to-end retrieval process. However, when generative cross-modal retrieval technology is directly applied to cross-modal retrieval tasks, identifier design typically relies on manual annotation, clustering generation, or atomic identifiers that require expanding the vocabulary. This makes it difficult to achieve a good balance between semantic alignment and scalability. Such design limitations can easily cause the generated results to deviate from the predefined semantic space, ultimately leading to a decline in cross-modal retrieval performance. Summary of the Invention

[0004] In view of the above problems, this application provides a cross-modal retrieval method and apparatus.

[0005] This application provides a cross-modal retrieval method, comprising: acquiring a query sample, a pre-trained multimodal large language model, and an identifier index library, wherein the identifier index library is constructed using a sample dataset of multiple preset modalities and the multimodal large language model, and the query sample has at least one preset modality; inputting the query sample into the multimodal large language model and outputting a structured semantic identifier corresponding to the query sample; performing matching based on the semantic identifier in the identifier index library, and determining the query result corresponding to the query sample based on the matching result.

[0006] According to an embodiment of this application, the identifier index library is constructed in the following manner: Based on a pre-built prompt template, multiple sample datasets are processed to obtain a model input sequence. The prompt template is constructed according to a predetermined input format and a predetermined output format. The input format of the prompt template is used to constrain the modality of the processed multiple sample datasets. The model input sequence is input into a multimodal large language model to obtain a structured semantic identifier corresponding to each sample dataset. The output format of the prompt template is used to constrain the output format of the multimodal large language model. The semantic identifier has the same format as the output format of the prompt template. Based on the semantic identifier corresponding to each sample dataset, the association relationship between each sample dataset and its corresponding semantic identifier is determined. Based on the association relationship, an identifier index library is constructed.

[0007] According to embodiments of this application, the input format includes a concatenation order and placeholders corresponding to each modality; based on a pre-built prompt template, multiple sample datasets are processed to obtain a model input sequence, including: for any sample dataset, encoding the sample dataset to obtain the encoding features of the sample dataset; based on the encoding features, determining the target placeholder corresponding to the modality of the sample dataset from multiple placeholders; concatenating the sample dataset with the target placeholder to obtain a concatenated sample dataset; and concatenating multiple concatenated sample datasets a second time based on the concatenation order to obtain the model input sequence.

[0008] According to embodiments of this application, semantic identifiers include a first semantic identifier and a second semantic identifier; association relationships include a first association relationship and a second association relationship; based on the association relationships, constructing an identifier index library includes: when the semantic identifier is a first semantic identifier, extracting a first identifier vector of the first semantic identifier and a sample vector of the sample dataset; determining the similarity between the first identifier vector and the sample vector, and establishing a first association relationship between the first semantic identifier and the sample dataset when the similarity is greater than a preset similarity threshold; when the semantic identifier is a second semantic identifier, obtaining the hash value of the second semantic identifier and the hash value of the first semantic identifier; establishing a second association relationship between the second semantic identifier and the sample dataset when the difference between the hash value of the second semantic identifier and the hash value of the first semantic identifier is within a preset difference threshold range; and constructing an identifier index library based on the first association relationship and the second association relationship.

[0009] According to embodiments of this application, a semantic identifier is composed of at least two semantic units combined according to a predetermined combination rule. The semantic units are target object semantic units, behavior action semantic units, or scene environment semantic units.

[0010] According to an embodiment of this application, a query sample is input into a multimodal large language model, and a structured semantic identifier corresponding to the query sample is output. This includes: processing the multimodal large language model based on a preset identifier generation strategy to generate multiple candidate semantic identifiers corresponding to the query sample; matching the multiple candidate semantic identifiers with a pre-constructed tree constraint structure, wherein the tree constraint structure is constructed using multiple semantic units; and based on the matching results, selecting candidate semantic identifiers that meet preset matching conditions from the multiple candidate semantic identifiers as the structured semantic identifiers corresponding to the query sample.

[0011] According to an embodiment of this application, the tree constraint structure is constructed in the following manner: each semantic unit is instantiated as a node, and constraints are configured for each node, including the dependencies between each node and the semantic type of each node; based on the constraints, the connection relationships of each node are determined; and based on the nodes and connection relationships, the tree constraint structure is constructed.

[0012] According to an embodiment of this application, the tree constraint structure includes multiple connecting edges; based on the matching results, candidate semantic identifiers that meet preset matching conditions are selected from multiple candidate semantic identifiers as the structured semantic identifiers corresponding to the queried sample, including: determining the generation probability of each candidate semantic identifier and the semantic unit corresponding to each candidate semantic identifier; instantiating multiple semantic units corresponding to each candidate semantic identifier into multiple candidate nodes; determining candidate connecting edges corresponding to each candidate semantic identifier based on multiple candidate nodes and constraint conditions; when there is a connecting edge that is the same as the candidate connecting edge among multiple connecting edges, the candidate semantic identifier with the highest generation probability is selected as the structured semantic identifier corresponding to the queried sample.

[0013] According to an embodiment of this application, the method further includes: obtaining semantic explanation information of the semantic identifier corresponding to the sample to be queried; constructing a training sample pair using the semantic identifier corresponding to the sample to be queried as the key and the semantic explanation information as the value; performing secondary training on the multimodal large language model based on the training sample pair; and updating the identifier index library using the secondary trained multimodal large language model.

[0014] This application also provides a cross-modal retrieval device, comprising: a sample acquisition module for acquiring a query sample, a pre-trained multimodal large language model, and an identifier index library, wherein the identifier index library is constructed using sample datasets of multiple preset modalities and the multimodal large language model, and the query sample has at least one preset modality; an identifier output module for inputting the query sample into the multimodal large language model and outputting a structured semantic identifier corresponding to the query sample; and an identifier query module for performing matching based on the semantic identifier in the identifier index library and determining the query result corresponding to the query sample based on the matching result.

[0015] The cross-modal retrieval method and apparatus provided in this application can achieve the following beneficial effects:

[0016] (1) By guiding the multimodal large language model to autonomously generate semantic identifiers with fixed structures and combining semantic interpretation information as auxiliary supervision signals, the multimodal large language model's ability to model the semantic correspondence between samples and semantic identifiers is enhanced, thereby improving the stability and reliability of generative cross-modal retrieval tasks.

[0017] (2) By matching the constructed identifier index library with semantic identifiers, efficient and accurate cross-modal retrieval was achieved. Attached Figure Description

[0018] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 A flowchart illustrating a cross-modal retrieval method according to an embodiment of this application is shown schematically.

[0020] Figure 2 This schematic diagram illustrates the structure of training a multimodal large language model according to an embodiment of this application.

[0021] Figure 3 A schematic diagram illustrating the structure of a cross-modal retrieval device according to an embodiment of this application is shown.

[0022] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a cross-modal retrieval method according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] Figure 1 A flowchart illustrating a cross-modal retrieval method according to an embodiment of this application is shown schematically.

[0027] like Figure 1 As shown, the cross-modal retrieval method of this embodiment includes steps S110 to S130.

[0028] In step S110, the query sample, a pre-trained multimodal large language model, and an identifier index library are obtained. The identifier index library is constructed using sample datasets of multiple preset modalities and the multimodal large language model. The query sample has at least one preset modality among the multiple preset modalities.

[0029] Preset modalities can be the categories of sample types that can be processed, which are pre-defined before building the index and performing the retrieval. They can include one or more combinations of text modalities, image modalities, audio modalities, and video modalities.

[0030] The sample to be queried can be the target sample for cross-modal retrieval, which is the carrier of the retrieval request and includes at least one of the following modalities: text, image, audio, and video.

[0031] A multimodal large language model is an artificial intelligence model trained on a large number of samples with different modalities (including text modality, image modality, audio modality, video modality and other preset modalities). It has the ability to cross-modal semantic understanding, feature extraction and unified representation, and can convert samples from different modalities.

[0032] The sample dataset can be a collection of samples with multiple preset modalities. The identifier index can be a database used to store the retrieval matching criteria, and which establishes a unique association between the sample datasets with the sample datasets after processing by a multimodal large language model.

[0033] In step S120, the sample to be queried is input into the multimodal large language model, and the structured semantic identifier corresponding to the sample to be queried is output.

[0034] Semantic identifiers are semantic representations with a unified data format output by a multimodal large language model after performing cross-modal semantic understanding, feature extraction, and structured encoding on the query sample. Semantic identifiers include, but are not limited to, vector sequences, character encoded strings, and structured label groups, and maintain a unique semantic association with the query sample.

[0035] In step S130, a match is performed in the identifier index based on the semantic identifier, and the query result corresponding to the sample to be queried is determined according to the matching result.

[0036] The query results can be a list of existing semantic identifiers that successfully match the semantic identifiers of the sample to be queried, the similarity scores and sorting results of each existing semantic identifier, and each existing semantic identifier has a unique association with the image sample data pre-stored in the identifier index.

[0037] The cross-modal retrieval method based on embodiments of this application generates structured semantic identifiers through a multimodal large language model and matches them with a pre-constructed identifier index library, achieving efficient and accurate cross-modal retrieval. The identifier index library is constructed based on sample datasets of multiple preset modalities and the multimodal large language model, improving retrieval matching efficiency while reducing the complexity of modality conversion and feature alignment. The query samples support single-modal or multimodal input, making it applicable to a wider range of scenarios, and the retrieval results are directly associated with sample data from modalities such as images.

[0038] In the embodiments of this application, the identifier index library is constructed in the following manner: Based on a pre-built prompt template, multiple sample datasets are processed to obtain a model input sequence. The prompt template is constructed according to a predetermined input format and a predetermined output format. The input format of the prompt template is used to constrain the modality of the processed multiple sample datasets. The model input sequence is input into a multimodal large language model to obtain a structured semantic identifier corresponding to each sample dataset. The output format of the prompt template is used to constrain the output format of the multimodal large language model. The semantic identifier has the same format as the output format of the prompt template. Based on the semantic identifier corresponding to each sample dataset, the association relationship between each sample dataset and the corresponding semantic identifier is determined. Based on the association relationship, an identifier index library is constructed.

[0039] The prompt templates can handle both image and text modalities and employ a fixed, structured format, including task descriptions, constraints, response templates, and examples, to constrain the expected output format and content of the multimodal large language model.

[0040] By pre-configuring the order of fields, the form of separators, and key trigger words in the prompt template, the multimodal large language model can output the corresponding content in sequence according to the pre-defined structural rules during the generation process, thereby improving the performance of the generated content in terms of structural consistency and output stability.

[0041] For example, by inputting the model input sequence into a multimodal large language model, structured semantic identifiers and corresponding semantic explanations can be sequentially output through autoregressive sequence generation. Specifically, the process of generating semantic identifiers by a multimodal large language model can be represented by the following formula;

[0042]

[0043] In the formula, For the first The model input sequence of a sample dataset, For multimodal large language models, For the first Structured semantic identifiers generated from a sample dataset, This provides semantic explanation information for the generated structured semantic identifiers.

[0044] The cross-modal retrieval method based on this application constrains the input modality and output format of semantic identifiers in the sample dataset by using preset prompt templates. This ensures that the semantic identifiers generated by the multimodal large language model have a unified and standardized structure, improving the data consistency of the identifier index library and avoiding matching errors caused by format differences. The standardization of the sample dataset and the structured generation of identifiers are achieved by relying on prompt templates, simplifying the feature alignment and format conversion process for building the identifier index library and reducing implementation complexity. Simultaneously, building the index library based on explicit relationships ensures accurate mapping between semantic identifiers and the sample dataset, further improving retrieval efficiency and accuracy.

[0045] In embodiments of this application, the input format includes a concatenation order and placeholders corresponding to each modality; based on a pre-built prompt template, multiple sample datasets are processed to obtain a model input sequence, including: for any sample dataset, encoding the sample dataset to obtain the encoding features of the sample dataset; based on the encoding features, determining the target placeholder corresponding to the modality of the sample dataset from multiple placeholders; concatenating the sample dataset with the target placeholder to obtain a concatenated sample dataset; and concatenating multiple concatenated sample datasets a second time based on the concatenation order to obtain the model input sequence.

[0046] For example, when the sample dataset includes images and corresponding query text (which can be empty), the following formula can be used to concatenate the prompt template with the sample dataset.

[0047]

[0048] In the formula, For the prompt template, For the first One image, In order to be with the first The query text corresponding to each image.

[0049] The model input sequence serves as the standardized input for the multimodal large language model, ensuring that sample datasets from different modalities have a consistent sequence structure during the input stage, which facilitates unified modeling and processing in subsequent stages.

[0050] The cross-modal retrieval method based on the embodiments of this application extracts modal features from the sample dataset through encoding, thereby achieving accurate identification and classification of the modalities in the sample dataset. Based on the encoded features, target placeholders are matched, and then the model input sequence is constructed through two concatenations. This not only clearly marks the modal attributes of each sample using placeholders, but also standardizes the input structure of multimodal samples by fixing the concatenation order, enabling the multimodal large language model to accurately parse the semantic relationships between sample datasets of different modalities.

[0051] In embodiments of this application, semantic identifiers include a first semantic identifier and a second semantic identifier; association relationships include a first association relationship and a second association relationship; based on the association relationships, constructing an identifier index library includes: when the semantic identifier is a first semantic identifier, extracting a first identifier vector of the first semantic identifier and a sample vector of the sample dataset; determining the similarity between the first identifier vector and the sample vector, and establishing a first association relationship between the first semantic identifier and the sample dataset when the similarity is greater than a preset similarity threshold; when the semantic identifier is a second semantic identifier, obtaining the hash value of the second semantic identifier and the hash value of the first semantic identifier; establishing a second association relationship between the second semantic identifier and the sample dataset when the difference between the hash value of the second semantic identifier and the hash value of the first semantic identifier is within a preset difference threshold range; and constructing an identifier index library based on the first association relationship and the second association relationship.

[0052] For example, using the generated structured semantic identifiers as the keys of the index and the corresponding sample datasets of the semantic identifiers as the values, an offline mapping relationship between the structured semantic identifiers and their corresponding sample datasets can be established. The identifier index library can be represented by the following formula;

[0053]

[0054] In the formula, For the first Structured semantic identifiers generated from a sample dataset, In order to be with the first One image, For identifier index library, This represents the number of sample datasets.

[0055] The identifier index is stored in key-value pairs and used for fast matching in subsequent retrieval stages.

[0056] The cross-modal retrieval method based on the embodiments of this application employs differentiated association strategies for different types of semantic identifiers. For the first semantic identifier, association is established through vector similarity, ensuring the accuracy of semantic matching; for the second semantic identifier, association is established through hash value difference comparison, improving the efficiency of association establishment. These dual association relationships complement each other, ensuring both the accuracy of the mapping between semantic identifiers and sample datasets and the efficiency of identifier index construction.

[0057] In the embodiments of this application, the semantic identifier is composed of at least two semantic units combined according to a predetermined combination rule. The semantic units are target object semantic units, behavior action semantic units, or scene environment semantic units.

[0058] For example, target object semantic units can include subjects such as people and objects in the sample dataset. Behavior / action semantic units can include the subject's behavior or state, such as walking, displaying, or running. Scene / environment semantic units can include the background or scene in the sample dataset, such as indoor, outdoor, or shopping mall.

[0059] Predefined combination rules can be pre-defined semantic unit concatenation and sorting logic, used to standardize the combination forms of different types of semantic units and ensure the structural uniformity and parsability of semantic identifiers.

[0060] The cross-modal retrieval method based on embodiments of this application achieves hierarchical and refined representation of sample semantics by decomposing semantic identifiers into combinations of multiple types of semantic units, overcoming the limitations of single-dimensional semantic encoding. By standardizing the combination of semantic units through preset combination rules, the uniformity of semantic identifier structure is ensured, facilitating the storage and matching of the identifier index. Accurate comparison of local semantic units is achieved, improving the flexibility and matching accuracy of cross-modal retrieval.

[0061] In the embodiments of this application, the sample to be queried is input into a multimodal large language model, and the structured semantic identifier corresponding to the sample to be queried is output. This includes: processing the multimodal large language model based on a preset identifier generation strategy to generate multiple candidate semantic identifiers corresponding to the sample to be queried; matching the multiple candidate semantic identifiers with a pre-constructed tree constraint structure, which is constructed using multiple semantic units; and based on the matching results, selecting candidate semantic identifiers that meet preset matching conditions from the multiple candidate semantic identifiers as the structured semantic identifiers corresponding to the sample to be queried.

[0062] For example, when a sample to be queried is input into a multimodal large language model, a multi-path candidate generation strategy can be used to generate multiple structured candidate semantic identifiers. The generation process of candidate semantic identifiers can be constrained based on a pre-built tree constraint structure, so that the candidate results are limited to the identifier index space and the generated results are prevented from deviating from the expected semantic space.

[0063] For example, for a query sample of "a dog running on the lawn", the candidate semantic identifiers generated by the path candidate generation strategy can include "husky-lawn-running", "husky-beach-running", "husky-lawn-walking", etc. From multiple candidate semantic identifiers, the candidate semantic identifier that meets the preset matching conditions can be selected as "husky-lawn-running".

[0064] The cross-modal retrieval method based on the embodiments of this application generates multiple candidate semantic identifiers through a preset identifier generation strategy, expanding the coverage of semantic representation and avoiding semantic bias of a single semantic identifier. A tree-structured constraint is used to match and filter candidate identifiers, verifying their semantic completeness and logical rationality, thus achieving standardized filtering of candidate semantic identifiers and effectively improving the accuracy and standardization of semantic identifiers.

[0065] In the embodiments of this application, the tree constraint structure is constructed in the following manner: each semantic unit is instantiated as a node, and constraints are configured for each node, including the dependencies between each node and the semantic type of each node; based on the constraints, the connection relationships of each node are determined; and based on the nodes and connection relationships, the tree constraint structure is constructed.

[0066] For example, a tree-structured constraint includes multiple nodes, including a root node and leaf nodes. Taking a girl running in an outdoor scene as an example, the semantic units include "outdoor" (i.e., the scene environment semantic unit), "girl" (i.e., the target object semantic unit), and "running" (i.e., the behavior action semantic unit). Instantiating the semantic units as nodes, the root node is the overall semantic node, and the leaf nodes include the scene environment node (corresponding to "outdoor"), the target object node (corresponding to "girl"), and the behavior action node (corresponding to "running").

[0067] For example, a dependency relationship could be that the root node is the parent node of all leaf nodes, and there is a hierarchical dependency between the root node and all leaf nodes. All leaf nodes are child nodes, and each child node is related to others but has no hierarchical dependency.

[0068] For example, semantic types can be such that scene environment nodes can only represent environmental information, target object nodes can only represent subject information, and behavior action nodes can only represent action information.

[0069] For example, the root node establishes a one-way connection with the three leaf nodes (i.e., the scene environment node, the target object node, and the behavior action node), and there is no connection between the three leaf nodes, forming a tree-shaped constraint structure of root node-leaf node.

[0070] The cross-modal retrieval method based on the embodiments of this application instantiates semantic units as nodes and configures constraints, enabling the tree-structured constraint structure to possess clear hierarchical logic and semantic verification capabilities. By using the tree-structured constraint structure to filter semantic identifiers, candidate identifiers with missing semantics or incorrect types can be accurately identified, improving the semantic completeness and logical accuracy of semantic identifiers.

[0071] In the embodiments of this application, the tree constraint structure includes multiple connecting edges; based on the matching results, candidate semantic identifiers that meet preset matching conditions are selected from multiple candidate semantic identifiers as the structured semantic identifiers corresponding to the queried sample, including: determining the generation probability of each candidate semantic identifier and the semantic unit corresponding to each candidate semantic identifier; instantiating multiple semantic units corresponding to each candidate semantic identifier into multiple candidate nodes; determining the candidate connecting edges corresponding to each candidate semantic identifier based on multiple candidate nodes and constraint conditions; when there is a connecting edge that is the same as the candidate connecting edge among multiple connecting edges, the candidate semantic identifier with the highest generation probability is selected as the structured semantic identifier corresponding to the queried sample.

[0072] The cross-modal retrieval method based on the embodiments of this application filters candidate semantic identifiers through a dual dimension of generation probability and tree constraint structure, ensuring that the selected identifiers have both high generation credibility and semantic logic correctness.

[0073] In embodiments of this application, the cross-modal retrieval method further includes: obtaining semantic interpretation information of the semantic identifier corresponding to the query sample; constructing a training sample pair using the semantic identifier corresponding to the query sample as the key and the semantic interpretation information as the value; performing secondary training on the multimodal large language model based on the training sample pair; and updating the identifier index library using the secondary trained multimodal large language model.

[0074] The following is combined Figure 2 The training of the multimodal large language model in this embodiment will be described in detail.

[0075] Figure 2 The diagram illustrates the structure of training a multimodal large language model according to an embodiment of this application.

[0076] like Figure 2 As shown, the training of the multimodal large language model in this embodiment includes steps S210 to S260.

[0077] In step S210, multiple sample datasets and prompt templates are concatenated to obtain the model input sequence.

[0078] In step S220, the model input sequence is input into the multimodal large language model to obtain semantic identifiers and semantic interpretation information corresponding to each sample dataset.

[0079] In step S230, a training sample pair is constructed using the semantic identifier corresponding to the sample to be queried as the key and the semantic explanation information as the value.

[0080] For example, the samples to be queried may include sample dataset A and sample dataset B. Sample dataset A, the semantic identifiers and semantic explanations corresponding to sample dataset A constitute training sample pair A, and sample dataset B, the semantic identifiers and semantic explanations corresponding to sample dataset B constitute training sample pair B.

[0081] In step S240, the multimodal large language model is trained a second time based on the training sample pairs.

[0082] For example, structured semantic identifiers and semantic explanation information can be concatenated to form the target output sequence. This sequence generation training can be used to enhance the multimodal large language model's ability to model the semantic correspondence between input samples and structured semantic identifiers, thereby improving the cross-modal retrieval performance of the multimodal large language model. Specifically, the multimodal large language model can be trained a second time using the following formula;

[0083]

[0084]

[0085] In the formula, This is the semantic identifier corresponding to the sample to be queried. This provides semantic explanation information for the semantic identifier corresponding to the sample to be queried. Output the target sequence. This is the semantic identifier corresponding to the sample to be queried. These are the training parameters for a multimodal large language model. The loss value for a multimodal large language model. Samples sampled from each training sample pair Take the negative after calculating the expectation. For training sample pairs, For the training sample pair set, Based on and The Middle Predict the i-th element from all elements before the i-th position. The probability of an element at each position.

[0086] In step S250, the sample to be queried is input into the trained multimodal large language model to obtain the semantic interpretation information of the semantic identifier corresponding to the sample to be queried.

[0087] In step S260, based on the semantic identifier of the sample to be queried, a match is performed in the identifier index library, and the query result corresponding to the sample to be queried is determined according to the matching result.

[0088] For example, the following formula can be used to perform a match in an identifier index;

[0089]

[0090] In the formula, The semantic identifier for the sample to be queried. The query results are the samples to be queried. This is a key-value pair mapping.

[0091] The cross-modal retrieval method based on the embodiments of this application updates the identifier index library after secondary training of the multimodal large language model using training sample pairs composed of semantic identifiers and semantic explanation information. This makes the semantic matching during retrieval more accurate, reduces the false matching rate of cross-modal retrieval, and makes the semantic association between the query results and the query sample closer.

[0092] Based on the above-described cross-modal retrieval method, this application also provides a cross-modal retrieval device. The following will combine... Figure 3 The device is described in detail.

[0093] Figure 3 A schematic block diagram of a cross-modal retrieval device according to an embodiment of this application is shown.

[0094] like Figure 3 As shown, the cross-modal retrieval device 300 of this embodiment includes a sample acquisition module 310, an identifier output module 320, and an identifier query module 330.

[0095] The sample acquisition module 310 is used to acquire the query sample, a pre-trained multimodal large language model, and an identifier index library. The identifier index library is constructed using sample datasets of multiple preset modalities and the multimodal large language model. The query sample has at least one preset modality among the multiple preset modalities. In one embodiment, the sample acquisition module 310 can be used to perform step S110 described above, which will not be repeated here.

[0096] The identifier output module 320 is used to input the query sample into the multimodal large language model and output a structured semantic identifier corresponding to the query sample. In one embodiment, the identifier output module 320 can be used to perform step S120 described above, which will not be repeated here.

[0097] The identifier query module 330 is used to perform matching in the identifier index library based on semantic identifiers, and determine the query result corresponding to the sample to be queried based on the matching result. In one embodiment, the identifier query module 330 can be used to execute step S130 described above, which will not be repeated here.

[0098] According to embodiments of this application, any multiple modules among the sample acquisition module 310, the identifier output module 320, and the identifier query module 330 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the sample acquisition module 310, the identifier output module 320, and the identifier query module 330 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPMA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the sample acquisition module 310, the identifier output module 320, and the identifier query module 330 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0099] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a cross-modal retrieval method according to an embodiment of this application.

[0100] like Figure 4 As shown, the electronic device 400 suitable for implementing the cross-modal retrieval method in this embodiment includes a bus 401, a memory 402, an image processor 403, a central processing unit 404, an input device 405, and an output device 406.

[0101] The memory 402, image processor 403, central processing unit 404, input device 405, and output device 406 are interconnected via bus 401. The memory 402 may be a non-transitory computer-readable storage medium, and the memory 402 stores a computer program that can be executed by the image processor 403 or the central processing unit 404 to enable the cross-modal retrieval method provided in this application to be executed.

[0102] Input device 405 is used to provide and receive control signals input to electronic device 400, including a keyboard that generates numeric or character information, and a mouse or touch device for controlling the device to generate other functional signals. Output device 406 is used to provide feedback information to electronic device 400, including a display that shows the execution results or execution process.

[0103] It should be understood that Figure 4The number of memory, image processor, central processing unit, input devices, and output devices shown is merely illustrative. Depending on implementation requirements, any number of memory, image processor, central processing unit, input devices, and output devices can be included.

[0104] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0105] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include one or more memories other than the memory 402 described above.

[0106] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the abnormal behavior handling method provided in the embodiments of this application.

[0107] When the computer program is executed by the image processor 403 or the central processing unit 404, it performs the functions defined in the apparatus of the embodiments of this application. According to the embodiments of this application, the apparatus, modules, etc. described above can be implemented by computer program modules.

[0108] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices or magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed as signals over a network medium, and downloaded and installed via a communication component, and / or installed from a removable medium. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0109] In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by the image processor 403 or the central processing unit 404, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0110] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0112] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined or combined in various ways without departing from the spirit and teachings of this application. All such combinations or combinations fall within the scope of this application.

[0113] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A cross-modal retrieval method, characterized in that, include: The system acquires a query sample, a pre-trained multimodal large language model, and an identifier index library. The identifier index library is constructed using a sample dataset of multiple preset modalities and the multimodal large language model. The query sample has at least one preset modality among the multiple preset modalities. The query sample is input into the multimodal large language model, and the structured semantic identifier corresponding to the query sample is output. Based on the semantic identifier, a match is performed in the identifier index library, and the query result corresponding to the sample to be queried is determined according to the matching result.

2. The method according to claim 1, characterized in that, The identifier index library is constructed in the following manner: Based on a pre-built prompt template, multiple sample datasets are processed to obtain a model input sequence. The prompt template is constructed according to a predetermined input format and a predetermined output format. The input format of the prompt template is used to constrain the modalities of the multiple sample datasets after processing. The input sequence of the model is input into the multimodal large language model to obtain a structured semantic identifier corresponding to each sample dataset. The output format of the prompt template is used to constrain the output format of the multimodal large language model. The semantic identifier has the same format as the output format of the prompt template. Based on the semantic identifier corresponding to each of the sample datasets, determine the association between each sample dataset and the corresponding semantic identifier; Based on the aforementioned association, the identifier index library is constructed.

3. The method according to claim 2, characterized in that, The input format includes the concatenation order and a placeholder corresponding to each modality; Based on a pre-built prompt template, multiple sample datasets are processed to obtain the model input sequence, including: For any of the sample datasets, the sample dataset is encoded to obtain the encoded features of the sample dataset; Based on the encoding features, a target placeholder corresponding to the modality of the sample dataset is determined from a plurality of placeholders; The sample dataset and the target placeholder are concatenated to obtain the concatenated sample dataset; Based on the splicing order, multiple spliced ​​sample datasets are spliced ​​a second time to obtain the model input sequence.

4. The method according to claim 2, characterized in that, The semantic identifier includes a first semantic identifier and a second semantic identifier; the association relationship includes a first association relationship and a second association relationship; Based on the aforementioned association, the identifier index library is constructed, including: If the semantic identifier is a first semantic identifier, extract the first identifier vector of the first semantic identifier and the sample vector of the sample dataset; Determine the similarity between the first identifier vector and the sample vector, and if the similarity is greater than a preset similarity threshold, establish a first association between the first semantic identifier and the sample dataset; If the semantic identifier is a second semantic identifier, obtain the hash value of the second semantic identifier and the hash value of the first semantic identifier; If the difference between the hash value of the second semantic identifier and the hash value of the first semantic identifier is within a preset difference threshold range, a second association relationship between the second semantic identifier and the sample dataset is established. Based on the first association relationship and the second association relationship, the identifier index library is constructed.

5. The method according to claim 1, characterized in that, The semantic identifier is composed of at least two semantic units combined according to a predetermined combination rule. The semantic units are target object semantic units, behavior action semantic units, or scene environment semantic units.

6. The method according to claim 5, characterized in that, The query sample is input into the multimodal large language model, which outputs a structured semantic identifier corresponding to the query sample, including: The multimodal large language model is processed based on a preset identifier generation strategy to generate multiple candidate semantic identifiers corresponding to the query sample. The plurality of candidate semantic identifiers are matched with a pre-constructed tree constraint structure, which is constructed using the plurality of semantic units; Based on the matching results, candidate semantic identifiers that meet the preset matching conditions are selected from the multiple candidate semantic identifiers and used as the structured semantic identifiers corresponding to the sample to be queried.

7. The method according to claim 6, characterized in that, The tree-shaped constraint structure is constructed in the following way: Each semantic unit is instantiated as a node, and constraints are configured for each node, including the dependencies between each node and the semantic type of each node; Based on the constraints, the connection relationships of each node are determined; Based on the nodes and the connection relationships, the tree constraint structure is constructed.

8. The method according to claim 6, characterized in that, The tree-shaped constraint structure includes multiple connecting edges; based on the matching results, candidate semantic identifiers that meet preset matching conditions are selected from the multiple candidate semantic identifiers and used as the structured semantic identifiers corresponding to the queried sample, including: Determine the generation probability of each candidate semantic identifier and the semantic unit corresponding to each candidate semantic identifier; Each candidate semantic identifier is instantiated into multiple candidate nodes. Based on the multiple candidate nodes and the constraints, determine the candidate connection edge corresponding to each candidate semantic identifier; When there is a connection edge that is the same as the candidate connection edge among the multiple connection edges, the candidate semantic identifier with the highest probability will be generated as the structured semantic identifier corresponding to the query sample.

9. The method according to claim 1, characterized in that, The method further includes: Obtain the semantic interpretation information of the semantic identifier corresponding to the sample to be queried; Using the semantic identifier corresponding to the sample to be queried as the key and the semantic explanation information as the value, a training sample pair is constructed; Based on the training sample pairs, the multimodal large language model is trained a second time; The identifier index is updated using the multimodal large language model after secondary training.

10. A cross-modal retrieval device, characterized in that, include: The sample acquisition module is used to acquire the sample to be queried, a pre-trained multimodal large language model, and an identifier index library. The identifier index library is constructed using sample datasets of multiple preset modalities and the multimodal large language model. The sample to be queried has at least one preset modality among the multiple preset modalities. The identifier output module is used to input the query sample into the multimodal large language model and output a structured semantic identifier corresponding to the query sample. The identifier query module is used to match the semantic identifier in the identifier index library and determine the query result corresponding to the sample to be queried based on the matching result.

Images