Multi-modal retrieval model generation method and apparatus, device, and storage medium

Abstract

Disclosed in embodiments of the present application are a multi-modal retrieval model generation method and apparatus, a device, and a storage medium. The embodiments of the present application can be applied to various scenarios such as artificial intelligence, intelligent transportation, and assisted driving. The multi-modal retrieval model generation method comprises: by means of a prefix vector module, performing feature recognition on first sample image modal data to obtain a first image prefix vector; by means of a modal identification module and on the basis of the first image prefix vector, the first sample image modal data, and first sample text modal data, generating a first retrieval character; by means of a constrained decoding module, acquiring from a pre-generated database a first sample retrieval result associated with the first retrieval character; and on the basis of a first reference retrieval result and the first sample retrieval result, adjusting the model parameters corresponding to the prefix vector module and the constrained decoding module to obtain a retrieval model after the adjustment. The present application can improve the training efficiency of multi-modal retrieval models.

Description

A method, device, equipment and storage medium for generating a multimodal retrieval model

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 29, 2024, with application number 2024102346489 and application name “A data processing method, device, equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for generating a multimodal retrieval model. Background Art

[0003] With the development of internet technology and the ever-increasing scale of online data, modal data retrieval technology can meet user data retrieval needs such as question-and-answer and conversational data. Recently, there has been a trend in modal data retrieval technology from a single modality to multimodal data, such as images and text. Multimodal retrieval models that can accurately retrieve single or multimodal data can greatly facilitate user data retrieval needs.

[0004] At present, during the training process of the multimodal retrieval model, all model parameters in the multimodal retrieval model need to be adjusted according to the model loss of the multimodal retrieval model. Therefore, large-scale high-quality sample data is required for training, resulting in high model training cost and low training efficiency of the multimodal retrieval model. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, device, and storage medium for generating a multimodal retrieval model, which can improve the training efficiency of the multimodal retrieval model and reduce the training cost of the multimodal retrieval model.

[0006] On the one hand, an embodiment of the present application provides a method for generating a multimodal retrieval model, including:

[0007] Acquire an initial retrieval model, wherein the initial retrieval model includes a modality recognition module, a prefix vector module, and a constraint decoding module;

[0008] Obtaining a first sample retrieval request, the first sample retrieval request including mutually associated first sample text modality data and first sample image modality data, and including a first reference retrieval result pre-annotated for the first sample retrieval request;

[0009] Performing feature recognition on the first sample image modality data by the prefix vector module to obtain a first image prefix vector;

[0010] Generate a first search character according to the first image prefix vector, the first sample image modality data, and the first sample text modality data by the modality recognition module;

[0011] Obtaining, by the constraint decoding module, a first sample search result associated with the first search character from a pre-generated database;

[0012] According to the first reference retrieval result and the first sample retrieval result, the model parameters corresponding to the prefix vector module and the constraint decoding module are adjusted respectively to obtain an adjusted retrieval model.

[0013] In one aspect, an embodiment of the present application provides a device for generating a multimodal retrieval model, comprising:

[0014] An acquisition module is configured to acquire an initial retrieval model, the initial retrieval model comprising a modality recognition module, a prefix vector module, and a constraint decoding module; acquire a first sample retrieval request, the first sample retrieval request comprising mutually associated first sample text modality data and first sample image modality data, and comprising a first reference retrieval result pre-annotated for the first sample retrieval request;

[0015] a first generating module, configured to perform feature recognition on the first sample image modality data through the prefix vector module to obtain a first image prefix vector;

[0016] a second generating module, configured to generate a first search character according to the first image prefix vector, the first sample image modal data, and the first sample text modal data through the modality recognition module;

[0017] A first retrieval module, configured to obtain a first sample retrieval result associated with the first retrieval character from a pre-generated database through the constraint decoding module;

[0018] An adjustment module is used to adjust the model parameters corresponding to the prefix vector module and the constraint decoding module respectively according to the first reference retrieval result and the first sample retrieval result to obtain an adjusted retrieval model.

[0019] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the method provided by the embodiment of the present application.

[0020] In one aspect, embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the embodiments of the present application.

[0021] In an embodiment of the present application, the initial retrieval model has a strong content comprehension ability and language generation ability, and has a large processing potential for data retrieval. Therefore, in the training process of the initial retrieval model, the model parameters of the original module (modal recognition module) in the initial retrieval model remain unchanged, and only the model parameters of the prefix vector module and the constraint decoding model need to be lightly fine-tuned so that the prefix vector module and the constraint decoding model can learn knowledge about data retrieval and obtain a multimodal retrieval model. This can reduce the adjustment of model parameters, reduce the training cost of the multimodal retrieval model, and improve the training efficiency of the multimodal retrieval model. By keeping the model parameters of the original module (i.e., modal recognition) in the initial retrieval model unchanged, the content comprehension ability and generation ability of the initial retrieval model can be reused, and the problem of catastrophic forgetting of the generative language module due to the update of training data can be avoided, and the training stability of the multimodal retrieval model can be ensured. During the training of the initial retrieval model, only lightweight fine-tuning of the constraint decoding module and the prefix vector module is required. Therefore, a large amount of sample data is not required, which can avoid the problem of low model training accuracy caused by insufficient sample data. This is conducive to improving the accuracy of model training and, in turn, improving the retrieval accuracy of the multimodal model during the retrieval process. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] FIG1 is a schematic diagram of the structure of a system for generating a multimodal retrieval model according to an embodiment of the present application;

[0024] FIG2 is a schematic diagram of a training method for a multimodal classification model provided in an embodiment of the present application;

[0025] FIG3 is a flow chart of a method for generating a multimodal retrieval model provided in an embodiment of the present application;

[0026] FIG4 is a schematic diagram of obtaining a first search character provided by an embodiment of the present application;

[0027] FIG5 is a schematic diagram of searching a first sample search result according to a first search character provided by an embodiment of the present application;

[0028] FIG6 is a schematic diagram of a multimodal retrieval model for generative retrieval provided in an embodiment of the present application;

[0029] FIG7 is a flow chart of a method for generating a multimodal retrieval model provided in an embodiment of the present application;

[0030] FIG8 is a schematic structural diagram of a device for generating a multimodal retrieval model according to an embodiment of the present application;

[0031] FIG9 is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0033] This application relates to the field of artificial intelligence technology. Specifically, embodiments of this application can train an initial retrieval model to which a prefix vector module and a constraint decoding module are added to obtain a multimodal retrieval model, thereby reducing the model training cost and improving the training efficiency of the multimodal retrieval model. Furthermore, the multimodal retrieval model can be used for single-modal data retrieval or multimodal data retrieval, thereby improving the accuracy and efficiency of data retrieval.

[0034] Please refer to Figure 1, which is a structural diagram of a system for generating a multimodal retrieval model provided in an embodiment of the present application. As shown in Figure 1, the system for generating a multimodal retrieval model may include a server 10 and a terminal device cluster. The terminal device cluster may include one or more terminal devices, and the number of terminal devices is not limited here. As shown in Figure 1, it may specifically include terminal device 100a, terminal device 100b, terminal device 100c, ..., terminal device 100n. As shown in Figure 1, terminal device 100a, terminal device 100b, terminal device 100c, ..., terminal device 100n can be respectively connected to the above-mentioned server 10 through a network, so that each terminal device can exchange data with the server 10 through the network connection. Of course, terminal device 100a, terminal device 100b, terminal device 100c, ..., terminal device 100n can communicate with each other through direct network connection, that is, point-to-point communication can be achieved between each terminal device; that is, when data interaction is required between each two terminal devices, one terminal device (i.e., the sending terminal device) can send data directly to another terminal device (i.e., the receiving terminal).

[0035] Among them, each terminal device in the terminal device cluster may include: smart phones, tablet computers, laptop computers, desktop computers, intelligent voice interaction devices, smart home appliances (for example, smart TVs), wearable devices, vehicle-mounted terminals and other smart terminals with data processing functions. It should be understood that each terminal device in the terminal device cluster shown in Figure 1 can be installed with an application with a modal data processing function. When the application is running in each terminal device, it can interact with the server 10 shown in Figure 1 above. For example, the application can specifically include a multimodal feature extraction application, a multimodal retrieval application, etc. For ease of understanding, the embodiment of the present application can select a terminal device from the multiple terminal devices shown in Figure 1 as the target terminal device. For example, the embodiment of the present application can use the terminal device 100a shown in Figure 1 as the target terminal device. The target terminal device can be installed with an application with a modal data processing function. At this time, the target terminal device can interact with the server 10 through the application in the target terminal device.

[0036] As shown in FIG1 , the server 10 can be a device that provides background services for applications in terminal devices. The server 10 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0037] It should be understood that the generation system based on a multimodal retrieval model in FIG1 can be applied to multimodal retrieval model training scenarios. It should be understood that multimodality can be used to indicate that the input data of the multimodal retrieval model has multiple data forms, that is, the multimodal retrieval model can be used to process data in multiple data forms, which may include voice, image, video, text, etc. Alternatively, multimodality can be used to indicate that the input data of the multimodal retrieval model includes data from multiple information sources, that is, the multimodal retrieval model can be used to process data from multiple information sources, which may include different sensors such as radar, infrared, accelerometer, and camera.

[0038] The multimodal retrieval model in the embodiment of the present application can be used for data retrieval of a single modality (such as text modality or image modality, etc.). For example, when the multimodal retrieval model is applied to data retrieval of a single modality, the multimodal retrieval model can be used to perform data retrieval on the text retrieval request (i.e., text modality data, such as text questions) input by the business object to obtain the retrieval document data corresponding to the text retrieval request. The retrieval document data corresponding to the text modality retrieval request can be used as the answer to the text retrieval request to display the corresponding retrieval document data to the business object.

[0039] The multimodal retrieval model in the embodiment of the present application can be used for multimodal (such as any one modality such as text modality or image modality) data retrieval. For example, when the multimodal retrieval model is applied to multimodal data retrieval, the multimodal retrieval model can be used to perform data retrieval on the multimodal retrieval request of the business object (that is, multimodal data, such as a combination of image modality data and text modality data), obtain the retrieval document data corresponding to the multimodal retrieval request, and display the retrieval document data corresponding to the multimodal retrieval request to the business object.

[0040] Among them, the initial retrieval model can be obtained by adding a prefix vector module and a constraint decoding module to a trained generative language model. The trained generative language model can refer to a generative language model that has been pre-trained and meets the convergence condition. The convergence condition can refer to that the model loss is less than or equal to the loss threshold, or that the number of model training times is greater than or equal to the number threshold. The trained generative language model has strong content understanding and language generation capabilities, and has great processing potential for data retrieval. The generative language model includes a modality recognition module, which can be used to perform image feature recognition on the first sample image modality data and text feature recognition on the first sample text modality data, and generate a retrieval character associated with the retrieval request, and the retrieval character is used to retrieve the corresponding retrieval document data. Specifically, the modality recognition module can include a visual representation submodule for performing image feature recognition on the image modality data, and a language recognition submodule for performing text feature recognition on the text modality data.

[0041] Wherein, the visual representation submodule may include N Transformer layer structures, which are used to perform image feature recognition on the first sample image modal data to obtain a visual representation vector of the first sample image modal data. Wherein, Transformer is a neural network that learns context and thus learns meaning by identifying relationships in input sequence data. Wherein, the language recognition submodule may be a large language model (i.e., Large Language Model) structure, which is a statistical model used to predict the probability of a series of words in a text sequence, and to understand the language and predict the next word in the sequence by training on large-scale text data. The language recognition submodule may include network structures such as GPT network structure (i.e., Generative Pre-Training, a natural language processing model based on deep learning), BERT network structure (Bidirectional Encoder Representations from Transformer, an unsupervised pre-training language model for natural language processing tasks).

[0042] It is understandable that the embodiment of the present application can add the prefix vector module and the constraint decoding module associated with data retrieval to the trained generative language model, so that the initial retrieval model can include the modal recognition module, the prefix vector module and the constraint decoding model. By training the added generative language model, by keeping the model parameters of the original module (i.e., modal recognition) in the initial retrieval model unchanged, the content understanding ability and generation ability of the initial retrieval model can be reused, so that the prefix vector module and the constraint decoding model learn knowledge about data retrieval and obtain a multimodal retrieval model. It is understandable that the embodiment of the present application can adjust and improve the image feature recognition of image modal data by the modal recognition module through the prefix vector module to improve the accuracy of the image feature recognition of image modal data by the modal recognition module.

[0043] A sample retrieval request refers to a retrieval request for training the added generative language model, which may include single modal data (such as text questions, language questions, etc.) or multimodal data (multimodal questions combining text and images, multimodal questions combining text and video, etc.). The association between the first sample text modal data and the first sample image modal data may mean that the first sample text modal data is a text question generated for the first sample image modal data. In other words, the first sample text modal data is used to indicate the retrieval of a portion of the data in the first sample image modal data. For example, the first sample image modal data is an image including a motorcycle, and the first sample text modal data can be used to retrieve the motorcycle in the first sample image modal data (such as retrieving the brand of the motorcycle in the first sample image modal data).

[0044] Specifically, taking the training of the generated language model after the addition of the first sample retrieval request including the first sample text modal data and the first sample image modal data with an associated relationship as an example, the prefix vector model can generate a first image prefix vector based on the first sample image modal data. The modality recognition module can generate a first retrieval character based on the first image prefix vector, the first sample image modal data and the first sample text modal data. It can be understood that the first image prefix vector can be used as a prefix of the image modal feature of the first sample image modal data to achieve accurate extraction of the image modal feature of the first sample image modal data. The constrained decoding model can retrieve the first sample retrieval result associated with the first sample retrieval request based on the first retrieval character input by the modality recognition module, that is, the first sample retrieval result is the retrieval result output by the initial retrieval model based on the first sample retrieval request.

[0045] Furthermore, the initial retrieval model can be trained based on the difference between the first sample retrieval result output by the initial retrieval model (i.e., the retrieval result output by the initial retrieval model) and the first reference retrieval result (i.e., the annotated retrieval result). Specifically, since the modal recognition module has been trained, the model parameters corresponding to the modal recognition module are frozen. This can reduce the adjustment of the model parameters and avoid catastrophic forgetting of the modal recognition module due to the update of training data (e.g., the model parameters in the modal recognition module are chaotically adjusted), thereby ensuring high efficiency and stability of model training.

[0046] Then, based on the first reference retrieval result and the first sample retrieval result, the model parameters corresponding to the prefix vector module and the constrained decoding model in the initial retrieval model can be adjusted to obtain a multimodal retrieval model. In this way, only the parameters of the newly added prefix vector module and the constrained decoding model need to be adjusted, which can reduce the parameter adjustment range of the initial retrieval model after addition, and can also ensure that the trained multimodal retrieval model can accurately perform single modality data retrieval or multimodal data retrieval, thereby improving the training efficiency and training stability of the multimodal retrieval model.

[0047] As shown in FIG2 , FIG2 is a schematic diagram of a training multimodal retrieval model provided in an embodiment of the present application. As shown in FIG2 , the terminal device 201a, the terminal device 202a, the terminal device 203a, etc. in the terminal device cluster 20a can be the terminal device of the terminal device cluster in the embodiment corresponding to FIG1 above, and the server 20b shown in FIG2 can be the server 10 in the embodiment corresponding to FIG1 above. As shown in FIG2 , the terminal device 201a, the terminal device 202a, and the terminal device 203a in the terminal device cluster 20a can obtain the retrieval request input by the sample object, and use the retrieval request input by the sample object as a sample retrieval request, that is, as a model training sample. The retrieval request input by the business object can include a single modality retrieval request, such as a text retrieval request, an image retrieval request, a video retrieval request, and an audio retrieval request. Of course, the retrieval request input by the business object can also include a multimodal retrieval request, and the multimodal retrieval request can include a retrieval request composed of multimodal data such as text modality data, image modality data, and video modality data. Each terminal device in the terminal device cluster 20a can use the obtained retrieval request including text modal data and image modal data with an associated relationship as a first sample retrieval request, and refer to the text modal data included in the first sample retrieval request as first sample text modal data, and refer to the image modal data included in the first sample retrieval request as first sample image modal data.

[0048] The terminal device 201a, the terminal device 202a, and the terminal device 203a in the terminal device cluster 20a can send the first sample retrieval request obtained by each of them to the server 20b. The server 20b can add the prefix vector module and the constraint decoding module associated with the data retrieval to the trained generative language model to obtain an initial retrieval model. The trained generative language model can refer to a generative language model that has been pre-trained and meets the convergence conditions. The generative language model may include a modality recognition module, which is used to perform image feature recognition on the first sample image modality data and text feature recognition on the first sample text modality data. It can be understood that the initial retrieval model includes a prefix vector module, a constraint decoding module, and a modality recognition module.

[0049] Specifically, the server 20b can input the first sample image modal data into the prefix vector module in the initial retrieval model, and generate a first image prefix vector based on the first sample image modal data through the prefix vector module. At the same time, the server 20b can input the first sample image modal data, the first sample text modal data, and the first image prefix vector into the modal recognition module. The modal recognition module can perform feature extraction on the first sample image modal data to obtain the image modal feature vector of the first sample image modal data, and then use the first image prefix vector as the prefix of the image modal feature vector. The first image prefix vector and the image modal feature vector are used together to perform feature recognition on the first sample image modal data to obtain the visual representation vector corresponding to the first sample image modal data. At the same time, the modal recognition module performs text feature recognition on the first sample text modal data to obtain the text modal features corresponding to the first sample text modal data, and then generates the first search character based on the visual representation vector and the text modal features.

[0050] As shown in Figure 2, the modality recognition module includes a visual representation submodule for performing image feature recognition on image modality data, and a language recognition submodule for performing text feature recognition on text modality data. The server 20b can input the first image prefix vector and the first sample image modality data into the visual representation submodule, and perform image feature recognition on the first sample image modality data through the visual representation submodule to obtain a visual representation vector. Among them, the visual representation submodule can be trained and can accurately extract features from the image modality data. Further, the visual representation vector and the first sample text modality data are input into the language recognition submodule, and feature extraction is performed on the first sample text modality data to obtain text modality features, and then the text modality features and the visual representation vector are associated with feature extraction to generate the first search character. It can be understood that the language recognition submodule can refer to a large language model that has been trained. The large language model has powerful content understanding and character generation capabilities and has great potential for understanding complex search requests. In this way, it is possible to regard the large language model as a virtual knowledge base and effectively utilize the inherent knowledge of the large language model.

[0051] Furthermore, the server 20b can retrieve the first sample retrieval result associated with the first sample retrieval request according to the first retrieval character through the constraint decoding module, which can improve the retrieval efficiency and accuracy of the first sample retrieval result. It can be understood that the first sample retrieval result can be the retrieval result of the initial retrieval model for the first sample retrieval request, and the server 20b can obtain the first reference retrieval result, which can be the correct retrieval result corresponding to the first sample retrieval request. The server 20b can determine the model loss of the initial retrieval model based on the first sample retrieval result and the first reference retrieval result. Since the modal recognition module is pre-trained and meets the convergence conditions, that is, the modal recognition module has been trained, the model parameters corresponding to the modal recognition module can be frozen, that is, the model parameters corresponding to the modal recognition module are not adjusted. In this way, the adjustment of the model parameters can be reduced, and the problem of catastrophic forgetting of the modal recognition module due to the update of the training data can be avoided, which can ensure the high efficiency and stability of the model training.

[0052] The server 20b can adjust the model parameters corresponding to the prefix vector module and the constraint decoding module in the initial retrieval model according to the model loss until the initial retrieval model meets the convergence condition, thereby obtaining a multimodal retrieval model. It is understandable that since the modal recognition module has been trained and can be used to perform feature recognition on image modal data and text modal data respectively, the model parameters in the modal recognition module do not need to be adjusted. Only by adjusting the model parameters in the prefix vector module (i.e., a lightweight prefix fine-tuning method) and the model parameters in the constraint decoding module, a multimodal retrieval model for modal data retrieval is obtained. In this way, the range of model parameter adjustment can be reduced, lightweight model parameter fine-tuning can be achieved, the heavy training burden can be reduced, and the stability of model training can be ensured. Even in the case of limited labeled data, the intrinsic knowledge in the trained initial retrieval model can be effectively utilized to obtain a higher-performance multimodal retrieval model, which can improve the training efficiency of the multimodal retrieval model.

[0053] Further, please refer to Figure 3, which is a flow chart of a method for generating a multimodal retrieval model provided in an embodiment of the present application. As shown in Figure 3, the method can be executed by any terminal device in Figure 1, or by the server 10 in Figure 1, or by both the terminal device and the server in Figure 1. The devices used to execute the method for generating the multimodal retrieval model in the present application can be collectively referred to as computer devices. The method for generating the multimodal retrieval model may include, but is not limited to, the following steps:

[0054] S101: Obtain an initial retrieval model and a first sample retrieval request.

[0055] The first sample retrieval request includes mutually related first sample text modality data and first sample image modality data, and includes a first reference retrieval result pre-annotated for the first sample retrieval request.

[0056] Among them, the initial retrieval model includes a modality recognition module, a prefix vector module and a constraint decoding module. The initial retrieval model can be obtained by adding the prefix vector module and the constraint decoding module to the trained generative language model, or the initial retrieval model can be a retrieval model to be trained.

[0057] The first reference search result may refer to a search result obtained by manually annotating the first sample image modal data and the first sample text modal data in the first sample search request. Specifically, the first sample text modal data is a text question regarding the first sample image modal data, and the first reference search result is a reference answer to the text question indicated by the first sample text modal data. For example, if the first sample image modal data includes a motorcycle belonging to brand A, and the first sample text modal data is: Which brand does the motorcycle in the first sample image modal data belong to? Then, the first reference search result is: The motorcycle in the first sample image modal data belongs to brand A.

[0058] Specifically, the computer device can add the prefix vector module and the constraint decoding model associated with data retrieval to the trained generative language model to obtain an initial retrieval model, train the initial retrieval model, and obtain a multimodal retrieval model for modal data retrieval. Among them, the generative language model includes a modal recognition module, which can be used to perform image feature recognition on image modal data and text feature recognition on text modal data. Since the generative language model is trained (i.e., pre-trained and meets the convergence conditions), the model parameters in the modal recognition module do not need to be retrained and adjusted. Only the model parameters in the prefix vector module and the constraint decoding model need to be trained and adjusted, which can reduce the range of model parameter adjustment and thus improve the efficiency of multimodal retrieval model training. Among them, the prefix vector module can be used to adjust and improve the image feature recognition of the modal recognition module for the image modal data (i.e., adjust and improve the multi-granularity visual feature learning) to increase the accuracy of the modal recognition module for the image feature recognition of the image modal data. The constraint decoding model can accurately retrieve the corresponding retrieval document data based on the retrieval characters output by the modal recognition module.

[0059] Specifically, the computer device can obtain a retrieval request input by a sample object as a sample retrieval request, which can be used to request data retrieval for knowledge or objects that need to be understood, such as a retrieval request that can be a question, a query text, or a query image. The retrieval request can include a single modality data request, which can include any one modality data such as text modality data, image modality data, and video modality data. The retrieval request can also include a multimodality data request, which can include a combination of any multiple modality data such as text modality data, image modality data, and video modality data. Among them, the retrieval request including text modality data and image modality data with an associated relationship can be referred to as a first sample retrieval request, the text modality data in the first sample retrieval request can be referred to as first sample modality data, and the image modality data in the first sample retrieval request can be referred to as first sample image modality data. The association relationship between the first sample text modal data and the first sample image modal data may mean that the first sample text modal data is a text question generated for the first sample image modal data, that is, the association relationship indicates that the first sample text modal data is a text question generated for the first sample image modal data. In other words, the first sample text modal data is used to indicate the retrieval of a certain part of the first sample image modal data. For example, the first sample image modal data is an image including a motorcycle, and the first sample text modal data can be used to retrieve the motorcycle in the first sample image modal data (such as retrieving what brand the motorcycle in the first sample image modal data is).

[0060] S102 : Perform feature recognition on the first sample image modality data through the prefix vector module to obtain a first image prefix vector.

[0061] Specifically, the computer device can generate a first image prefix vector based on the first sample image modal data through a prefix vector module, that is, perform feature recognition on the first sample image modal data to obtain the first image prefix vector. The first image prefix vector generated by the prefix vector module can be used to adjust and improve the image feature recognition of the first sample image modal data, thereby improving the image feature recognition of the first sample image modal data. It can be understood that through the first image prefix vector, a more granular visual representation of the first sample image modal data can be obtained, that is, more image features can be extracted from the first sample image modal data, which facilitates subsequent modal retrieval.

[0062] Among them, the prefix vector can be used to enable the modal recognition module to have multimodal data recognition capabilities without changing the original parameters of the modal recognition module. That is, the image prefix vector can include the prefix vector of the image feature. Since the modal recognition module itself has the recognition capability of text modal data, the image prefix vector here is used to guide the modal recognition module to recognize the image features of the first sample image modality without changing the original parameters of the modal recognition module, so that the modal recognition module increases the recognition capability for image modal data, and ultimately enables the modal recognition module to have the recognition capability for image modal data and text modal data. The image features here can refer to color, texture, shape, pattern, etc.

[0063] Optionally, the computer device may generate the first image prefix vector based on the first sample image modal data through the prefix vector module. The specific method may include: performing object cropping on the first sample image modal data to obtain object image data, and performing feature extraction on the object image data through the feature extraction layer in the prefix vector module to obtain an object image feature vector. Performing linear transformation and dimensionality reduction processing on the object image feature vector through the projection layer in the prefix vector module to obtain a processed object image feature vector. Performing vector summation on the adjusted feature vector in the prefix vector layer and the processed object image feature vector through the prefix vector layer in the prefix vector module to obtain the first image prefix vector.

[0064] Specifically, the computer device may perform object cropping on the first sample image modal data to obtain object image data. Specifically, the computer device may perform object recognition on the first sample image modal data to determine the region where the object is located in the first sample image modal data, and crop the region where the object is located in the first sample image modal data to obtain object image data. For example, if the first sample image modal data includes a motorcycle, the computer device may perform object cropping on the region where the motorcycle is located in the first sample image modal data to obtain object image data corresponding to the motorcycle.

[0065] The prefix vector module includes a feature extraction layer, a projection layer, and a prefix vector layer. The computer device can convert the object image data to a target resolution, and further perform feature extraction on the object image data with the target resolution through the feature extraction layer in the prefix vector module to obtain an object image feature vector. Furthermore, the computer device can perform linear transformation and dimensionality reduction processing on the object image feature vector through the projection layer (i.e., Projection layer) in the prefix vector layer to obtain a processed object image feature vector.

[0066] Among them, the projection layer in the prefix vector module can perform matrix multiplication operations to multiply the object image feature vector with the weight matrix in the projection layer to obtain the processed object image feature vector. In this way, the object image feature vector can be reduced in dimensionality, that is, the feature dimension of the processed object image feature vector is smaller than the feature dimension of the object image feature vector. Among them, the weight matrix in the projection layer of the prefix vector module can be learned during the model training process, and the projection layer in the prefix vector module can be similar to a "bridge" to convert and transfer feature data between different layers, so that the neural network can adaptively adjust the data representation method, that is, provide adaptation for different model network layers, and provide more convenience for the processing of subsequent model network layers.

[0067] Specifically, the prefix vector layer in the prefix vector module includes an adjusted feature vector. The adjusted feature vector may be randomly initialized, and the feature dimension of the adjusted feature vector may be the same as the feature dimension of the processed object image feature vector. The computer device may perform vector summation on the adjusted feature vector in the prefix vector layer and the processed object image feature vector via the prefix vector layer in the prefix vector module to obtain a first image prefix vector. In other words, the computer device may add the adjusted feature vector to the processed object image feature vector to obtain the first image prefix vector.

[0068] The number of prefix vector layers in the prefix vector module may be one or more. When there are multiple prefix vector layers, the corresponding adjustment feature vectors in the multiple prefix vector layers may be different or the same.

[0069] S103 , generating a first search character according to the first image prefix vector, the first sample image modality data, and the first sample text modality data through a modality recognition module.

[0070] The first search character is used to reflect the first sample text modal data and the first sample image modal data, that is, the first search character is used to reflect the text content of the first sample text modal data and the first sample image modal data.

[0071] Specifically, the modal recognition module can be used to perform feature recognition on image modal data and text modal data respectively. The computer device can input the first image prefix vector, the first sample image modal data and the first sample text modal data into the modal recognition module, and generate the first search character through the modal recognition module. It can be understood that the first image prefix vector output by the prefix vector model can increase the visual representation of the first sample image modal data at more granularity, thereby improving the accuracy of image feature recognition of the first sample image modal data, and thereby improving the accuracy of generating the first search character. Optionally, the modal recognition module includes a visual representation submodule and a language recognition submodule, the visual submodule is used to perform feature recognition on the image modal data, and the language recognition submodule is used to perform feature recognition on the text modal data.

[0072] The specific method for a computer device to generate a first search character based on the first image prefix vector, the first sample image modal data, and the first sample text modal data through the modal recognition module may include: performing feature extraction on the first sample image modal data through the visual representation submodule to obtain an image modal feature vector, performing attention feature extraction on the first image prefix vector and the image modal feature vector to obtain a visual representation vector. Adding the first sample text modal data to the text slot in the data template, adding the visual representation vector to the image slot in the data template to obtain the added data template, and inputting the added data template into the language recognition submodule. Performing feature extraction on the first sample text modal data in the text slot through the language recognition submodule to obtain text modal features, and generating the first search character based on the text modal features and the visual representation vector in the image slot.

[0073] Specifically, the computer device can perform patch processing on the first sample image modality data through the visual representation submodule to obtain multiple patch image data, and further perform feature extraction on the multiple patch image data through the patch linear layer (i.e., Patch Linear Layer) in the visual representation submodule to obtain an image modality feature vector. The visual representation submodule may include an attention mechanism, and the visual representation submodule performs attention feature extraction on the first image prefix vector and the image modality feature vector to obtain a visual representation vector. It can be understood that the first image prefix vector can be used as a prefix of the image modality feature vector and input into the visual representation submodule together to adjust and improve the image feature recognition of the image modality feature vector, so as to obtain a multi-granularity visual representation vector. In order to make the visual representation vector output by the visual representation submodule layer compatible with the language recognition submodule layer, the visual representation vector output by the visual representation submodule can be input into the adaptive projection layer (i.e., Projection layer), and the adaptive projection layer and the projection layer in the above-mentioned prefix vector module have different model parameters.

[0074] The computer device can perform linear transformation and dimensionality reduction processing on the visual representation vector through the adaptive projection layer to obtain the processed visual representation vector. Through the adaptive projection layer, the visual representation vector can be mapped to the feature space of the language recognition submodule, so that the language recognition submodule can better process the visual representation vector. Among them, the weight matrix in the adaptive projection layer can be learned and adjusted during the model training process. Furthermore, the computer device can add the first sample text modality data to the text slot in the data template, and add the processed visual representation vector to the image slot in the data template to obtain the added data template. It can be understood that the embodiment of the present application constructs a data modality for instruction fine-tuning, and the data template includes a text slot and an image slot. By inserting the visual representation feature and the first sample text modality data into the predefined slots of the data template respectively, it can facilitate the language recognition submodule to distinguish the first visual representation and the first sample text modality data.

[0075] The computer device can perform feature extraction on the first sample text modality data in the text slot through the language recognition submodule to obtain text modality features, such as performing word vector conversion on the first sample text modality data, and using the converted word vector as the text modality feature. Furthermore, the computer device can generate a first search character based on the text modality features and the visual representation vector in the image slot through the language recognition submodule. Specifically, the computer device can splice the text modality features and the visual representation vector to obtain a spliced ​​feature vector, and generate the first search character based on the spliced ​​feature vector through the language recognition submodule. Among them, the language recognition submodule can be a large language model structure, which is a statistical model for predicting the probability of a series of words in a text sequence. It is trained on large-scale text data to understand the language and predict the next word in the sequence. For example, the language recognition submodule can be a large language model structure such as a GPT network structure (i.e., Generative Pre-Training, a natural language processing model based on deep learning), a BERT network structure (Bidirectional Encoder Representations from Transformer, an unsupervised pre-training language model for natural language processing tasks), etc.

[0076] Optionally, to better adapt the language recognition submodule to multimodal retrieval scenarios, an adaptive fine-tuning layer (i.e., the adaptive fine-tuning layer can include a LORA Adapter fine-tuning method) can be used to efficiently fine-tune the language recognition submodule, addressing the over-reliance on the generative model and overfitting issues that occur during the fine-tuning process when training the added initial retrieval model. Specifically, the adaptive fine-tuning layer introduces an additional adaptive fine-tuning layer (e.g., a linear layer) into the trained language recognition submodule and uses the training data of the multimodal retrieval task to fine-tune the model parameters in this adaptive fine-tuning layer. This method enables the added language recognition submodule to better adapt to specific tasks (i.e., multimodal retrieval tasks) while reducing over-reliance on the initial language recognition submodule (i.e., the language recognition submodule in its initial state in the trained initial retrieval model). The adaptive fine-tuning layer can be added to the self-attention layer (i.e., self-attention) in the language recognition submodule, which is used to perform self-attention feature extraction on the visual representation vector and text modality features.

[0077] Optionally, the visual representation submodule includes N conversion layers (i.e., N Tansfomer layers), the prefix vector module includes N prefix vector layers corresponding to the N conversion layers, one conversion layer corresponds to one prefix vector layer, and the first image prefix vector includes the image prefix vectors output by the N prefix vector layers, respectively, where N is a positive integer. It is understandable that the adjusted feature vectors corresponding to the N prefix vector layers may be different, and each prefix vector layer in the N prefix vector layers performs vector summation on the corresponding adjusted feature vector and the processed object image feature vector to obtain the image prefix vector output by each layer. The computer device performs attention feature extraction on the first image prefix vector and the image modality feature vector, and a specific method for obtaining the visual representation vector may include: obtaining the image conversion feature corresponding to the i-th conversion layer in the N conversion layers. When i=1, the image conversion feature corresponding to the first conversion layer is generated by the first conversion layer based on the image modality feature vector and the image prefix vector output by the first prefix vector layer; i is a positive integer less than or equal to N. The image conversion features corresponding to the i+1th conversion layer among the N conversion layers are generated based on the image conversion features corresponding to the i-th conversion layer and the image prefix vector output by the i+1th prefix vector layer. This process continues until the image conversion features corresponding to the N-th conversion layer among the N conversion layers are obtained, and a visual representation vector is generated based on the image conversion features corresponding to the N-th conversion layer.

[0078] Specifically, the computer device can obtain the image conversion features corresponding to the i-th conversion layer among N conversion layers. Taking i=1 as an example, the image conversion features corresponding to the first conversion layer are generated by the first conversion layer based on the image modality feature vector and the image prefix vector output by the first prefix vector layer. Wherein, i is a positive integer less than or equal to N, the first prefix vector layer belongs to the N prefix vector layers included in the prefix vector module, and the first conversion layer corresponds to the first prefix vector layer. Specifically, taking the computer device obtaining the image conversion features corresponding to the first conversion layer as an example, the computer device can multiply the image modality feature vector with the query weight vector in the first conversion layer through the first conversion layer to obtain a processed image modality feature vector. Furthermore, the computer device can perform self-attention feature extraction on the processed image modality feature vector and the image modality feature vector to obtain a third attention feature, and perform self-attention feature extraction on the processed image modality feature vector and the image prefix vector output by the first prefix vector layer to obtain a fourth attention feature.

[0079] Specifically, the computer device performs effective feature screening on the fourth attention feature through the gating layer corresponding to the first prefix vector layer to obtain the effective fourth attention feature, and then performs denoising on the fourth attention feature to avoid invalid attention features in the fourth attention feature from interfering with the extraction of the image conversion feature corresponding to the first conversion layer. Among them, each of the N prefix vector layers has a corresponding gating layer, and the gating layer includes a gate function. The model parameters in the gating layer (such as the function parameters of the gate function) can be learned and adjusted during the model training process. The computer device can perform feature summation on the effective fourth attention feature and the third attention feature to obtain the attention feature sum corresponding to the first conversion layer, and then perform a linear transformation on the attention feature sum corresponding to the first conversion layer through the feedforward neural network in the first conversion layer to obtain the image conversion feature corresponding to the first conversion layer.

[0080] Furthermore, the computer device can use the image conversion features corresponding to the i-th conversion layer and the image prefix vector output by the i+1-th prefix vector layer as inputs to the i+1-th conversion layer among the N conversion layers, and generate the image conversion features corresponding to the i+1-th conversion layer through the i+1-th conversion layer among the N conversion layers based on the image conversion features corresponding to the i-th conversion layer and the image prefix vector output by the i+1-th prefix vector layer. The i+1-th conversion layer is the next conversion layer after the i-th conversion layer. For example, when i=1, the i+1-th conversion layer is the second conversion layer among the N conversion layers. The i+1-th prefix vector layer belongs to the N prefix vector layers, and the i+1-th prefix vector layer corresponds to the i+1-th conversion layer. This process is repeated until the image conversion features corresponding to the N-th conversion layer among the N conversion layers are obtained, and the image conversion features corresponding to the N-th conversion layer are determined as the visual representation vector.

[0081] Optionally, the specific method for the computer device to generate the image conversion feature corresponding to the i+1th conversion layer may include: multiplying the image conversion feature corresponding to the i-th conversion layer by the query weight vector in the i+1th conversion layer through the i+1th conversion layer among the N conversion layers to obtain the processed image conversion feature. Performing self-attention feature extraction on the processed image conversion feature and the image prefix vector output by the i+1th prefix vector layer to obtain the first attention feature. Performing self-attention feature extraction on the processed image conversion feature and the image conversion feature corresponding to the i-th conversion layer to obtain the second attention feature. Performing effective feature screening on the first attention feature to obtain an effective first attention feature, summing the effective first attention feature and the second attention feature to obtain the total attention feature. Generate the image conversion feature corresponding to the i+1th conversion layer based on the total attention feature.

[0082] Specifically, the computer device can use the image conversion features corresponding to the i-th conversion layer and the query weight vector in the i+1-th conversion layer as the input of the i+1-th conversion layer among the N conversion layers, and multiply the image conversion features corresponding to the i-th conversion layer and the query weight vector in the i+1-th conversion layer through the i+1-th conversion layer among the N conversion layers to obtain the processed image conversion features. Each conversion layer in the N conversion layers has a corresponding query weight vector. The computer device can perform self-attention feature extraction on the processed image conversion features and the image prefix vector output by the i+1-th prefix vector layer to obtain a first attention feature. The computer device can perform self-attention feature extraction on the processed image conversion features and the image conversion features corresponding to the i-th conversion layer to obtain a second attention feature.

[0083] Specifically, the computer device can perform effective feature screening on the first attention feature through the gating layer corresponding to the i+1th prefix vector layer to obtain an effective first attention feature. In this way, the first attention feature can be denoised to avoid invalid attention features in the first attention feature from interfering with the extraction of the image conversion feature corresponding to the 1+1th conversion layer. Furthermore, the computer device can sum the effective first attention feature and the second attention feature to obtain a total attention feature. The total attention feature is linearly transformed through the feedforward neural network in the i+1th conversion layer to obtain the image conversion feature corresponding to the i+1th conversion layer.

[0084] Optionally, the computer device performs self-attention feature extraction on the processed image conversion features and the image prefix vector output by the i+1th prefix vector layer. The specific method for obtaining the first attention feature may include: multiplying the image prefix vector output by the i+1th prefix vector layer with the key weight vector corresponding to the i+1th prefix vector layer to obtain a prefix key vector. Multiplying the image prefix vector output by the i+1th prefix vector layer with the value weight vector corresponding to the i+1th prefix vector layer to obtain a prefix value vector. Multiplying the processed image conversion features with the transpose of the prefix key vector to obtain a first similarity vector, and multiplying the first similarity vector and the prefix value vector to obtain a first attention feature.

[0085] Specifically, the i+1th conversion layer may include a key weight vector corresponding to the i+1th prefix vector layer and a key weight vector corresponding to the i-th conversion layer. The computer device may multiply the image prefix vector output by the i+1th prefix vector layer with the key weight vector corresponding to the i+1th prefix vector layer in the i+1th conversion layer to obtain a prefix key vector. Similarly, the i+1th conversion layer may include a value weight vector corresponding to the i+1th prefix vector layer and a value weight vector corresponding to the i-th conversion layer. The computer device may multiply the image prefix vector output by the i+1th prefix vector layer with the value weight vector corresponding to the i+1th prefix vector layer in the i+1th conversion layer to obtain a prefix value vector. Furthermore, the computer device may multiply the processed image conversion features with the transpose of the prefix key vector to obtain an initial first similarity vector, obtain the ratio between the initial first similarity vector and the square root of the feature dimension of the image prefix vector, and then normalize the ratio using the SoftMax function to obtain a first similarity vector. Among them, the SoftMax function can output continuous numbers into a number between 0 and 1.

[0086] Furthermore, the computer device may multiply the first similarity vector and the prefix value vector to obtain a first attention feature. It is understandable that the computer device calculates the prefix key vector (i.e., the K vector in the self-attention mechanism) and the prefix value vector (i.e., the V vector in the self-attention mechanism) based on the prefix vector output by the i+1th prefix vector layer, and uses the processed image conversion feature as the query vector (i.e., the Q vector in the self-attention mechanism) to jointly calculate the self-attention feature.

[0087] Optionally, the computer device performs self-attention feature extraction on the processed image conversion features and the image conversion features corresponding to the i-th conversion layer. The specific method for obtaining the second attention feature may include: multiplying the image conversion features corresponding to the i-th conversion layer with the key weight vector corresponding to the i-th conversion layer to obtain an image key vector. Multiplying the image conversion features corresponding to the i-th conversion layer with the value weight vector corresponding to the i-th conversion layer to obtain an image value vector. Multiplying the processed image conversion features with the transpose of the image key vector to obtain a second similarity vector, and multiplying the second similarity vector and the image value vector to obtain a second attention feature.

[0088] Specifically, the i+1th conversion layer includes a key weight vector corresponding to the i-th conversion layer, and the computer device can multiply the image conversion feature corresponding to the i-th conversion layer with the key weight vector corresponding to the i-th conversion layer to obtain an image key vector. The key weight vector corresponding to the i-th conversion layer and the key weight vector corresponding to the i+1th prefix vector layer can be the same or different. Similarly, the i+1th conversion layer includes a value weight vector corresponding to the i-th conversion layer, and the computer device can multiply the image conversion feature corresponding to the i-th conversion layer with the value weight vector corresponding to the i-th conversion layer to obtain an image value vector. Similarly, the value weight vector corresponding to the i-th conversion layer and the value weight vector corresponding to the i+1th prefix vector layer can be the same or different. Furthermore, the computer device may multiply the processed image conversion feature with the transpose of the image key vector to obtain an initial second similarity vector, obtain the ratio between the initial second similarity vector and the square root of the feature dimension of the processed image conversion feature, and then normalize the ratio using a SoftMax function to obtain a second similarity vector. The computer device may multiply the second similarity vector and the image value vector to obtain a second attention feature.

[0089] As shown in Figure 4, Figure 4 is a schematic diagram of obtaining a first search character provided by an embodiment of the present application. As shown in Figure 4, taking the first sample text modal data 401a included in the first sample retrieval request 40a as "What car does this picture show", and the first sample image modal data 402a included in the first sample retrieval request as an image including a car as an example, the computer device can perform object cropping on the first sample image modal data to obtain object image data 40b. The computer device can perform feature extraction on the object image data 40b to obtain an object image feature vector, and perform linear transformation and dimensionality reduction processing on the object image feature vector through the first projection layer 40d (i.e., the projection layer in the prefix vector module) to obtain a processed object image feature vector. Furthermore, the computer device can perform vector summation on the adjusted feature vector in the prefix vector layer and the processed object image feature vector through the prefix vector layer 40f in the prefix vector module to obtain a first image prefix vector.

[0090] As shown in FIG. 4 , the number of prefix vector layers 40 f may be N. The adjusted feature vectors in each prefix vector layer may be randomly initialized, and the adjusted feature vectors in each prefix vector layer may be different.

[0091] As shown in Figure 4, the computer device can perform patch processing on the first sample image modality data 402a to obtain multiple patch image data 40c, and further perform feature extraction on the multiple patch image data 40c through the patch linear layer 40e (i.e., Patch Linear Layer) to obtain an image modality feature vector. The computer device can perform attention feature extraction on the first image prefix vector and the image modality feature vector through the conversion layer 40g in the visual representation submodule to obtain a visual representation vector. The number of conversion layers 40g in the visual representation submodule can be N layers, one prefix vector layer corresponds to one conversion layer, and the image prefix vector output by each prefix vector layer can be used as the input of the corresponding conversion layer. It can be understood that the first image prefix vector can be used as the prefix of the image modality feature vector and input into the conversion layer of the visual representation submodule to adjust and improve the image feature recognition of the image modality feature vector, so as to obtain a multi-granularity visual representation vector.

[0092] Taking the first conversion layer as an example, the computer device can control the image prefix vector output by the first prefix vector layer to participate in the self-attention calculation of the first conversion layer through the gate function, which can reduce the interference of invalid image prefix vectors and improve the accuracy and stability of visual representation vector extraction.

[0093] In order to make the visual representation vector output by the visual representation submodule layer compatible with the language recognition submodule layer, the visual representation vector output by the visual representation submodule can be input into the second projection layer 40h (i.e., the adaptation projection layer), and the model parameters of the second projection layer 40h and the first projection layer 40d are different. The computer device performs linear transformation and dimensionality reduction processing on the visual representation vector through the second projection layer 40h to obtain a processed visual representation vector. It can be understood that through the second projection layer 40h, the visual representation vector can be mapped to the feature space of the language recognition submodule 40j, so that the language recognition submodule 40j can better process the visual representation vector. Further, the computer device can add the first sample text modal data 401a to the text slot in the data template 40i, add the processed visual representation vector to the image slot in the data template 40i, obtain the added data template, and input the added data template into the language recognition submodule 40j.

[0094] The computer device can use language identification submodule 40j to perform feature extraction on the first sample text modality data in the text slot to obtain text modality features. For example, the computer device can perform word vector conversion on the first sample text modality data and use the converted word vector as the text modality feature. Furthermore, the computer device can use language identification submodule 40j to generate a first search character 40k based on the text modality feature and the visual representation vector in the image slot.

[0095] S104: Obtain a first sample search result associated with the first search character from a pre-generated database through a constraint decoding module.

[0096] Among them, the first sample retrieval result may refer to the retrieval result obtained through the initial modal model based on the first sample image modal data and the first sample text modal data. The first sample retrieval result is the retrieval answer to the text question indicated by the first sample text modal data (i.e., the answer obtained by the search).

[0097] The association of the first search character with the first sample search result may mean that the first search character is included in the first sample search result, or the association of the first search character with the first sample search result may mean that the similarity between the first sample search result and the first search character is greater than a first similarity threshold, or the association of the first search character with the first sample search result may mean that the similarity between the result identifier corresponding to the first sample search result and the first search character is greater than a second similarity threshold. The result identifier may refer to a number, keyword, or data identifier corresponding to the first sample search result. For example, when the first sample search result is a document data, the data identifier may refer to a document identifier, i.e., a document name.

[0098] Specifically, the computer device can use the constraint decoding module to dynamically determine any distinguishable fixed-length character in the document data based on the first search character, use the determined character as a knowledge clue, and then retrieve the first sample retrieval result associated with the first sample retrieval request based on the knowledge clue. Compared to static identifiers such as titles and URLs, knowledge clues offer greater flexibility and generalizability in large-scale knowledge retrieval scenarios, improving the retrieval efficiency and accuracy of the first sample retrieval result.

[0099] Optionally, the computer device may retrieve the first sample retrieval result associated with the first sample retrieval request based on the first retrieval character through the constraint decoding module. The specific method may include: querying a set of candidate characters that match the first retrieval character from a pre-generated database based on the first retrieval character through the character acquisition interface in the constraint decoding module; the pre-generated database includes characters corresponding to P pieces of document data, where P is a positive integer. The matching probability between the candidate characters in the candidate character set and the first retrieval character is obtained. The candidate character with the maximum matching probability is determined from the candidate character set as the suffix character of the first retrieval character. The first retrieval character and the suffix character are combined to obtain a combined character sequence, and the first sample retrieval result associated with the first sample retrieval request is retrieved based on the combined character sequence.

[0100] Specifically, the constrained decoding module can include a Beam Search decoding framework, which can introduce constraints during the decoding of the first search character to generate a dynamic "knowledge clue" while ensuring that the dynamic "knowledge clue" only appears in a document in a pre-generated database. The Beam Search decoding framework is an algorithm for finding the optimal solution in a search control. It searches for the most likely solution by selecting a set of candidate solutions with the highest probability at each time step.

[0101] The characters corresponding to the document data may refer to characters used to distinguish the document data in the document data, and the characters corresponding to different document data are unique. The pre-generated database may refer to a database used for retrieval, and the database may include various data, such as document data, session data, etc.

[0102] The constraint decoding module includes a character acquisition interface (i.e., the next character acquisition interface, GetNext interface), a validity verification interface (i.e., the ValidDistinct interface), and a document retrieval interface (i.e., the LookupDoc interface). The computer device can use the character acquisition interface in the constraint decoding module to query a pre-generated database for a set of candidate characters that match a first search character based on the first search character.

[0103] Among them, in order to ensure the efficiency and accuracy of the first sample retrieval result search, the pre-generated database can be stored in the FM-Index index database. The FM-Index index database is a self-index structure used to efficiently process text data. The FM-Index index database uses a secondary storage method for efficient sorting and access. By sampling the suffix array (SA) in part of the interval, the occurrence position of the pattern string in the text can be efficiently located. The FM-Index index database can restore the original text of any range and is usually used to process large-scale text data sets or scenarios that require efficient text processing. It combines multiple technologies and algorithms to achieve fast and efficient text retrieval and access. At the same time, it encapsulates three interfaces (i.e., character acquisition interface, validity verification interface, and document retrieval interface). Each interface and its function can be shown in Table 1.

[0104] Table 1

[0105] As shown in Table 1, the GetNext interface can be used to obtain the next feasible word set (i.e., candidate characters that match the first search character), the ValidDistinct interface can be used to verify whether the current generation result (i.e., the combined character sequence obtained by combining the first search character and the suffix character) is uniquely present in a document data in the pre-generated database, and the LookupDoc interface is used to find the corresponding search document data based on the "knowledge clue" (i.e., a valid combined character sequence).

[0106] Among them, the pre-generated database includes characters corresponding to P document data respectively, and P is a positive integer. The computer device can use the first search character as a prefix condition, and search all characters that match the prefix condition (i.e., the first search character) from the pre-generated database to obtain a candidate character set. The computer device can obtain the matching probability between the candidate characters in the candidate character set and the first search character. For example, taking the target candidate character in the candidate character set as an example, the constraint decoding module can obtain the probability that the target candidate character is the next character of the first search character, and obtain the matching probability between the target candidate character and the first search character. The computer device can determine the candidate character with the maximum matching probability from the candidate character set as the suffix character of the first search character, and combine the first search character and the suffix character to obtain a combined character sequence. Furthermore, the computer device can retrieve the first sample retrieval result associated with the first sample retrieval request based on the combined character sequence.

[0107] Optionally, the computer device may retrieve a first sample retrieval result associated with the first sample retrieval request based on the combined character sequence by: verifying the validity of the combined character sequence through a validity verification interface in the constraint decoding module to obtain a verification result. If the verification result indicates that the combined character sequence is valid, retrieving document data having the combined character sequence from a pre-generated database through a document retrieval interface in the constraint decoding module. Determining the retrieved document data as the first sample retrieval result associated with the first sample retrieval request.

[0108] Specifically, the computer device can verify the validity of the combined character sequence through the validity verification interface in the constraint decoding module to obtain a verification result. It is understandable that the computer device can verify whether the combined character sequence is uniquely present in a document data in the pre-generated database, or whether the sequence length of the combined character sequence is greater than or exceeds a sequence length threshold, to obtain the verification result. Only when the verification result indicates that the combined character sequence is valid, the document data having the combined character sequence is retrieved from the pre-generated database through the document retrieval interface in the constraint decoding module, and the retrieved document data is determined as the first sample retrieval result associated with the first sample retrieval request.

[0109] Of course, if the verification result indicates that the combined character sequence is not valid, the character acquisition interface in the constraint decoding module is continued, using the combined character sequence as a prefix condition, and searching the pre-generated database for all characters that match the combined character sequence to obtain a matching character set for the combined character sequence. Similarly, the computer device can obtain the matching probability between the characters in the matching character set for the combined character sequence and the combined character sequence. Further, based on the matching probability, the next combined character sequence is generated, and the validity value of the next combined character sequence is verified through the validity verification interface until a valid combined character sequence is obtained.

[0110] It can be seen that the first search character output by the language recognition submodule can be the first predicted character, such as the first word output by the language recognition submodule, and the constraint decoding module can query the candidate characters that match the first search character from the pre-generated database based on the first search character. It can be seen that the characters generated after the first search character are all determined from the characters included in the pre-generated database, which can ensure that the valid combined character sequence (i.e., knowledge clue) generated finally appears and only appears in one document data in the pre-generated database, and can avoid the situation where the character finally generated by the language recognition submodule does not exist in the pre-generated database, resulting in retrieval failure, and can ensure the accuracy and efficiency of document data retrieval. At the same time, the constraint decoding module can dynamically generate a combined character sequence, and the valid combined character sequence finally generated can be any distinguishable fixed-length character in one document data in the pre-generated database, which has stronger flexibility and generalization compared to static identifiers such as titles and URLs.

[0111] It is understandable that in the process of obtaining a valid combined character sequence (i.e., a knowledge clue), each decoding needs to call the GetNext interface to determine the next feasible character sequence (i.e., the suffix character of the first search character) from the pre-generated database based on the first search character. Generate a combined character sequence based on the next feasible character sequence and the first search character, and then call the ValidDistinct interface to verify the uniqueness of the generated combined character sequence. If the return value is True, it indicates that the current combined character sequence is a qualified knowledge clue, and the generation stops. Otherwise, the generation continues, and each subsequent step of generating a combined character sequence is verified once through the ValidDistinct interface until a valid combined character sequence (i.e., a qualified knowledge clue) is obtained or the sequence length of the generated combined character sequence reaches the stopping condition (such as a sequence length threshold). We use the generated knowledge clues as the search conditions of the LookupDoc interface, and the corresponding document data can be obtained from the pre-generated database.

[0112] Optionally, the computer device verifies the validity of the combined character sequence through a validity verification interface in the constraint decoding module. A specific method for obtaining the verification result may include: the computer device may obtain a sequence length of the combined character sequence through the validity verification interface in the constraint decoding module, and detect whether the sequence length is greater than or equal to a sequence length threshold. If the sequence length is greater than or equal to the sequence length threshold, determining that the combined character sequence is valid and generating a verification result indicating that the combined character sequence is valid.

[0113] Optionally, if the sequence length is less than the sequence length threshold, the computer device obtains document data having a combined character sequence from a pre-generated database to obtain matching document data. The number of matching document data is obtained. If the number of matching document data is equal to the number threshold, the combined character sequence is determined to be valid, and a verification result is generated to indicate that the combined character sequence is valid. The number threshold can be 1, that is, the matching document data is unique in the pre-generated database. In this way, the retrieval accuracy of the first sample retrieval result can be ensured. If the number of matching document data is not equal to the number threshold, it is determined that the combined character sequence is not valid, and the character acquisition interface is continued to be called to generate the next combined character sequence until a valid combined character sequence is obtained. It can be seen that through the knowledge-guided constraint decoding module, the correct "knowledge clue" (that is, a valid combined character sequence) can be dynamically generated as a document identifier, and the first sample retrieval result can be retrieved from the pre-generated database, which can improve the retrieval accuracy and efficiency of the first sample retrieval result.

[0114] As shown in FIG5 , FIG5 is a schematic diagram of a method for retrieving a first sample search result based on a first search character according to an embodiment of the present application. As shown in FIG5 , the language identification submodule 50a can output multiple candidate characters (such as candidate character 50c, candidate character 50d, and first search character 50b). The first search character 50b can be the candidate character with the highest output probability among the multiple candidate characters output by the language identification submodule 50a. Furthermore, the computer device can call the character acquisition interface in the constraint decoding module to obtain all next feasible words of the first search character 50b (i.e., the set of candidate characters that match the first search character) from the pre-generated database 50i. The computer device can determine the candidate character with the highest matching probability with the first search character from all next feasible words, and use it as the suffix character of the first search character. The first search character 50b and the suffix character of the first search character are combined to obtain a combined character sequence 50e. The first search character is combined with other candidate characters to obtain its characteristic candidate combined character sequence, such as candidate combined character sequence 50f and candidate combined character sequence 50g.

[0115] Furthermore, the computer device can call the validity verification interface in the constraint decoding module to verify the validity of the combined character sequence 50e (i.e., verify whether the combined character sequence 50e is uniquely present in a document data in the pre-generated database 50i, or whether the sequence length of the combined character sequence 50e is greater than or equal to the sequence length threshold). If the validity verification interface returns a verification result indicating that the combined character sequence 50e is not valid, the computer device can use the character acquisition interface in the constraint decoding module to use the combined character sequence 50e as a prefix condition to continue to acquire all the next feasible words of the combined character sequence 50e, thereby acquiring the next combined character sequence 50h. Similarly, the computer device can use the validity verification interface in the constraint decoding module to verify the validity of the next combined character sequence 50h. If the validity verification interface returns a verification result indicating that the next combined character sequence 50h is valid, the computer device can call the document retrieval interface in the constraint decoding module to acquire the first sample retrieval result 50j from the pre-generated database 50i based on the next combined character sequence 50h.

[0116] S105 , adjusting the model parameters corresponding to the prefix vector module and the constraint decoding module respectively according to the first reference search result and the first sample search result, to obtain an adjusted retrieval model.

[0117] Specifically, because the generative language model is already trained—that is, it's pre-trained to convergence—and includes a modal recognition module, the model parameters in the modal recognition module are also already trained. Therefore, the computer device can freeze the model parameters corresponding to the modal recognition module, reducing the adjustment range of the model parameters. Even with limited annotated data, the inherent knowledge in the trained generative language model can be effectively utilized, resulting in a higher-performance multimodal retrieval model and improving the training efficiency of the multimodal retrieval model.

[0118] Since the prefix vector module and the constraint decoding module are newly added to the trained generative language model, the computer device may need to adjust the parameters of the prefix vector module and the constraint decoding module. Specifically, the computer device can determine the model loss value of the initial retrieval model based on the first reference retrieval result and the first sample retrieval result. Further, according to the model loss value, the model parameters corresponding to the prefix vector module and the constraint decoding module are adjusted to obtain a multimodal retrieval model. It can be seen that the embodiment of the present application can reduce the range of model parameter adjustment by freezing the model parameters of the modal recognition module (that is, not adjusting the model parameters of the modal recognition module) and only adjusting the model parameters corresponding to the prefix vector module and the constraint decoding module, thereby achieving lightweight model parameter fine-tuning and avoiding the problem of catastrophic forgetting of the model parameters corresponding to the modal recognition module when adjusting the model parameters corresponding to the modal recognition module. In this way, the stability of the model training can be ensured, and even in the case of limited labeled data, the intrinsic knowledge in the trained initial retrieval model can be effectively utilized to obtain a multimodal retrieval model with higher performance, which can reduce the burden of model training and improve the training efficiency of the multimodal retrieval model.

[0119] It is understandable that the modal recognition module includes a visual representation submodule. In order to efficiently fine-tune the visual representation submodule, an embodiment of the present application proposes a prefix fine-tuning method for visual object perception. The prefix fine-tuning method fixes the original parameters of the visual representation submodule, only keeps the parameters of the prefix vector module learnable, and integrates the object image features of the visual object in the first sample image modal data into the learnable prefix vector (i.e., the adjustment feature vector) in the prefix vector model, and adjusts and improves the multi-granularity visual feature learning through the prefix vector module. In this way, the efficiency and stability of the model training can be guaranteed, and the problem of catastrophic forgetting of the visual representation submodule due to the scale of the training data can be avoided.

[0120] Optionally, the computer device adjusts the model parameters corresponding to the prefix vector module and the constraint decoding module, respectively, to obtain a multimodal retrieval model. A specific method may include: determining a model loss value of the initial retrieval model based on the first reference retrieval result and the first sample retrieval result. The computer device may compare the model loss value with a loss threshold. If the model loss value is greater than the loss threshold, the computer device may derive the model loss function of the initial retrieval model to obtain a derived model loss function. Based on the derived model loss function and a gradient descent method, the parameter adjustment gradients corresponding to the prefix vector module and the constraint decoding module are determined. Furthermore, based on the parameter adjustment gradient corresponding to the prefix vector module, the model parameters corresponding to the prefix vector module are adjusted. Based on the parameter adjustment gradient corresponding to the constraint decoding module, the model parameters of the constraint decoding module are adjusted, thereby obtaining an initial retrieval model after parameter adjustment. When the initial retrieval model after parameter adjustment meets the convergence condition, the training of the initial retrieval model after parameter adjustment continues. When the initial retrieval model after parameter adjustment meets the convergence condition, the initial retrieval model after parameter adjustment is determined as the multimodal retrieval model. The convergence condition may be that the model loss value is less than or equal to the loss threshold, or the number of model training times is greater than or equal to the target number.

[0121] The optimized training of the initial retrieval model after adding it can use the teacher-forcing strategy and the negative log-likelihood loss function. The teacher-forcing strategy predicts the rth character based on the first r-1 predicted characters. The negative log-likelihood loss function is a commonly used loss function in machine learning, especially in classification tasks. It is used to measure the difference between the model's predictions and the actual results, and helps optimize the performance of the classification model.

[0122] Among them, the model loss function of the initial retrieval model can be shown as formula (1).

[0123] Wherein, r in formula (1) refers to the position of the predicted character. When r=1, it means the first character position. l refers to the total character length. y r Refers to the character at position r, y <r refers to the character at the first r-1 positions, T refers to the image modality feature of the image modality data, V is the text modality feature of the text modality data, and Θ refers to the current model parameters in the initial retrieval model. r |y <r ; T; V; Θ) refers to the predicted y r In y <r , T, V and the output probability under Θ (ie yr with y <r The matching probability between them is used to predict the output probability of the character at each character position. Refers to the model loss function of the initial retrieval model.

[0124] Optionally, the trained multimodal retrieval model can be applied to data retrieval of a single modality (such as any modality such as text modality, image modality, etc.), and can also be applied to data retrieval of multiple modalities (such as a combination of text modality and image modality). Specifically, the computer device can obtain a data retrieval request input by a business object, and the data retrieval request includes retrieval text modality data and retrieval image modality data with an associated relationship. The computer device can generate a retrieval image prefix vector based on the retrieval image modality data through the prefix vector module in the multimodal retrieval model. Through the modality recognition module in the multimodal retrieval model, the target retrieval character associated with the data retrieval request is identified based on the retrieval image prefix vector, the retrieval text modality data and the retrieval image modality data. Through the constraint decoding module in the multimodal retrieval model, the target retrieval document data associated with the data retrieval request is obtained based on the target retrieval character, and the target retrieval document data is output to the business object. In this way, through the multimodal retrieval model, the target retrieval document data can be accurately and quickly retrieved, the retrieval efficiency and accuracy of the target retrieval document data can be improved, and the user experience can be improved.

[0125] The multimodal retrieval model in the embodiments of the present application can also be applied to scenarios such as video modal data and audio modal data. Specifically, the computer device can perform image conversion on the video modal data, invoke the multimodal retrieval model, and perform data retrieval based on the converted image modal data. Specifically, the computer device can perform text conversion on the audio modal data, invoke the multimodal retrieval model, and perform data retrieval based on the converted text modal data.

[0126] It is understandable that the embodiment of the present application provides a generative retrieval method for multimodal retrieval, which uses the initial retrieval model as the model base, the data retrieval request input by the business object (such as the query content to be queried) as input, and directly outputs the identifier of the best matching document data in the pre-generated database (i.e., knowledge clue) through the constraint decoding model, and then retrieves the corresponding retrieval document data, providing accurate knowledge input for downstream tasks (such as knowledge-based question-answering tasks). The embodiment of the present application regards the initial retrieval model as a virtual knowledge base, keeps the model parameters of the trained initial retrieval model unchanged, and performs lightweight fine-tuning on the model parameters of the prefix vector module and the constraint decoding model of the initial retrieval model. In this way, even in the case of limited training sample data, the intrinsic knowledge of the initial retrieval model can be effectively utilized (i.e., the knowledge prior of the initial retrieval model is fully utilized), so that the prefix vector module and the constraint decoding model learn knowledge about data retrieval and obtain a multimodal retrieval model, that is, only a small amount of training sample data is needed to obtain a multimodal retrieval model with superior retrieval performance. It can be understood that the multimodal retrieval model in the embodiment of the present application provides a generative multimodal retrieval framework that can achieve accurate knowledge retrieval performance.

[0127] At the same time, the embodiment of the present application dynamically generates "knowledge clues" (i.e., valid combined character sequences) through a knowledge-guided constraint decoding algorithm (i.e., a constraint decoding module). The "knowledge clue" can be any distinguishable fixed-length string in any document data in the pre-generated database. Using the "knowledge clue" as a document identifier, compared with static identifiers such as titles and URLs, knowledge clues have greater flexibility and generalization in large-scale data retrieval scenarios. The embodiment of the present application can be applied to multimodal data retrieval scenarios. When a business object communicates with a dialogue system (such as a dialogue robot), the multimodal retrieval model in the embodiment of the present application can be used to give the dialogue system the ability to acquire multimodal knowledge, so that it can perform data retrieval based on the multimodal context of the current dialogue context (i.e., the data retrieval request input by the business object) to obtain relevant common sense or factual knowledge. The multimodal retrieval model generates smarter and more accurate dialogue responses.

[0128] It is understandable that the embodiment of the present application will include a language recognition submodule (such as generating a large language model) and a visual representation submodule as a base, on which a prefix vector module and a constraint decoding model are introduced. Furthermore, the prefix fine-tuning method of visual object perception is introduced through the prefix vector module, and the visual representation submodule is efficiently fine-tuned to obtain a more granular visual representation. At the same time, the correct "knowledge clue" is generated as a document identifier through the constraint decoding algorithm.

[0129] As shown in FIG6 , FIG6 is a schematic diagram of a multimodal retrieval model provided by an embodiment of the present application for generative retrieval. As shown in FIG6 , when a data retrieval request input into the multimodal retrieval model includes retrieval text modal data (i.e., what is this in the image) and retrieval image data (i.e., including an image of a flying device) with an associated relationship, the computer device can input the retrieval text modal data (i.e., what is this in the image) and the retrieval image data (i.e., including an image of a drone) into the multimodal retrieval model. The knowledge retrieval generated by the multimodal retrieval model is “a drone is an unmanned aircraft that is controlled by a radio remote control device and a self-contained program control device.” Then, based on the generated knowledge retrieval, “a drone is an unmanned aircraft that is controlled by a radio remote control device and a self-contained program control device,” document data with a text identifier of 19254 is retrieved from the pre-generated database, i.e., “a drone is an unmanned aircraft that is controlled by a radio remote control device and a self-contained program control device, or is operated completely or intermittently autonomously by an onboard computer….”

[0130] As shown in FIG6 , when the data retrieval request input into the multimodal retrieval model includes retrieval text modal data (i.e., what is the plant in the image) and retrieval image data (i.e., an image of a plant) with an associated relationship, the computer device can input the retrieval text modal data (i.e., what is the plant in the image) and the retrieval image data (i.e., an image of a plant) into the multimodal retrieval model. The knowledge retrieval generated by the multimodal retrieval model is “the coconut tree is a large plant of the genus Cocos in the family Palmae.” Then, based on the generated knowledge retrieval, “the coconut tree is a large plant of the genus Cocos in the family Palmae.” The document data with the text identifier 89997 is retrieved from the pre-generated database, namely, “the coconut tree (English: Cocos nucifera L.) is a large plant of the genus Cocos in the family Palmae. Coconut is the fruit of the coconut tree and is a fruit that is very common in tropical regions. The popularity of the coconut tree is also related to the fact that its fruit, the coconut, can drift thousands of kilometers in the sea with the wind and waves and then reproduce in a place very far away from the mother tree…”

[0131] As shown in FIG6 , when a data retrieval request input into a multimodal retrieval model includes associated text modal data (i.e., what is the toy of the little boy in the picture) and image data (i.e., an image of the little boy playing with the toy), the computer device can input the text modal data (i.e., what is the toy of the little boy in the picture) and the image data (i.e., an image of the little boy playing with the toy) into the multimodal retrieval model. The knowledge retrieval generated by the multimodal retrieval model is "Yo-yo, also known as yo-yo, is two spheres connected by an axis." Furthermore, based on the generated knowledge retrieval, "Yo-yo, also known as yo-yo, is two spheres connected by an axis," the document data with the text identifier 11053 is retrieved from the pre-generated database, namely, "Yo-yo, also known as yo-yo, is two spheres connected by an axis, and then a thin string is tied to the axis, and the other end of the string is tied to the finger with a loop for playing. Yo-yo is one of the most versatile and spectacular hand-skill sports in the world..."

[0132] In addition, the hardware environment support during the use of the multimodal retrieval model for multimodal queries in the embodiment of the present application is relatively simple, and it can be trained and deployed online by providing a common server environment. For details, please refer to Table 2.

[0133] Table 2

[0134] As shown in Table 2, the operating system of the multimodal retrieval model can be Linux, a free and open source UNIX-like operating system. The multimodal retrieval model requires no more than 16GB of memory, and the language environment of the multimodal retrieval model is Python (a programming language) or C++ (a programming language).

[0135] The generative retrieval method proposed in the embodiment of the present application can effectively realize multimodal data retrieval. Compared with traditional methods, it can avoid redundant retrieval pipelines and heavy training burdens, while achieving superior retrieval performance. At the same time, due to the powerful content understanding and generation capabilities of the large language model (i.e., the language recognition submodule), it has great potential for understanding complex retrieval requests (such as user queries). The generative retrieval method proposed in the implementation of this application has been verified to be effective on multiple multimodal knowledge retrieval public data sets, such as OKVQA-GS112K, OKVQA-WK21M and ReMuQ data sets. Among them, the multimodal retrieval model in the embodiment of the present application has higher performance than the integrated retrieval method VRR, the text sparse retriever BM25, the text dual tower retriever DPR, the cross-modal retriever CLIP, the retrieval method ReViz based on the single-stream multimodal encoder, ReViz-ICT and the text generative retriever CorpusBrain, etc., as shown in Table 3.

[0136] Table 3

[0137] As shown in Table 3, the multimodal retrieval model in the embodiment of the present application has relatively superior retrieval performance. P@5 in Table 3 refers to the accuracy of the first five retrieved document data output by the model, P@10 refers to the accuracy of the first ten retrieved document data output by the model, and P@1 refers to the accuracy of the first retrieved document data output by the model. R@5 in Table 3 refers to the coverage of the first five retrieved document data output by the model.

[0138] In an embodiment of the present application, the initial retrieval model has a strong content comprehension ability and language generation ability, and has a large processing potential for data retrieval. Therefore, in the training process of the initial retrieval model, the model parameters of the original module (modal recognition module) in the initial retrieval model remain unchanged, and only the model parameters of the prefix vector module and the constraint decoding model need to be lightly fine-tuned so that the prefix vector module and the constraint decoding model can learn knowledge about data retrieval and obtain a multimodal retrieval model. This can reduce the adjustment of model parameters, reduce the training cost of the multimodal retrieval model, and improve the training efficiency of the multimodal retrieval model. By keeping the model parameters of the original module (i.e., modal recognition) in the initial retrieval model unchanged, the content comprehension ability and generation ability of the initial retrieval model can be reused, and the problem of catastrophic forgetting of the generative language module due to the update of training data can be avoided, and the training stability of the multimodal retrieval model can be ensured. During the training of the initial retrieval model, only lightweight fine-tuning of the constraint decoding module and the prefix vector module is required. Therefore, a large amount of sample data is not required, which can avoid the problem of low model training accuracy caused by insufficient sample data. This is conducive to improving the accuracy of model training and, in turn, improving the retrieval accuracy of the multimodal model during the retrieval process.

[0139] Further, please refer to Figure 7, which is a flow chart of a method for generating a multimodal retrieval model provided by an embodiment of the present application. As shown in Figure 7, the method can be executed by any terminal device in Figure 1, or by the server 10 in Figure 1, or by both the terminal device and the server in Figure 1. In this application, the devices used to execute the method for generating the multimodal retrieval model can be collectively referred to as computer devices. Among them, the method for generating the multimodal retrieval model can include but is not limited to the following steps:

[0140] S201: Obtain an initial retrieval model and a first sample retrieval request.

[0141] S202 : Perform feature recognition on the first sample image modality data through the prefix vector module to obtain a first image prefix vector.

[0142] S203 : Generate a first search character according to the first image prefix vector, the first sample image modality data, and the first sample text modality data through a modality recognition module.

[0143] S204: Obtain a first sample search result associated with the first search character from a pre-generated database through a constraint decoding module.

[0144] Specifically, the contents of step S201 to step S204 in the embodiment of the present application can refer to the contents of the above-mentioned step S101 to step S104, and the embodiment of the present application will not be repeated here.

[0145] S205 , obtaining a second sample retrieval request, and performing feature recognition on the first sample image modality data through the prefix vector module to obtain a second image prefix vector.

[0146] The second sample retrieval request includes second sample image modality data and a second reference retrieval result pre-annotated for the second sample retrieval request;

[0147] Specifically, if the second sample retrieval request includes second sample image modal data, the computer device can generate a second image prefix vector based on the second sample image modal data through the prefix vector module. Specifically, the computer device can perform object cropping on the second sample image modal data to obtain object image data corresponding to the second sample image modal data, and perform feature extraction on the object image data corresponding to the second sample image modal data through the feature extraction layer in the prefix vector module to obtain an object image feature vector corresponding to the second sample image modal data. The computer device can perform linear transformation and dimensionality reduction processing on the object image feature vector corresponding to the second sample image modal data through the projection layer in the prefix vector module to obtain a transformed object image feature vector. The computer device can perform vector summation on the adjusted feature vector in the prefix vector layer and the transformed object image feature vector through the prefix vector layer in the prefix vector module to obtain the second image prefix vector.

[0148] Specifically, the computer device may train the added initial retrieval model using a first sample retrieval request (i.e., multimodal sample data) including first sample image modal data and first sample text modal data having an associated relationship, or may train the added initial retrieval model using a second sample retrieval request (i.e., single modal sample data) including second sample image modal data. It is understandable that the computer device may train the added initial retrieval model using multimodal sample data and single modal samples, so that the trained multimodal retrieval model can perform single modal data retrieval and multimodal data retrieval.

[0149] S206 , generating a second search character according to the second image prefix vector and the second sample image modality data through the modality recognition module.

[0150] Specifically, the computer device can generate a second search character for the second image prefix vector and the second sample image modal data through the modal recognition module and the language recognition submodule. The modal recognition module includes a visual representation submodule, and the computer device can call the visual representation submodule to perform feature extraction on the second sample image modal data to obtain image features corresponding to the second sample image modal data. Furthermore, the computer device can call the visual representation submodule to perform attention feature extraction on the image features corresponding to the second sample image modal data and the second image prefix vector to obtain the visual representation corresponding to the second sample image modal data. Among them, the generation process of the visual representation corresponding to the second sample image modal data can refer to the content of the above-mentioned step S103, and the embodiments of the present application will not be repeated here. Furthermore, the computer device can generate a second search character according to the visual representation corresponding to the second sample image modal data through the language recognition submodule.

[0151] S207 , obtaining a second sample search result associated with the second search character from the pre-generated database through a constraint decoding module.

[0152] Specifically, the computer device may use the constraint decoding module to retrieve the second sample search result associated with the second sample search request according to the second search character. For details, please refer to the content of the above step S104, which will not be repeated in this embodiment of the present application.

[0153] S208 , adjusting model parameters corresponding to the prefix vector module and the constraint decoding module respectively according to the first reference retrieval result and the first sample retrieval result, and the second reference retrieval result and the second sample retrieval result, to obtain a multimodal retrieval model.

[0154] Specifically, since the generative language model has been trained, the modal identification module included in the trained generative language model may also have been trained, so the computer device can freeze the model parameters corresponding to the modal identification module. In this way, the adjustment range of the modal parameters can be reduced, thereby reducing the burden of model training and improving the efficiency of model training. Since the prefix vector module and the constraint decoding module are newly added to the trained generative language model, the computer device can determine the first retrieval loss of the initial retrieval model for the first sample retrieval request based on the first reference retrieval result and the first sample retrieval result. The computer device can determine the second retrieval loss of the initial retrieval model for the second sample retrieval request based on the second reference retrieval result and the second sample retrieval result. Furthermore, the computer device can adjust the model parameters corresponding to the prefix vector module and the constraint decoding module respectively according to the first retrieval loss and the second retrieval loss to obtain a multimodal retrieval model.

[0155] Optionally, the computer device may also use a third sample retrieval request including the second sample text modal data to train the added initial retrieval model. Specifically, if the third sample retrieval request includes the second sample text modal data, the computer device generates a third retrieval character according to the second sample text modal data through the language recognition submodule in the modal recognition module. The third sample retrieval result associated with the third retrieval character is obtained from the pre-generated database through the constraint decoding module. Furthermore, the computer device may determine the first retrieval loss of the added initial retrieval model for the first sample retrieval request based on the first reference retrieval result and the first sample retrieval result. The computer device may determine the third retrieval loss of the initial retrieval model for the third sample retrieval request based on the third reference retrieval result and the third sample retrieval result. The computer device may adjust the model parameters corresponding to the prefix vector module and the constraint decoding module respectively according to the first retrieval loss and the second retrieval loss to obtain a multimodal retrieval model.

[0156] Of course, the computer device may also use a mixed sample set including the first sample retrieval request, the second sample retrieval request, and the third sample retrieval request to train the added initial retrieval model. In this way, the trained multimodal retrieval model can perform both single-modal data retrieval and multimodal data retrieval, thereby improving the applicability of the multimodal retrieval model.

[0157] In an embodiment of the present application, the initial retrieval model has a strong content comprehension ability and language generation ability, and has a large processing potential for data retrieval. Therefore, in the training process of the initial retrieval model, the model parameters of the original module (modal recognition module) in the initial retrieval model remain unchanged, and only the model parameters of the prefix vector module and the constraint decoding model need to be lightly fine-tuned so that the prefix vector module and the constraint decoding model can learn knowledge about data retrieval and obtain a multimodal retrieval model. This can reduce the adjustment of model parameters, reduce the training cost of the multimodal retrieval model, and improve the training efficiency of the multimodal retrieval model. By keeping the model parameters of the original module (i.e., modal recognition) in the initial retrieval model unchanged, the content comprehension ability and generation ability of the initial retrieval model can be reused, and the problem of catastrophic forgetting of the generative language module due to the update of training data can be avoided, and the training stability of the multimodal retrieval model can be ensured. During the training of the initial retrieval model, only lightweight fine-tuning of the constraint decoding module and the prefix vector module is required. Therefore, a large amount of sample data is not required, which can avoid the problem of low model training accuracy caused by insufficient sample data. This is conducive to improving the accuracy of model training and, in turn, improving the retrieval accuracy of the multimodal model during the retrieval process.

[0158] Further, please refer to Figure 8, which is a structural diagram of a device for generating a multimodal retrieval model provided in an embodiment of the present application. The device for generating the multimodal retrieval model can be a computer program (including program code) running in a computer device. For example, the device for generating the multimodal retrieval model is an application software; the device for generating the multimodal retrieval model can be used to execute the corresponding steps in the method provided in an embodiment of the present application. As shown in Figure 8, the device for generating the multimodal retrieval model can be any blockchain node in the blockchain network. The device for generating the multimodal retrieval model can include: an acquisition module 11, a first generation module 12, a second generation module 13, a first retrieval module 14, an adjustment module 15, a third generation module 16, a fourth generation module 17, a second retrieval module 18, a fifth generation module 19, a sixth generation module 20, an acquisition module 21, a seventh generation module 22, an identification module 23, and an output module 24.

[0159] An acquisition module 11 is configured to acquire an initial retrieval model, the initial retrieval model comprising a modality recognition module, a prefix vector module, and a constraint decoding module; acquire a first sample retrieval request, the first sample retrieval request comprising first sample text modality data and first sample image modality data associated with each other, and comprising a first reference retrieval result pre-annotated for the first sample retrieval request;

[0160] a first generating module 12, configured to perform feature recognition on the first sample image modality data through the prefix vector module to obtain a first image prefix vector;

[0161] A second generating module 13 is configured to generate a first search character according to the first image prefix vector, the first sample image modal data, and the first sample text modal data through the modality recognition module;

[0162] A first retrieval module 14 is configured to obtain a first sample retrieval result associated with the first retrieval character from a pre-generated database through a constraint decoding module;

[0163] The adjustment module 15 is used to freeze the model parameters corresponding to the modal identification module, and adjust the model parameters corresponding to the prefix vector module and the constraint decoding module according to the first reference retrieval result and the first sample retrieval result to obtain a multimodal retrieval model.

[0164] The first generating module 12 is specifically configured to:

[0165] Performing object cropping on the first sample image modality data to obtain object image data, and performing feature extraction on the object image data through a feature extraction layer in a prefix vector module to obtain an object image feature vector;

[0166] Through the projection layer in the prefix vector module, the object image feature vector is linearly transformed and dimensionally reduced to obtain the processed object image feature vector;

[0167] The first image prefix vector is obtained by performing vector summation on the adjusted feature vector in the prefix vector layer and the processed object image feature vector through the prefix vector layer in the prefix vector module.

[0168] The modality recognition module includes a visual representation submodule and a language recognition submodule; the second generation module 13 is specifically used to:

[0169] Performing feature extraction on the first sample image modality data through the visual representation submodule to obtain an image modality feature vector, and performing attention feature extraction on the first image prefix vector and the image modality feature vector to obtain a visual representation vector;

[0170] Adding the first sample text modality data to the text slot in the data template, adding the visual representation vector to the image slot in the data template to obtain an added data template, and inputting the added data template into the language recognition submodule;

[0171] The language recognition submodule is used to extract features from the first sample text modality data in the text slot to obtain text modality features. The first retrieval character is generated based on the text modality features and the visual representation vector in the image slot.

[0172] The visual representation submodule includes N conversion layers, the prefix vector module includes N prefix vector layers corresponding to the N conversion layers, one conversion layer corresponds to one prefix vector layer, and the first image prefix vector includes image prefix vectors outputted by the N prefix vector layers respectively; N is a positive integer;

[0173] The second generating module 13 is further specifically configured to:

[0174] Obtain the image conversion feature corresponding to the i-th conversion layer among the N conversion layers; when i=1, the image conversion feature corresponding to the first conversion layer is generated by the first conversion layer based on the image modal feature vector and the image prefix vector output by the first prefix vector layer; i is a positive integer less than or equal to N;

[0175] Generate an image conversion feature corresponding to the i+1th conversion layer through the i+1th conversion layer among the N conversion layers based on the image conversion feature corresponding to the i-th conversion layer and the image prefix vector output by the i+1th prefix vector layer;

[0176] Until the image conversion feature corresponding to the Nth conversion layer among the N conversion layers is obtained, a visual representation vector is generated according to the image conversion feature corresponding to the Nth conversion layer.

[0177] The second generating module 13 is further specifically configured to:

[0178] Through the i+1th conversion layer among the N conversion layers, the image conversion feature corresponding to the i-th conversion layer is multiplied by the query weight vector in the i+1th conversion layer to obtain the processed image conversion feature;

[0179] The processed image conversion features are combined with the image prefix vector output by the i+1th prefix vector layer to perform self-attention feature extraction to obtain the first attention feature;

[0180] For the processed image conversion features, the image conversion features corresponding to the i-th conversion layer are subjected to self-attention feature extraction to obtain the second attention feature;

[0181] Perform effective feature screening on the first attention feature to obtain an effective first attention feature, and sum the effective first attention feature and the second attention feature to obtain a total attention feature;

[0182] Generate the image conversion features corresponding to the i+1th conversion layer based on the total attention features.

[0183] The second generating module 13 is further specifically configured to:

[0184] Multiply the image prefix vector output by the i+1th prefix vector layer by the key weight vector corresponding to the i+1th prefix vector layer to obtain a prefix key vector;

[0185] Multiply the image prefix vector output by the i+1th prefix vector layer by the value weight vector corresponding to the i+1th prefix vector layer to obtain a prefix value vector;

[0186] Multiplying the processed image conversion feature and the transpose of the prefix key vector to obtain a first similarity vector;

[0187] The first similarity vector and the prefix value vector are multiplied to obtain the first attention feature.

[0188] The second generating module 13 is further specifically configured to:

[0189] Multiply the image conversion feature corresponding to the i-th conversion layer by the key weight vector corresponding to the i-th conversion layer to obtain the image key vector;

[0190] Multiply the image conversion feature corresponding to the i-th conversion layer by the value weight vector corresponding to the i-th conversion layer to obtain an image value vector;

[0191] Multiplying the processed image conversion feature and the transpose of the image key vector to obtain a second similarity vector;

[0192] The second similarity vector and the image value vector are multiplied to obtain a second attention feature.

[0193] The first search module 14 is specifically used for:

[0194] Through the character acquisition interface in the constraint decoding module, according to the first search character, a candidate character set matching the first search character is searched from a pre-generated database; the pre-generated database includes characters corresponding to P document data, where P is a positive integer;

[0195] Obtaining a matching probability between a candidate character in the candidate character set and the first search character;

[0196] Determine a candidate character with the greatest matching probability from the candidate character set as a suffix character of the first search character;

[0197] The first search character and the suffix character are combined to obtain a combined character sequence, and a first sample search result associated with the first sample search request is retrieved according to the combined character sequence.

[0198] The first search module 14 is further specifically configured to:

[0199] The validity of the combined character sequence is verified through the validity verification interface in the constraint decoding module to obtain the verification result;

[0200] If the verification result indicates that the combined character sequence is valid, document data having the combined character sequence is retrieved from the pre-generated database through the document retrieval interface in the constraint decoding module;

[0201] The retrieved document data is determined as a first sample retrieval result associated with the first sample retrieval request.

[0202] The first search module 14 is further specifically configured to:

[0203] Obtain the sequence length of the combined character sequence through the validity verification interface in the constraint decoding module;

[0204] If the sequence length is greater than or equal to the sequence length threshold, the combined character sequence is determined to be valid;

[0205] Generates a validation result indicating that the combining character sequence is valid.

[0206] The first search module 14 is further specifically configured to:

[0207] If the sequence length is less than the sequence length threshold, the document data with the combined character sequence is obtained from the pre-generated database to obtain the matching document data;

[0208] Obtaining the number of matching document data, and if the number of matching document data is equal to a quantity threshold, determining that the combined character sequence is valid;

[0209] Generates a validation result indicating that the combining character sequence is valid.

[0210] The adjustment module 15 is specifically configured to:

[0211] Determining a model loss value of the initial retrieval model according to the first reference retrieval result and the first sample retrieval result;

[0212] Determine the parameter adjustment gradients corresponding to the prefix vector module and the constraint decoding module respectively according to the model loss value and the model loss function of the initial retrieval model;

[0213] The gradient is adjusted according to the parameters corresponding to the prefix vector module, the model parameters corresponding to the prefix vector module are adjusted, the gradient is adjusted according to the parameters corresponding to the constraint decoding module, the model parameters of the constraint decoding module are adjusted, and a multimodal retrieval model is obtained.

[0214] The generating device of the multimodal retrieval model further includes:

[0215] a third generation module 16 configured to obtain a second sample retrieval request, the second sample retrieval request including second sample image modality data and a second reference retrieval result pre-annotated for the second sample retrieval request; and perform feature recognition on the first sample image modality data using a prefix vector module to obtain a second image prefix vector;

[0216] A fourth generating module 17 is configured to generate a second search character according to the second image prefix vector and the second sample image modality data through the modality recognition module;

[0217] A second retrieval module 18 is configured to obtain a second sample retrieval result associated with the second retrieval character from the pre-generated database through a constraint decoding module;

[0218] The adjustment module 15 is further specifically configured to:

[0219] According to the first reference retrieval result and the first sample retrieval result, as well as the second reference retrieval result and the second sample retrieval result, the model parameters corresponding to the prefix vector module and the constraint decoding module are adjusted to obtain a multimodal retrieval model.

[0220] The generating device of the multimodal retrieval model further includes:

[0221] A fifth generation module 19 is configured to obtain a third sample search request, the third sample search request including the second sample text modal data and a third reference search result pre-annotated for the third sample search request; and generate a third search character based on the second sample text modal data using the modality recognition module.

[0222] a sixth generating module 20, configured to obtain, from the pre-generated database through a constraint decoding module, a third sample search result associated with the third search character;

[0223] The adjustment module 15 is further specifically configured to:

[0224] According to the first reference retrieval result and the first sample retrieval result, as well as the third reference retrieval result and the third sample retrieval result, the model parameters corresponding to the prefix vector module and the constraint decoding module are adjusted to obtain a multimodal retrieval model.

[0225] The generating device of the multimodal retrieval model further includes:

[0226] The acquisition module 21 is used to acquire a data retrieval request input by a business object; the data retrieval request includes retrieval text modal data and retrieval image modal data having an associated relationship;

[0227] a seventh generating module 22, configured to perform feature recognition on the retrieval image modality data through a prefix vector module in a multimodal retrieval model to obtain a retrieval image prefix vector;

[0228] The recognition module 23 is configured to obtain a target retrieval character by recognizing the target retrieval character according to the retrieval image prefix vector, the retrieval text modal data, and the retrieval image modal data through the modal recognition module in the multimodal retrieval model;

[0229] The output module 24 is configured to obtain target retrieval document data associated with the data retrieval request according to the target retrieval character through the constraint decoding module in the multimodal retrieval model, and output the target retrieval document data to the business object.

[0230] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part using software, hardware (such as processing circuits or memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that contains the functions of the module or unit. According to one embodiment of the present application, the various modules in the device for generating a multimodal retrieval model shown in Figure 8 can be individually or completely combined into one or several units to form a unit, or one (or some) of the units can be further divided into at least two functionally smaller sub-units to achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In actual applications, the functions of one module can also be implemented by at least two units, or the functions of at least two modules can be implemented by one unit. In other embodiments of the present application, the device for generating a multimodal retrieval model can also include other units. In actual applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of at least two units.

[0231] According to one embodiment of the present application, a computer program (including program code) capable of executing the steps involved in the corresponding method as shown in Figure 3 can be run on a general-purpose computer device such as a computer, which includes a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM), and other processing elements and storage elements, to construct a device for generating a multimodal retrieval model as shown in Figure 8, and to implement a method for generating a multimodal retrieval model according to an embodiment of the present application. The above-mentioned computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computer device through the computer-readable recording medium and run therein.

[0232] In an embodiment of the present application, the initial retrieval model has a strong content comprehension ability and language generation ability, and has a large processing potential for data retrieval. Therefore, in the training process of the initial retrieval model, the model parameters of the original module (modal recognition module) in the initial retrieval model remain unchanged, and only the model parameters of the prefix vector module and the constraint decoding model need to be lightly fine-tuned so that the prefix vector module and the constraint decoding model can learn knowledge about data retrieval and obtain a multimodal retrieval model. This can reduce the adjustment of model parameters, reduce the training cost of the multimodal retrieval model, and improve the training efficiency of the multimodal retrieval model. By keeping the model parameters of the original module (i.e., modal recognition) in the initial retrieval model unchanged, the content comprehension ability and generation ability of the initial retrieval model can be reused, and the problem of catastrophic forgetting of the generative language module due to the update of training data can be avoided, and the training stability of the multimodal retrieval model can be ensured. During the training of the initial retrieval model, only lightweight fine-tuning of the constraint decoding module and the prefix vector module is required. Therefore, a large amount of sample data is not required, which can avoid the problem of low model training accuracy caused by insufficient sample data. This is conducive to improving the accuracy of model training and, in turn, improving the retrieval accuracy of the multimodal model during the retrieval process.

[0233] Further, refer to Figure 9, which is a schematic diagram of a computer device provided in an embodiment of the present application. As shown in Figure 9, the computer device 3000 can be a terminal device or server in the embodiment corresponding to Figure 2 above, and the computer device 3000 can include: at least one processor 3001, such as a CPU, at least one network interface 3004, a user interface 3003, a memory 3005, and at least one communication bus 3002. Wherein, the communication bus 3002 is used to realize the connection communication between these components. Wherein, the user interface 3003 can include a display screen (Display), a keyboard (Keyboard), and the network interface 3004 can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 3005 can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The storage 3005 can optionally also be at least one storage device located away from the aforementioned processor 3001. As shown in Figure 9, the memory 3005 as a computer storage medium can include an operating system, a network communication module, a user interface module, and a computer program control application.

[0234] In the computer device 3000 shown in FIG9 , the network interface 3004 is mainly used for the second node device to communicate with the target relay server and the target oracle server through the network; the user interface 3003 is mainly used to provide an interface for user input; and the processor 3001 can be used to call the computer program control application stored in the memory 3005 to achieve:

[0235] Acquire an initial retrieval model, wherein the initial retrieval model includes a modality recognition module, a prefix vector module, and a constraint decoding module;

[0236] Obtaining a first sample retrieval request, the first sample retrieval request including mutually associated first sample text modality data and first sample image modality data, and including a first reference retrieval result pre-annotated for the first sample retrieval request;

[0237] Performing feature recognition on the first sample image modality data by the prefix vector module to obtain a first image prefix vector;

[0238] Generate a first search character according to the first image prefix vector, the first sample image modality data, and the first sample text modality data by the modality recognition module;

[0239] Obtaining, by the constraint decoding module, a first sample search result associated with the first search character from a pre-generated database;

[0240] According to the first reference retrieval result and the first sample retrieval result, the model parameters corresponding to the prefix vector module and the constraint decoding module are adjusted respectively to obtain an adjusted retrieval model.

[0241] It should be understood that the computer device 3000 described in the embodiments of the present application can also execute the description of the method for generating a multimodal retrieval model in the embodiment corresponding to FIG. 7 , and the computer device 3000 described in the embodiments of the present application can also execute the description of the apparatus for generating a multimodal retrieval model in the embodiment corresponding to FIG. 8 , and these descriptions will not be repeated here. Furthermore, the description of the beneficial effects of employing the same method will not be repeated here.

[0242] In addition, it should be noted here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the aforementioned multimodal retrieval model generation device, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the description of the method for generating a multimodal retrieval model in the embodiment corresponding to Figure 3 or Figure 7 above, so it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network. Multiple computing devices distributed at multiple locations and interconnected by a communication network can constitute a blockchain system.

[0243] On one hand, the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, so that the computer device can perform the description of the method for generating a multimodal retrieval model in the embodiment corresponding to Figure 3 or Figure 7 above, which will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated.

[0244] It should be noted that the collection and processing of relevant data in this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in practice, and the informed consent or separate consent of the personal information subject (or a legal basis) should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject. For example, when this application obtains data such as retrieval requests (such as data retrieval requests, first sample retrieval requests, second sample retrieval requests, etc.) input by sample objects or business objects, the informed consent or separate consent of the corresponding business objects or sample objects must be obtained.

[0245] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0246] The above disclosure is merely a preferred embodiment of the present invention and certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for generating a multimodal retrieval model, applied to a computer device, characterized in that: include: Acquire an initial retrieval model, wherein the initial retrieval model includes a modality recognition module, a prefix vector module, and a constraint decoding module; Obtaining a first sample retrieval request, the first sample retrieval request including mutually associated first sample text modality data and first sample image modality data, and including a first reference retrieval result pre-annotated for the first sample retrieval request; Performing feature recognition on the first sample image modality data by the prefix vector module to obtain a first image prefix vector; Generate a first search character according to the first image prefix vector, the first sample image modality data, and the first sample text modality data by the modality recognition module; Obtaining, by the constraint decoding module, a first sample search result associated with the first search character from a pre-generated database; According to the first reference retrieval result and the first sample retrieval result, the model parameters corresponding to the prefix vector module and the constraint decoding module are adjusted respectively to obtain an adjusted retrieval model.

2. The method according to claim 1, characterized in that The method of performing feature recognition on the first sample image modality data by the prefix vector module to obtain a first image prefix vector includes: Performing object cropping on the first sample image modality data to obtain object image data, and performing feature extraction on the object image data through a feature extraction layer in the prefix vector module to obtain an object image feature vector; Performing linear transformation and dimensionality reduction processing on the object image feature vector through the projection layer in the prefix vector module to obtain a processed object image feature vector; The prefix vector layer in the prefix vector module performs vector summation on the adjusted feature vector in the prefix vector layer and the processed object image feature vector to obtain a first image prefix vector.

3. The method according to claim 1, characterized in that The modality recognition module includes a visual representation submodule and a language recognition submodule; Generating a first search character according to the first image prefix vector, the first sample image modality data, and the first sample text modality data by the modality recognition module includes: Performing feature extraction on the first sample image modality data through the visual representation submodule to obtain an image modality feature vector, and performing attention feature extraction on the first image prefix vector and the image modality feature vector to obtain a visual representation vector; Adding the first sample text modality data to the text slot in the data template, adding the visual representation vector to the image slot in the data template to obtain an added data template, and inputting the added data template into the language recognition submodule; The language recognition submodule performs feature extraction on the first sample text modality data in the text slot to obtain text modality features, and generates a first search character based on the text modality features and the visual representation vector in the image slot.

4. The method according to claim 3, characterized in that The visual representation submodule includes N conversion layers, the prefix vector module includes N prefix vector layers corresponding to the N conversion layers, one conversion layer corresponds to one prefix vector layer, and the first image prefix vector includes image prefix vectors outputted respectively by the N prefix vector layers; N is a positive integer; The performing attention feature extraction on the first image prefix vector and the image modality feature vector to obtain a visual representation vector includes: Obtaining an image conversion feature corresponding to an i-th conversion layer among the N conversion layers; when i=1, the image conversion feature corresponding to the first conversion layer is generated by the first conversion layer according to the image modal feature vector and the image prefix vector output by the first prefix vector layer; i is a positive integer less than or equal to N; Generate, through the (i+1)th conversion layer among the N conversion layers, an image conversion feature corresponding to the (i+1)th conversion layer according to the image conversion feature corresponding to the (i)th conversion layer and the image prefix vector output by the (i+1)th prefix vector layer; Until the image conversion feature corresponding to the Nth conversion layer among the N conversion layers is obtained, a visual representation vector is generated according to the image conversion feature corresponding to the Nth conversion layer.

5. The method according to claim 4, characterized in that The step of generating, through the (i+1)th conversion layer among the N conversion layers, an image conversion feature corresponding to the (i+1)th conversion layer according to the image conversion feature corresponding to the (i)th conversion layer and the image prefix vector output by the (i+1)th prefix vector layer, comprising: multiplying the image conversion feature corresponding to the i-th conversion layer by the query weight vector in the i+1-th conversion layer through the i+1-th conversion layer among the N conversion layers to obtain a processed image conversion feature; Performing self-attention feature extraction on the processed image conversion feature and the image prefix vector output by the (i+1)th prefix vector layer to obtain a first attention feature; Performing self-attention feature extraction on the processed image conversion feature and the image conversion feature corresponding to the i-th conversion layer to obtain a second attention feature; Performing effective feature screening on the first attention feature to obtain an effective first attention feature, and summing the effective first attention feature and the second attention feature to obtain a total attention feature; Generate the image conversion feature corresponding to the (i+1)th conversion layer according to the total attention feature.

6. The method according to claim 5, characterized in that The processed image conversion feature and the image prefix vector output by the (i+1)th prefix vector layer are subjected to self-attention feature extraction to obtain a first attention feature, including: Multiplying the image prefix vector output by the (i+1)th prefix vector layer by the key weight vector corresponding to the (i+1)th prefix vector layer to obtain a prefix key vector; Multiplying the image prefix vector output by the i+1th prefix vector layer by the value weight vector corresponding to the i+1th prefix vector layer to obtain a prefix value vector; multiplying the processed image conversion feature by the transpose of the prefix key vector to obtain a first similarity vector; The first similarity vector and the prefix value vector are multiplied to obtain a first attention feature.

7. The method according to claim 5, characterized in that The performing self-attention feature extraction on the processed image conversion feature and the image conversion feature corresponding to the i-th conversion layer to obtain a second attention feature includes: Multiplying the image conversion feature corresponding to the i-th conversion layer by the key weight vector corresponding to the i-th conversion layer to obtain an image key vector; Multiplying the image conversion feature corresponding to the i-th conversion layer by the value weight vector corresponding to the i-th conversion layer to obtain an image value vector; multiplying the processed image conversion feature by the transpose of the image key vector to obtain a second similarity vector; The second similarity vector and the image value vector are multiplied to obtain a second attention feature.

8. The method according to claim 1, characterized in that The step of obtaining, from a pre-generated database, a first sample search result associated with the first search character by the constraint decoding module includes: Through the character acquisition interface in the constraint decoding module, according to the first search character, a candidate character set matching the first search character is searched from a pre-generated database; the pre-generated database includes characters corresponding to P document data, where P is a positive integer; Obtaining a matching probability between a candidate character in the candidate character set and the first search character; Determining a candidate character with the greatest matching probability from the candidate character set as a suffix character of the first search character; The first search character and the suffix character are combined to obtain a combined character sequence, and a first sample search result associated with the first search character is obtained from a pre-generated database according to the combined character sequence.

9. The method according to claim 8, characterized in that The step of obtaining a first sample search result associated with the first search character from a pre-generated database according to the combined character sequence includes: Verifying the validity of the combined character sequence through a validity verification interface in the constraint decoding module to obtain a verification result; If the verification result indicates that the combined character sequence is valid, retrieving document data having the combined character sequence from the pre-generated database through a document retrieval interface in the constraint decoding module; The retrieved document data is determined as a first sample retrieval result associated with the first retrieval character.

10. The method according to claim 9, characterized in that The validation of the combined character sequence is performed through the validation interface in the constraint decoding module to obtain a validation result, including: Obtaining the sequence length of the combined character sequence through the validity verification interface in the constraint decoding module; If the sequence length is greater than or equal to the sequence length threshold, determining that the combined character sequence is valid; A verification result is generated indicating that the combined character sequence is valid.

11. The method according to claim 10, characterized in that The method further comprises: If the sequence length is less than the sequence length threshold, obtaining document data having the combined character sequence from the pre-generated database to obtain matching document data; Acquiring the number of the matching document data, and if the number of the matching document data is equal to a quantity threshold, determining that the combined character sequence is valid; A verification result is generated indicating that the combined character sequence is valid.

12. The method according to claim 1, characterized in that The adjusting, based on the first reference search result and the first sample search result, the model parameters corresponding to the prefix vector module and the constraint decoding module, respectively, to obtain an adjusted retrieval model, includes: Determining a model loss value of the initial retrieval model according to the first reference retrieval result, the first reference retrieval result, and the first sample retrieval result; Determining parameter adjustment gradients corresponding to the prefix vector module and the constraint decoding module respectively according to the model loss value and the model loss function of the initial retrieval model; The gradient is adjusted according to the parameters corresponding to the prefix vector module, and the model parameters corresponding to the prefix vector module are adjusted. The gradient is adjusted according to the parameters corresponding to the constraint decoding module, and the model parameters of the constraint decoding module are adjusted to obtain a multimodal retrieval model.

13. The method according to claim 1, wherein The method further comprises: Obtaining a second sample retrieval request, where the second sample retrieval request includes second sample image modality data and a second reference retrieval result pre-annotated for the second sample retrieval request; Performing feature recognition on the first sample image modality data by the prefix vector module to obtain a second image prefix vector; generating, by the modality recognition module, a second search character according to the second image prefix vector and the second sample image modality data; Obtaining, by the constraint decoding module, a second sample search result associated with the second search character from the pre-generated database; The method further comprises adjusting model parameters corresponding to the prefix vector module and the constraint decoding module according to the first reference search result and the first sample search result to obtain a multimodal retrieval model, including: According to the first reference retrieval result and the first sample retrieval result, and the second reference retrieval result and the second sample retrieval result, the model parameters corresponding to the prefix vector module and the constraint decoding module are adjusted respectively to obtain a multimodal retrieval model.

14. The method according to claim 1, wherein The method further comprises: Obtaining a third sample retrieval request, the third sample retrieval request including the second sample text modal data and a third reference retrieval result pre-annotated for the third sample retrieval request; generating a third search character according to the second sample text modality data by the modality recognition module; Obtaining, by the constraint decoding module, a third sample search result associated with the third search character from the pre-generated database; The method further comprises adjusting model parameters corresponding to the prefix vector module and the constraint decoding module according to the first reference search result and the first sample search result to obtain a multimodal retrieval model, including: According to the first reference retrieval result and the first sample retrieval result, and the third reference retrieval result and the third sample retrieval result, the model parameters corresponding to the prefix vector module and the constraint decoding module are adjusted respectively to obtain a multimodal retrieval model.

15. The method according to claim 1, wherein The method further comprises: Obtaining a data retrieval request input by a business object; the data retrieval request includes retrieval text modal data and retrieval image modal data having an associated relationship; Performing feature recognition on the retrieval image modality data through a prefix vector module in the multimodal retrieval model to obtain a retrieval image prefix vector; The modality recognition module in the multimodal retrieval model recognizes the retrieval image prefix vector, the retrieval text modality data, and the retrieval image modality data to obtain a target retrieval character; The constraint decoding module in the multimodal retrieval model is used to obtain target retrieval document data associated with the data retrieval request according to the target retrieval character, and the target retrieval document data is output to the business object.

16. A device for generating a multimodal retrieval model, characterized in that: include: An acquisition module, configured to acquire an initial retrieval model, wherein the initial retrieval model includes a modality recognition module, a prefix vector module, and a constraint decoding module; Obtaining a first sample retrieval request, the first sample retrieval request including mutually associated first sample text modality data and first sample image modality data, and including a first reference retrieval result pre-annotated for the first sample retrieval request; a first generating module, configured to perform feature recognition on the first sample image modality data through the prefix vector module to obtain a first image prefix vector; a second generating module, configured to generate a first search character according to the first image prefix vector, the first sample image modal data, and the first sample text modal data through the modality recognition module; A first retrieval module, configured to obtain a first sample retrieval result associated with the first retrieval character from a pre-generated database through the constraint decoding module; An adjustment module is used to adjust the model parameters corresponding to the prefix vector module and the constraint decoding module respectively according to the first reference retrieval result and the first sample retrieval result to obtain an adjusted retrieval model.

17. A computer device, characterized in that: include: processor and memory; The processor is connected to a memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 15.

19. A computer program product or a computer program, characterized in that The computer program product or computer program includes computer instructions, which are stored in a computer-readable storage medium. The computer instructions are suitable for being read and executed by a processor, so as to enable a computer device having the processor to perform the method according to any one of claims 1 to 15.

Images