Data processing method and device, storage medium, equipment and program product

Abstract

The invention discloses a data processing method and device, a storage medium, equipment and a program product, which are applied to a data processing scene. The method comprises the following steps: obtaining query data, wherein the query data comprises at least one modal data of a text modal, an image modal and an audio modal; based on the query data, a query feature vector of the target feature dimension is generated through a multi-modal retrieval model, and the multi-modal retrieval model is configured to be capable of outputting feature vectors of multiple different feature dimensions; and performing similarity calculation on the query feature vector of the target feature dimension and a candidate feature vector corresponding to each candidate data in the candidate data set to obtain a retrieval result. According to the method, by maintaining the multi-modal retrieval model, the output of the feature vectors of different feature dimensions is supported, the resource consumption is reduced, the flexibility is improved, and the retrieval performance can be ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data processing method and device, storage medium, equipment and program product. BACKGROUND

[0002] With the rapid development of information technology, the generation and propagation of multi-modal data on the Internet has increased dramatically. How to efficiently retrieve the information required by users from massive multi-modal data has become a problem to be solved. Traditional multi-modal retrieval models usually output fixed feature dimensions. To meet different feature dimension requirements (such as 128-dimensional, 256-dimensional, and 512-dimensional feature vectors), multiple different models need to be trained independently. Training and maintaining multiple fixed dimension models require a large amount of computing resources and storage space, especially in the case of supporting multiple feature dimensions, the cost will increase exponentially, and the resource consumption is high. The traditional technology is difficult to dynamically adjust the feature dimension according to the actual demand, and the flexibility is poor. Although model compression, pruning and feature selection methods can reduce the model size and speed up the inference, they usually sacrifice the retrieval performance, and the dimension conversion method will also affect the retrieval effect due to information loss, resulting in performance impairment.

[0003] Therefore, there are problems of high resource consumption, poor flexibility and performance impairment in the traditional technology. SUMMARY

[0004] The embodiments of the present application provide a data processing method, device, storage medium, equipment and program product, by maintaining one multi-modal retrieval model, supporting output of feature vectors of different feature dimensions, reducing resource consumption, improving flexibility, and ensuring retrieval performance.

[0005] In one aspect, the embodiments of the present application provide a data processing method, comprising: acquiring query data, the query data comprising at least one modal data in text modal, image modal and audio modal; generating a query feature vector of a target feature dimension based on the query data through a multi-modal retrieval model, wherein the multi-modal retrieval model is configured to be able to output feature vectors of multiple different feature dimensions; and performing similarity calculation on the query feature vector of the target feature dimension and candidate feature vectors corresponding to candidate data in a candidate data set to obtain a retrieval result.

[0006] In another aspect, the embodiments of the present application provide a data processing device, comprising: An acquisition unit is configured to acquire query data, the query data comprising at least one modal data in text modal, image modal and audio modal; The processing unit is configured to generate a query feature vector of the target feature dimension based on the query data through a multimodal retrieval model, wherein the multimodal retrieval model is configured to output feature vectors of multiple different feature dimensions. The retrieval unit is used to calculate the similarity between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset, so as to obtain the retrieval result.

[0007] On the other hand, an embodiment of this application provides a computer-readable storage medium storing a computer program adapted for loading by a processor to perform the data processing method as described in any of the above embodiments.

[0008] On the other hand, an embodiment of this application provides a computer device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the data processing method described in any of the above embodiments by calling the computer program stored in the memory.

[0009] On the other hand, an embodiment of this application provides a computer program product, including computer instructions, which, when executed by a processor, implement the data processing method as described in any of the above embodiments.

[0010] This application embodiment acquires query data, which includes at least one modality of data: text, image, and audio. Based on the query data, a multimodal retrieval model is used to generate a query feature vector of the target feature dimension. This multimodal retrieval model is configured to output feature vectors of multiple different feature dimensions. The similarity between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset is calculated to obtain the retrieval results. This application embodiment maintains a multimodal retrieval model, supports the output of feature vectors of different feature dimensions, reduces resource consumption, improves flexibility, and ensures retrieval performance. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram illustrating an application scenario of the data processing system provided in an embodiment of this application.

[0013] Figure 2 This is a flowchart illustrating the data processing method provided in an embodiment of this application.

[0014] Figure 3 This is a schematic diagram of a first application scenario of the data processing method provided in the embodiments of this application.

[0015] Figure 4 This is a schematic diagram of a second application scenario for the data processing method provided in the embodiments of this application.

[0016] Figure 5 This is a schematic diagram of a third application scenario of the data processing method provided in the embodiments of this application.

[0017] Figure 6 This is a schematic diagram of a fourth application scenario of the data processing method provided in the embodiments of this application.

[0018] Figure 7 This is a schematic diagram of the fifth application scenario of the data processing method provided in the embodiments of this application.

[0019] Figure 8 This is a schematic diagram of the sixth application scenario of the data processing method provided in the embodiments of this application.

[0020] Figure 9 This is a schematic diagram of the seventh application scenario of the data processing method provided in the embodiments of this application.

[0021] Figure 10 This is a schematic diagram of the eighth application scenario of the data processing method provided in the embodiments of this application.

[0022] Figure 11 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application.

[0023] Figure 12 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] This application provides a data processing method, apparatus, storage medium, device, and program product. Exemplarily, the data processing method of this application can be executed by a computer device, which can be a terminal or server, etc. The terminal can be a smartphone, tablet, laptop, desktop computer, smart TV, smart speaker, wearable smart device, personal computer (PC), smart vehicle terminal, etc., and may also include a client. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0026] The embodiments of this application can be applied to scenarios such as data processing, data retrieval, multimodal retrieval, similar video retrieval, cross-modal content recommendation, and video content review.

[0027] First, the following explanations are given for some of the nouns or terms that appear in the embodiments of this application: Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the vision field, such as Swin-transformer, ViT, V-MOE, and MAE, can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0028] Speech Technology: Key technologies include Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and Voiceprint Recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech being one of the most promising methods. Large-scale modeling technology has revolutionized speech technology development. Pre-trained models such as WavLM and UniSpeech, which utilize the Transformer architecture, possess strong generalization and versatility, enabling them to excel in various speech processing tasks.

[0029] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP deals with natural language, the language people use in daily life, and is closely related to linguistics; it also involves computer science and mathematics. Pre-trained models, a crucial technique for model training in artificial intelligence, evolved from large language models in NLP. After fine-tuning, large language models can be widely applied to downstream tasks. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0030] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning. Pre-trained models are the latest development in deep learning, integrating all of these techniques.

[0031] Large Language Models (LLMs) are models trained using deep learning techniques, particularly neural networks, to process and generate natural language text. They typically contain billions or even hundreds of billions of parameters and are capable of capturing and understanding complex language patterns and semantic relationships.

[0032] Multimodal Large Model: A large deep learning model capable of processing and fusing content from multiple modalities (such as vision, language, audio, etc.), typically containing billions or even hundreds of billions of parameters. These models are able to understand and generate content related to multiple modalities.

[0033] Attention is a technique that allows neural networks to dynamically focus on the most relevant parts of the input data. By calculating the importance weights of different elements, it enhances the processing of key information. Its core idea is "selective attention," simulating the attention allocation mechanism in human cognition.

[0034] Convolutional Neural Network (CNN): A special type of artificial neural network in deep learning, primarily used to process data with a grid structure, such as images.

[0035] Retrieval-Augmented Generation (RAG) is a natural language processing technique that combines retrieval and generation to improve the accuracy and richness of generation tasks such as question answering and text generation. RAG models enhance the generation process by retrieving relevant documents or information, thereby generating more accurate and context-sensitive text.

[0036] Contrastive Language-Image Pre-Training (CLIP) is a multimodal machine learning model that understands the relationships between images and text. This model is pre-trained on a large number of images and related descriptive text to learn how to connect visual and linguistic information.

[0037] Transformer: A deep learning model for processing sequential data, primarily relying on attention mechanisms. This model processes input data sequences through self-attention and multi-head attention mechanisms, effectively capturing the relationships between elements in the sequence. Transformer models typically include encoder and decoder structures and are widely used in fields such as natural language processing and image processing.

[0038] Bidirectional Encoder Representations from Transformers (BERT) is a pre-trained language model based on transformers. It performs exceptionally well in natural language processing tasks.

[0039] The Vision Transformer (ViT) is a computer vision model based on the Transformer architecture. While the Transformer was originally designed for natural language processing tasks, ViT has been successfully applied to visual tasks such as image classification.

[0040] Feature vectors (Embedding): In machine learning and natural language processing, embedding is a method for converting high-dimensional data (such as words, sentences, or images) into low-dimensional vector representations. This representation preserves the semantic information of the data and makes it suitable as input to machine learning models.

[0041] Cascade Structure: This refers to multiple neural network modules or processing stages connected sequentially, with the output of the previous stage serving as the input of the next stage, progressively optimizing or refining the task results.

[0042] Knowledge distillation is a deep learning model compression and optimization technique. Its core idea is to use a "teacher-student framework" to transfer knowledge from a complex large model (teacher model) to a lighter small model (student model), so that the student model can significantly reduce the amount of computation and storage requirements while maintaining high performance.

[0043] Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) technique used to adapt pre-trained Large Language Models (LLMs) or Multimodal Large Language Models (MLLMs) to new tasks without fully fine-tuning all parameters. Its core idea is to use low-rank matrix factorization to train only a small number of additional parameters, allowing the model to adapt to new tasks while preserving existing knowledge.

[0044] Information-Noise Contrastive Estimation Loss (Info-NCE Loss): This is a loss function used in contrastive learning that optimizes the model by comparing the similarity between positive and negative sample pairs, ensuring that the similarity between positive sample pairs is higher than that between negative sample pairs. A similar loss function may be used to optimize model performance in the contrastive learning framework of this application.

[0045] Kullback-Leibler Divergence (KL) is a method for measuring the difference between two probability distributions. In scenarios such as knowledge distillation, KL divergence can be used to measure the difference between the output distributions of the teacher model and the student model, thereby guiding the training of the student model. Although not explicitly stated in the direct disclosure, it is a commonly used concept in machine learning.

[0046] Multimodal retrieval technology integrates cross-media data such as text, images, video, and audio to achieve more accurate and intelligent information search and matching. Its core lies in extracting multimodal features and performing similarity retrieval, such as searching for related images based on text descriptions or retrieving related text based on image content.

[0047] However, in practical applications, different business scenarios have different requirements for the size of feature dimensions: Larger feature dimensions: can extract richer feature information and improve retrieval results, but will lead to slower retrieval speed and increased storage space occupied by the retrieval database.

[0048] Smaller feature dimensions: This enables faster retrieval speeds and reduces the storage space occupied by the retrieval database, but may result in the loss of some feature information, affecting retrieval performance.

[0049] In practical applications, appropriate solutions need to be selected based on the requirements of different business scenarios. This necessitates training a separate model for each feature dimension, increasing the cost of model training and maintenance. Therefore, designing a model capable of dynamically outputting different feature dimensions for various business scenarios has become a critical issue that urgently needs to be addressed.

[0050] This application proposes a multimodal retrieval model with dynamic feature dimension output, which enables the model to adapt to different business scenarios. This application mainly includes the following two key technologies: Multi-head structure: The model adopts a multi-head structure design, with each head corresponding to a different feature dimension output. Only one model needs to be maintained to support multiple feature dimension outputs simultaneously, meeting the needs of different business scenarios.

[0051] Multi-head distillation: To ensure consistent model performance across different feature dimensions, multi-head distillation is employed. First, a high-performing large model is trained as the teacher model. Then, its knowledge is distilled into the output heads of various student models with different feature dimensions. This allows even models with smaller feature dimensions to achieve retrieval performance close to that of the large model.

[0052] The multimodal retrieval model with dynamic feature dimension output proposed in this application provides the following improvements: Reduced resource consumption: Only one model needs to be trained and maintained to support multiple feature dimensions of output, which significantly reduces training and maintenance costs.

[0053] Enhanced flexibility: The appropriate feature dimensions can be dynamically selected according to the needs of different business scenarios without retraining the model, thus improving the model's adaptability.

[0054] Ensuring retrieval performance: By employing multi-head distillation, knowledge from the large model is transferred to sub-models with different feature dimensions. Even with feature vectors of lower dimensions, high retrieval performance is maintained, avoiding performance loss due to dimensionality reduction. During distillation, no dimensional alignment operations are required; that is, the teacher model can maintain optimal performance using high-dimensional features and distill them onto the student model's outputs at different feature dimensions.

[0055] This application provides a more efficient, flexible and high-performance multimodal retrieval model that can better meet the needs of different feature dimensions in practical applications.

[0056] The solutions provided in this application involve technologies such as data processing, which are specifically illustrated in the following embodiments. These embodiments are described in detail below. It should be noted that the order of description of the following embodiments is not intended to limit the priority of the embodiments.

[0057] It is understood that in the specific implementation of this application, user query data and other related data are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0058] Please see Figure 1 , Figure 1This is a schematic diagram illustrating an application scenario of the data processing system provided in this application embodiment. The data processing system includes a terminal 10 and a server 20, etc.; the terminal 10 and server 20 are connected via a network, such as a wired or wireless network. The terminal 10 can be a smartphone, tablet, laptop, desktop computer, smart TV, smart speaker, wearable smart device, personal computer (PC), smart vehicle terminal, etc. The terminal 10 may also include a client, which can be a client SDK, browser client, or instant messaging client, etc. The server 20 can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0059] Terminal 10 can be used to display a graphical user interface. Terminal 10 is used to interact with the user through the graphical user interface, such as downloading and installing a corresponding client and running it, calling and running a corresponding app, or presenting the corresponding graphical user interface by logging into the client.

[0060] This application provides a data processing method, which can be executed by a terminal or a server, or by both a terminal and a server. This application uses the example of a data processing method executed by a server to illustrate the method.

[0061] This application provides a data processing method, which can be executed by a terminal or a server, or by both a terminal and a server. This application uses the example of a data processing method executed by a server to illustrate the method.

[0062] Please see Figures 2 to 10 , Figure 2 This is a flowchart illustrating the data processing method provided in an embodiment of this application. Figures 3 to 10 These are all schematic diagrams illustrating application scenarios of the data processing method provided in the embodiments of this application. The method may include the following steps: Step 110: Obtain query data, which includes at least one modal data among text modality, image modality, and audio modality.

[0063] For example, query data can be applied to multiple business scenarios, such as video platforms, news platforms, and content creation platforms, to retrieve similar videos, similar audio, and article content. The form of query data may be text, images (or videos), audio, a combination of text and images (or videos), a combination of text and audio, a combination of images (or videos) and audio, or a combination of text and images (or videos) and audio. Query data can be specified through instructions to define the retrieval task to be performed, such as retrieving similar text, similar image and text content, or materials related to a question.

[0064] Step 120: Based on the query data, generate a query feature vector of the target feature dimension through a multimodal retrieval model, wherein the multimodal retrieval model is configured to output feature vectors of multiple different feature dimensions.

[0065] In particular, the choice of target feature dimensions for multimodal retrieval models is related to the needs of different business scenarios. For example, for business scenarios with high latency requirements, feature vectors with smaller feature dimensions are selected for output; while for business scenarios with high accuracy requirements and low latency sensitivity, feature vectors with larger feature dimensions are selected for output.

[0066] like Figure 3 As shown, when inputting query data (such as query1, query2), the instruction specifies the retrieval task to be executed. The multimodal retrieval model will output query feature vectors with different target feature dimensions according to the needs of different business scenarios. For example, if business 1 has high latency requirements, it is suitable to use a smaller feature dimension as the target feature dimension; if business 2 has better accuracy requirements and is not sensitive to latency, it is suitable to use a larger feature dimension as the target feature dimension.

[0067] In some embodiments, the multimodal retrieval model includes a feature processing layer and a multi-head output layer. The multi-head output layer includes multiple output heads for outputting feature vectors of different feature dimensions, and each output head shares the parameters of the feature processing layer. The step of generating a query feature vector of the target feature dimension based on the query data through the multimodal retrieval model includes: extracting features from the query data through the feature processing layer of the multimodal retrieval model to generate a multimodal fusion feature containing contextual semantic information; selecting a target output head from the multi-head output layer of the multimodal retrieval model that matches the business scenario requirements according to the business scenario requirements; and processing the multimodal fusion feature based on the target output head to output the query feature vector of the target feature dimension.

[0068] This multimodal retrieval model comprises a feature processing layer and a multi-head output layer. The multi-head output layer includes multiple output heads for outputting feature vectors of different feature dimensions, and each output head shares the parameters of the feature processing layer. This design allows a single model to replace multiple independent models, significantly reducing model training and maintenance costs while improving system deployment flexibility. The structure of this multi-head output layer enables the simultaneous support of multiple feature dimension outputs by training and maintaining only one multimodal retrieval model, significantly reducing model training and maintenance costs and enhancing system flexibility. In practical applications, appropriate output heads can be dynamically selected to obtain feature vectors of corresponding feature dimensions based on specific business scenario requirements (such as requirements for retrieval speed or accuracy).

[0069] Specifically, the feature processing layer of the multimodal retrieval model extracts features from the query data to generate multimodal fusion features containing contextual semantic information. This feature processing layer can process input data of multiple modalities such as text, images, videos, and audio, and generates multimodal fusion features with unified semantic representation through modal alignment and feature fusion techniques.

[0070] Specifically, based on the needs of the business scenario, including specific requirements for retrieval latency, accuracy, or computing resources, a target output head that matches the needs of the business scenario is selected from the multi-head output layer of the multimodal retrieval model; for example, a low-dimensional output head is selected for latency-sensitive business scenarios, and a high-dimensional output head is selected for business scenarios with high accuracy requirements.

[0071] For example, business scenario requirements are related to retrieval latency or retrieval accuracy; specifically, in response to the business scenario requirement of low latency, the output head that outputs a lower-dimensional feature vector is selected as the target output head; in response to the business scenario requirement of high accuracy, the output head that outputs a higher-dimensional feature vector is selected as the target output head.

[0072] Specifically, the multimodal fusion features are processed based on the target output head to output the query feature vector of the target feature dimension; each output head converts the shared multimodal fusion features into a feature vector of a specific dimension through an independent mapping function, which not only ensures the consistency between features of different dimensions, but also meets the needs of diverse business scenarios.

[0073] The multi-head output layer is optimized using knowledge distillation technology. A high-performance teacher model guides the training of each output head, ensuring high retrieval accuracy even for low-dimensional output heads, thus maintaining efficiency without sacrificing performance. For example, to guarantee the retrieval performance of output heads with different feature dimensions, this embodiment employs a multi-head knowledge distillation strategy. First, a high-performance teacher model with high-dimensional feature output is trained. Then, the knowledge from the teacher model is distilled into student model output heads of different dimensions. Unlike traditional knowledge distillation methods, the distillation process in this embodiment does not require feature dimension alignment; the high-dimensional features of the teacher model can simultaneously guide the learning of multiple student model output heads with different dimensions. This distillation strategy effectively transfers the superior performance of the teacher model to each student model output head, achieving retrieval results comparable to high-dimensional models even for lower-dimensional output heads. This effectively solves the problem of insufficient expressive power of low-dimensional features, preserving retrieval accuracy to the greatest extent while ensuring retrieval efficiency.

[0074] like Figure 4As shown, this embodiment employs a multimodal retrieval model based on a Multimodal Large Language Model (MLLM), and has been specifically optimized to efficiently extract multimodal feature embeddings (embedded elements) of different feature dimensions, i.e., feature vectors of different feature dimensions. The model input can be information such as audio, video, image, and text. The multimodal retrieval model can include a feature processing layer and a multi-head output layer. The feature processing layer can use a Multimodal Large Language Model (MLLM), and its structure can include a visual encoder (CV Encoder), a visual projection layer (CV Projector), an audio encoder (Audio Encoder), an audio projection layer (Audio Projector), a tokenizer, and a Large Language Model (LLM), etc. A multi-head output layer can include multiple output heads for outputting feature vectors of different feature dimensions. The last feed-forward neural network (FFN) layer of the Multimodal Large Language Model (MLLM) is replaced with a multi-head FFN, serving as multiple output heads (e.g., FFN1, FFN2, FFN3), which can be mapped to outputs of different dimensions. The output of each output head (FFN head) represents the feature vector (embedding) of that feature dimension. In terms of structural improvement, the LLM model, acting as an encoder (decoder), changes the original unidirectional attention mechanism to a bidirectional attention mechanism to fully leverage its feature extraction capabilities. To further enhance the multi-head feature representation capability, a better-performing teacher model can be obtained first. The teacher model can be a model with a larger network structure or a multi-stage model such as a recall + ranking model. Then, the output of the teacher model is distilled onto the multi-head features of the student model. Here, the contrastive learning results are distilled, rather than the features. Therefore, the feature dimensions of the teacher model and the student model do not need to be aligned, and the contrastive learning results of the teacher model can be distilled onto any head output of the student model.

[0075] A visual encoder (CV Encoder) is a component used to extract visual features from image or video data. In multimodal retrieval models, it is responsible for converting input image or video data into a machine-processable visual feature representation, typically implemented through deep learning networks (such as ViT).

[0076] Projector: A component that projects data from different modalities (such as visual and audio) into a common semantic space. In multimodal retrieval models, there are visual projection layers (CV Projector) and audio projection layers (Audio Projector), which align visual features and audio features to the text modality or other unified semantic space, respectively, for subsequent multimodal fusion and processing.

[0077] An audio encoder is a component used to extract audio features from audio data. In a multimodal model, it is responsible for converting the input audio data into a machine-processable audio feature representation, typically implemented using deep learning networks (such as Transformer-based architectures).

[0078] Audio Projector: This is a component specifically designed to project audio features into a unified semantic space. It works in conjunction with the audio encoder to ensure that audio features can be compared and fused with features from other modalities (such as visual and textual features) within the same semantic space.

[0079] A tokenizer is a component used to segment text data into individual tokens. In multimodal retrieval models, the tokenizer is responsible for processing the input text data, converting it into a sequence of tokens that the model can process for subsequent natural language processing tasks.

[0080] A feed-forward neural network (FFN) is a basic neural network structure consisting of a series of layers. Each layer receives the output of the previous layer as input and produces its own output for the next layer. In multimodal retrieval models, FFN layers are used in multi-head output layers, with each FFN head responsible for generating feature vectors of a specific feature dimension. The MLLM model structure in this embodiment replaces the last FFN layer with a multi-head FFN to achieve feature outputs of different dimensions.

[0081] In some embodiments, the feature processing layer includes a visual encoder, a visual projection layer, an audio encoder, an audio projection layer, a word segmenter, and a large language model; the step of extracting features from the query data through the feature processing layer of the multimodal retrieval model to generate multimodal fusion features containing contextual semantic information includes: encoding and modal alignment processing of at least one modality data contained in the query data to generate modal embedding features in a unified semantic space, wherein the modal embedding features include at least one of text modal embedding features, image modal embedding features, and audio modal embedding features; concatenating the modal embedding features in the unified semantic space to form a concatenated feature sequence; and inputting the concatenated feature sequence into the large language model for encoding processing to generate multimodal fusion features containing contextual semantic information.

[0082] In some embodiments, encoding and modality alignment of at least one modality data contained in the query data to generate modality embedding features in a unified semantic space includes: when the query data contains text modality data, segmenting the text modality data using the word segmenter to obtain text modality embedding features; when the query data contains image modality data, extracting visual features of the image modality data using the visual encoder, and mapping the visual features to the same dimensional space as the text modality embedding features using the visual projection layer to obtain image modality embedding features; when the query data contains audio modality data, extracting audio features of the audio modality data using the audio encoder, and mapping the audio features to the same dimensional space as the text modality embedding features using the audio projection layer to obtain audio modality embedding features.

[0083] Among them, such as Figure 4 As shown, the Visual Encoder (CV Encoder) extracts visual features from image or video data; the Visual Projection Layer (CV Projector) aligns the visual modality to the text modality. The Audio Encoder, based on the Transformer architecture, extracts audio features from audio data; the Audio Projection Layer aligns the audio features to the text modality. The Tokenizer segments text data into meaningful tokens, preparing for subsequent text embedding. The Large Language Model (LLM) receives and processes embedding features from different modalities, such as text modality embeddings, image modality embeddings, and audio modality embeddings, and then outputs multimodal fusion features (multimodal embeddings) containing contextual semantic information.

[0084] like Figure 5As shown, when the query data contains image modality data, this image modality data can be an image or a video. If it is a video, image frames can be extracted from the video first, and then further processing can be performed. For example, the input image is first segmented into multiple image patches, and then fed into a visual encoder (CV Encoder) for processing to extract the feature embeddings (Patch Token Embeddings) of each image patch as the visual features of the image modality data. These image patch feature embeddings are then mapped to the same dimensional space as the text modality embedding features through a visual projection layer (CV Projector) to obtain the image modality embedding features (CV Embedding).

[0085] like Figure 6 As shown, when the query data contains text modal data, the input text is first tokenized by a tokenizer, which converts the text into a series of token embeddings, i.e., text modal embedding features.

[0086] as follows Figure 7 As shown, when the query data contains audio modal data, the input audio is first sampled and sent to an audio encoder based on the Transformer structure to extract audio features. Then, the audio features are mapped to the same dimensional space as the text modal embedding features through the audio projection layer to obtain audio embedding features.

[0087] like Figure 8 As shown, modal embedding features within a unified semantic space are concatenated to form a concatenated feature sequence. For example, text modal embedding features, image modal embedding features, and audio modal embedding features are concatenated together to form a concatenated feature sequence, which serves as the input to the Large Language Model (LLM). The LLM model leverages its powerful language understanding capabilities to deeply encode the concatenated feature sequence. Through internal bidirectional attention mechanisms and other structures, it fully captures the contextual relationships between elements in the concatenated feature sequence, ultimately outputting multimodal fusion features containing rich semantic information.

[0088] In some embodiments, inputting the concatenated feature sequence into the large language model for encoding processing to generate multimodal fusion features containing contextual semantic information includes: inputting the concatenated feature sequence into the large language model, and encoding the concatenated feature sequence through a bidirectional attention mechanism in the large language model to output multimodal fusion features containing contextual semantic information; wherein the attention mask of the bidirectional attention mechanism is configured to allow a lexical at any position in the concatenated feature sequence to pay attention to lexicals at all other positions in the concatenated feature sequence.

[0089] like Figure 9 As shown, to better adapt to feature extraction tasks, the attention mechanism of the Large Language Model (LLM) was modified: the original unidirectional attention mechanism was replaced with a bidirectional attention mechanism. Specifically, the attention mask was changed from a lower triangle form (allowing attention only to previous words) to an all-1 form (allowing attention to all words), so that each word can utilize all information in the context.

[0090] In some embodiments, processing the multimodal fusion features based on the target output head to output a query feature vector of the target feature dimension includes: inputting the multimodal fusion features into the target output head, and performing independent linear transformation and nonlinear activation processing on each word in the multimodal fusion features through the target output head to generate an output vector corresponding to each word; performing mean pooling on the output vectors of all words output by the target output head to aggregate and generate a query feature vector of the target feature dimension.

[0091] like Figure 9As shown, to achieve dynamic feature dimension output and improve the training efficiency and robustness of the model, this embodiment improves the output layer of LLM by adopting a novel multi-head FFN structure to replace the traditional single FFN layer, forming a multi-head output layer. Each output head in the multi-head output layer, i.e., each FFN head, receives the output of the last Transformer layer of the LLM as input, undergoes independent linear transformations and non-linear activation functions, and finally outputs a feature vector of a specific feature dimension. To obtain the final multimodal feature embedding (the query feature vector of the target feature dimension), mean pooling is performed on the output of each FFN head, aggregating the feature vectors of all words into a fixed-dimensional feature vector to obtain the query feature vector of the target feature dimension. Furthermore, residual connections and layer normalization (Add & Norm) can be introduced to stabilize the training process, promote gradient flow, and improve the model's expressive power. Residual connections are used to directly add the input of the previous layer to the output of the current layer, allowing the gradient to "skip" the current layer during backpropagation. This effectively alleviates the vanishing gradient problem in deep networks, enabling deeper models to be built. Layer normalization normalizes the feature dimensions of each sample, adjusting them to a distribution with a mean of 0 and a variance of 1. This stabilizes the input distribution of each layer, reduces internal covariate bias, and thus accelerates model training convergence.

[0092] Compared with the traditional single FFN layer, the multi-head FFN structure has the following advantages: Dynamic feature dimension output: Each FFN head is responsible for mapping the LLM output to a feature space of a specific dimension. This means that by training only one model, multiple feature embeddings of different dimensions can be output simultaneously, thereby meeting the needs of different downstream tasks and greatly improving the flexibility of the model.

[0093] Parameter sharing for efficient training: In a multi-head FFN architecture, all FFN heads share the model parameters for the first N-1 layers, with only the parameters of the last layer being independent. This parameter sharing mechanism not only reduces the number of model parameters and lowers training costs, but also allows each FFN head to provide its own supervision information during training, jointly optimizing model parameters and thus improving training efficiency.

[0094] Enhancing model robustness: Each FFN head generates its own gradient information during training and backpropagates it to the shared model layers. This multi-gradient joint optimization approach enables the model to learn more general and robust feature representations, enhancing its generalization ability and adaptability to different tasks.

[0095] Through this multi-head FFN structure, the model can efficiently generate multimodal feature embeddings of different dimensions, and improve the training efficiency and robustness of the model through parameter sharing and multi-gradient optimization mechanisms, enabling it to better adapt to various downstream retrieval tasks.

[0096] Step 130: Calculate the similarity between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset to obtain the retrieval results.

[0097] The candidate dataset is a pool of candidates containing multimodal candidate data, including text, images (or videos), audio, combinations of text and images (or videos), combinations of text and audio, combinations of images (or videos) and audio, and combinations of text and images (or videos) and audio. During the retrieval phase, feature vectors are extracted from both the query data and all candidate data in the candidate pool. A similarity score is obtained by calculating the similarity between the query feature vector and each candidate feature vector, and the retrieval result is determined based on the similarity score.

[0098] like Figure 3 As shown, firstly, it receives query data (query1, query2) from different services (such as service 1, service 2, etc.). The query content can be multiple modal data such as text, image (or video), audio, or any combination thereof.

[0099] Subsequently, the query input is fed into a multimodal retrieval model for processing. This multimodal retrieval model can output query feature vectors of different dimensions (such as query feature vectors of feature dimension n and query feature vectors of feature dimension m) based on the needs of various business scenarios (e.g., business 1 focuses on low latency, business 2 focuses on high accuracy). Then, the generated query feature vectors are compared with the candidate feature vectors in the candidate pools (candidate pool 1 and candidate pool 2) corresponding to different feature dimensions to calculate similarity, ultimately obtaining retrieval results (retrieval result 1 and retrieval result 2) that match the needs of each business scenario.

[0100] In some embodiments, the step of calculating the similarity between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset to obtain the retrieval result includes: calculating the similarity between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset to obtain a similarity score; sorting each candidate data in the candidate dataset according to the similarity score, and selecting the candidate data with the highest similarity score as the retrieval result.

[0101] The retrieval mechanism calculates the similarity between the query feature vector of the target feature dimension and the corresponding candidate feature vectors of each candidate data in the candidate dataset. The similarity calculation can employ at least one metric, such as cosine similarity, inner product, or Euclidean distance, to obtain a similarity score. Then, the candidate data in the candidate dataset are sorted according to their similarity scores, forming a candidate list from highest to lowest similarity. The candidate data with the highest similarity score is selected as the retrieval result; alternatively, depending on the actual business requirements, the top K candidate data with the highest similarity scores are returned as the final retrieval result, where K is a positive integer greater than or equal to 1. This similarity-based retrieval mechanism effectively identifies the most relevant information to the query data from a large-scale candidate dataset, meeting the retrieval needs of different business scenarios.

[0102] In some embodiments, the training steps of the multimodal retrieval model include: obtaining an initial student model, the initial student model having the feature processing layer and the multi-head output layer; pre-training the initial student model through contrastive learning to obtain a pre-trained student model; training a teacher model, and fine-tuning the pre-trained student model based on the teacher model through multi-head knowledge distillation to obtain the multimodal retrieval model.

[0103] First, an initial student model is obtained. This initial student model has a feature processing layer and a multi-head output layer, where the multi-head output layer contains feature output heads of multiple dimensions. The initial student model refers to an initial retrieval model that has been built but not yet fully trained before knowledge distillation, and possesses multi-head output capabilities.

[0104] Then, the initial student model is pre-trained through contrastive learning. For example, the information noise contrastive estimation loss function is used to narrow the feature distance between positive sample pairs and widen the feature distance between negative sample pairs, resulting in a pre-trained student model with basic multimodal feature extraction capabilities. Contrastive learning is a self-supervised learning method that learns feature representations of data by having the model learn to distinguish between similar and dissimilar data sample pairs.

[0105] Then, a teacher model is trained first, and the pre-trained student model is fine-tuned based on the teacher model through multi-head knowledge distillation. The high-performance pre-trained teacher model simultaneously guides the learning of each output head of the student model. The knowledge distillation process is based on contrastive learning results rather than direct feature alignment, therefore, the feature dimensions of the teacher and student model output heads do not need to be the same. This results in a multimodal retrieval model, ensuring that even low-dimensional output heads maintain high retrieval accuracy close to that of the teacher model. Through fine-tuning via multi-head knowledge distillation, the various output heads of the student model can better adapt to the feature dimension requirements of different business scenarios, significantly improving the overall performance and practicality of the multimodal retrieval model.

[0106] In some embodiments, the feature processing layer includes a visual encoder, a visual projection layer, an audio encoder, an audio projection layer, a word segmenter, and a large language model, and the multi-head output layer includes multiple output heads; the step of pre-training the initial student model through contrastive learning to obtain a pre-trained student model includes: acquiring first training data, the first training data containing multiple first training sample pairs, each first training sample pair including a first query sample and at least one first candidate sample, the modality of the first query sample and the first candidate sample being at least one of text modality, image modality, and audio modality; extracting query feature vectors of the first query sample and candidate feature vectors of the first candidate sample from the initial student model; calculating contrastive learning loss based on the similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate sample; updating the parameters of the initial student model based on the contrastive learning loss to obtain the pre-trained student model; wherein, during the training process, the parameters of the visual encoder and the audio encoder are frozen, and the parameters of the visual projection layer, the audio projection layer, and the large language model are updated.

[0107] For example, regarding the first training data, a variety of first training sample pairs with different modalities can be collected. Each first training sample pair includes a first query sample and at least one first candidate sample. The modality of the first query sample and the first candidate sample is at least one of text modality, image modality, and audio modality. The first training sample pairs cover a variety of modal combinations, including but not limited to: text→text; image→image; text→image; image→text; text→audio; audio→audio; audio→text; image→audio; audio→image; text→text+image; image→text+image; text+image→text; text+image→image; text+image→text+image; text→text+audio; image→text+audio; text+audio→text; text+audio→image; text+audio→text+audio; image→image+audio; image→image+audio; image+audio→image. Image + Audio → Text; Image + Audio → Image + Audio; Text → Text + Image + Audio; Audio → Text + Image + Audio; Image → Text + Image + Audio; Text + Image → Text + Image + Audio; Text + Audio → Text + Image + Audio; Image + Audio → Text + Image + Audio; Text + Image + Audio → Text; Text + Image + Audio → Audio; Text + Image + Audio → Image; Text + Image + Audio → Text + Image; Text + Image + Audio → Image + Audio; Text + Image + Audio → Text + Image + Audio.

[0108] For example, text → text: If the query text sample is "Who was the first emperor to visit the city located at the source of the Adriatic Sea and with an ancient cathedral?", the candidate text samples must contain related information such as "Augustus was the first emperor to visit Aquileia"; Text + Image → Image: The query sample consists of the query text sample "a white, furry animal sitting next to a food bowl" and the corresponding query image sample. The candidate image sample must match this descriptive feature.

[0109] During training, the parameters of the visual encoder (CV Encoder) and audio encoder are frozen. Only the visual projection layer (CV Projector), audio projection layer (Audio Projector), and large language model (LLM) participate in training. Only the parameters of the visual projection layer (CV Projector), audio projection layer (Audio Projector), and large language model (LLM) are updated. This is to maintain the stability of basic feature extraction capabilities while focusing on optimizing modality alignment and semantic understanding capabilities.

[0110] In some embodiments, the initial student model includes a feature processing layer and a multi-head output layer. The multi-head output layer includes multiple output heads for outputting feature vectors of different feature dimensions, and each output head shares the parameters of the feature processing layer. Extracting the query feature vector of the first query sample using the initial student model includes: extracting features from the first query sample using the feature processing layer of the initial student model to generate a multimodal fusion feature corresponding to the first query sample that includes contextual semantic information; selecting a target output head from the multi-head output layer of the initial student model that matches the business scenario requirements based on those requirements; and processing the multimodal fusion feature corresponding to the first query sample based on the target output head to output a query feature vector of the target feature dimension corresponding to the first query sample.

[0111] In some embodiments, the feature processing layer includes a visual encoder, a visual projection layer, an audio encoder, an audio projection layer, a word segmenter, and a large language model; the step of extracting features from the first query sample through the feature processing layer of the initial student model to generate a multimodal fusion feature corresponding to the first query sample containing contextual semantic information includes: encoding and modal alignment processing of at least one modal data contained in the first query sample to generate modal embedding features in a unified semantic space corresponding to the first query sample, wherein the modal embedding features include at least one of text modal embedding features, image modal embedding features, and audio modal embedding features; concatenating the modal embedding features in the unified semantic space to form a concatenated feature sequence corresponding to the first query sample; and inputting the concatenated feature sequence corresponding to the first query sample into the large language model for encoding processing to generate a multimodal fusion feature corresponding to the first query sample containing contextual semantic information.

[0112] In some embodiments, encoding and modality alignment of at least one modality data contained in the first query sample to generate modality embedding features in a unified semantic space corresponding to the first query sample includes: when the first query sample contains text modality data, segmenting the text modality data using the word segmenter to obtain text modality embedding features corresponding to the first query sample; when the first query sample contains image modality data, extracting visual features of the image modality data using the visual encoder, and mapping the visual features to the same dimensional space as the text modality embedding features using the visual projection layer to obtain image modality embedding features corresponding to the first query sample; when the first query sample contains audio modality data, extracting audio features of the audio modality data using the audio encoder, and mapping the audio features to the same dimensional space as the text modality embedding features using the audio projection layer to obtain audio modality embedding features corresponding to the first query sample.

[0113] In some embodiments, the step of inputting the concatenated feature sequence corresponding to the first query sample into the large language model for encoding processing to generate a multimodal fusion feature corresponding to the first query sample containing contextual semantic information includes: inputting the concatenated feature sequence corresponding to the first query sample into the large language model, and encoding the concatenated feature sequence corresponding to the first query sample through a bidirectional attention mechanism in the large language model to output a multimodal fusion feature corresponding to the first query sample containing contextual semantic information; wherein, the attention mask of the bidirectional attention mechanism is configured to allow a lexical at any position in the concatenated feature sequence corresponding to the first query sample to pay attention to lexicals at all other positions in the concatenated feature sequence.

[0114] In some embodiments, processing the multimodal fusion features corresponding to the first query sample based on the target output head to output a query feature vector of the target feature dimension corresponding to the first query sample includes: inputting the multimodal fusion features corresponding to the first query sample into the target output head, and performing independent linear transformation and nonlinear activation processing on each word in the multimodal fusion features corresponding to the first query sample through the target output head to generate an output vector corresponding to each word; performing mean pooling on the output vectors of all words output by the target output head to aggregate and generate a query feature vector of the target feature dimension corresponding to the first query sample.

[0115] In some embodiments, the initial student model includes a feature processing layer and a multi-head output layer. The multi-head output layer includes multiple output heads for outputting feature vectors of different feature dimensions, and each output head shares the parameters of the feature processing layer. Extracting candidate feature vectors of the first candidate sample using the initial student model includes: extracting features from the first candidate sample using the feature processing layer of the initial student model to generate a multimodal fusion feature corresponding to the first candidate sample that includes contextual semantic information; selecting a target output head from the multi-head output layer of the initial student model that matches the business scenario requirements based on the business scenario requirements; and processing the multimodal fusion feature corresponding to the first candidate sample based on the target output head to output a candidate feature vector of the target feature dimension corresponding to the first candidate sample.

[0116] In some embodiments, the feature processing layer includes a visual encoder, a visual projection layer, an audio encoder, an audio projection layer, a word segmenter, and a large language model; the step of extracting features from the first candidate sample through the feature processing layer of the initial student model to generate a multimodal fusion feature corresponding to the first candidate sample containing contextual semantic information includes: encoding and modal alignment processing of at least one modal data contained in the first candidate sample to generate modal embedding features in a unified semantic space corresponding to the first candidate sample, wherein the modal embedding features include at least one of text modal embedding features, image modal embedding features, and audio modal embedding features; concatenating the modal embedding features in the unified semantic space to form a concatenated feature sequence corresponding to the first candidate sample; and inputting the concatenated feature sequence corresponding to the first candidate sample into the large language model for encoding processing to generate a multimodal fusion feature corresponding to the first candidate sample containing contextual semantic information.

[0117] In some embodiments, encoding and modality alignment of at least one modality data contained in the first candidate sample to generate modality embedding features in a unified semantic space corresponding to the first candidate sample includes: when the first candidate sample contains text modality data, segmenting the text modality data using the word segmenter to obtain text modality embedding features corresponding to the first candidate sample; when the first candidate sample contains image modality data, extracting visual features of the image modality data using the visual encoder, and mapping the visual features to the same dimensional space as the text modality embedding features using the visual projection layer to obtain image modality embedding features corresponding to the first candidate sample; when the first candidate sample contains audio modality data, extracting audio features of the audio modality data using the audio encoder, and mapping the audio features to the same dimensional space as the text modality embedding features using the audio projection layer to obtain audio modality embedding features corresponding to the first candidate sample.

[0118] In some embodiments, the step of inputting the concatenated feature sequence corresponding to the first candidate sample into the large language model for encoding processing to generate a multimodal fusion feature containing contextual semantic information corresponding to the first candidate sample includes: inputting the concatenated feature sequence corresponding to the first candidate sample into the large language model, and encoding the concatenated feature sequence corresponding to the first candidate sample through a bidirectional attention mechanism in the large language model to output a multimodal fusion feature containing contextual semantic information corresponding to the first candidate sample; wherein, the attention mask of the bidirectional attention mechanism is configured to allow a lexical at any position in the concatenated feature sequence corresponding to the first candidate sample to pay attention to lexicals at all other positions in the concatenated feature sequence.

[0119] In some embodiments, processing the multimodal fusion features corresponding to the first candidate sample based on the target output head to output a candidate feature vector of the target feature dimension corresponding to the first candidate sample includes: inputting the multimodal fusion features corresponding to the first candidate sample into the target output head, and performing independent linear transformation and nonlinear activation processing on each word in the multimodal fusion features corresponding to the first candidate sample through the target output head to generate an output vector corresponding to each word; performing mean pooling on the output vectors of all words output by the target output head to aggregate and generate a candidate feature vector of the target feature dimension corresponding to the first candidate sample.

[0120] In some embodiments, the first candidate sample includes a first candidate positive sample and a first candidate negative sample; the step of calculating the contrastive learning loss based on the similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate sample includes: obtaining a first similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate positive sample; obtaining a second similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate negative sample; and calculating the contrastive learning loss based on the first similarity and the second similarity.

[0121] The first candidate sample includes a first candidate positive sample and multiple first candidate negative samples. The first candidate positive sample is a candidate sample that has a semantic relationship with the first query sample, and the first candidate negative sample is a candidate sample that does not have a semantic relationship with the first query sample.

[0122] Among them, contrastive learning is used as the training objective. Based on the similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate sample, the contrastive learning loss is calculated. The loss function can be the information noise contrastive estimation loss (Info-NCE Loss, Info-NCE), which is expressed as the following formula (1): (1); Among them, e q This represents the query feature vector (Embedding) of the first query sample; e c e represents the candidate feature vector (Embedding) of the first candidate sample. c + e represents the candidate feature vector of the first candidate positive sample. c - This represents the candidate feature vector of the first candidate negative sample. This represents the candidate feature vector of the i-th first candidate sample in the current batch. It can be e c + or e c - N represents the total number of first candidate samples in the current batch; τ represents the temperature coefficient, used to adjust the sharpness of the similarity distribution; sim represents the cosine similarity calculation function. This represents the first similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate positive sample. L represents the similarity (first similarity or second similarity) between the query feature vector of the first query sample and the candidate feature vector of the i-th first candidate sample (first candidate positive sample or first candidate negative sample); InfoNCEThis represents the final calculated contrastive learning loss value.

[0123] This contrastive learning loss function optimizes the model's feature extraction capability by narrowing the feature distance between the first query sample and the first candidate positive sample, while simultaneously widening the feature distance between the first query sample and all first candidate negative samples.

[0124] In some embodiments, training the teacher model includes: constructing a cascaded teacher model, the cascaded teacher model including a recall model and a ranking model; acquiring second training data, the second training data including a second query sample, and a second candidate positive sample and a second candidate negative sample corresponding to the second query sample, the second candidate positive sample and the second candidate negative sample being selected from the top K candidate data samples in the recall results returned by the recall model; and encoding the second query sample, the second candidate positive sample, and preset prompt words using the ranking model to be trained to obtain a first encoding feature, and encoding the second query sample, the second candidate negative sample, and preset prompt words. The word is encoded to obtain a second encoded feature. The prompt word is used to guide the ranking model to be trained to determine whether the second query sample matches the second candidate positive sample or the second candidate negative sample. The ranking model to be trained decodes the first encoded feature to obtain a first ranking score sample and decodes the second encoded feature to obtain a second ranking score sample. Based on the first ranking score sample, the labeled score corresponding to the second candidate positive sample, the second ranking score sample, and the labeled score corresponding to the second candidate negative sample, a ranking loss is determined. Based on the ranking loss, the parameters of the ranking model to be trained are updated to obtain a pre-trained ranking model.

[0125] Specifically, during different training phases, the number of the top K candidate data samples in the recall results returned by the recall model is dynamically adjusted, and the value of K is gradually reduced as training progresses.

[0126] To further enhance the expressive power of the multimodal retrieval model, a more powerful teacher model can be trained, and then its capabilities can be distilled onto the retrieval model. The teacher model can be a model with a larger network structure or a cascaded multi-stage model, such as a recall model combined with a ranking model. For models using a larger network structure, the entire process is completely consistent with the initial retrieval model, and will not be elaborated upon here. The following section focuses on using a cascaded multi-stage model to obtain a more effective teacher model.

[0127] Among them, the cascaded teacher model includes a recall model and a ranking model. First, the recall model is used to obtain a rough recall result, and then the rough recall is sent into the ranking model for further fine ranking to obtain the final result. Here, the recall model uses the previous initial retrieval model (i.e., the initial student model), and then the result of the initial retrieval model is sent into the ranking model to obtain a better result than the original retrieval model. The ranking model can directly select a general multimodal large language model (Multimodal Large Language Model, MLLM), without training, and use its general capabilities for ranking, or it can be fine-tuned on this basis. The following mainly introduces the use of the ranking model and the fine-tuning method.

[0128] Among them, in order to accurately screen the retrieval results, the embodiment of this application adopts a ranking model based on the MLLM model to perform fine ranking on the top-k candidate results (i.e., the top K candidate data samples) returned by the recall model. This ranking model uses an autoregressive training method and gradually increasing negative sample difficulty, effectively enhancing the discriminative ability of the model.

[0129] (1) Model input: The input of the ranking model to be trained is the concatenation of the second query sample (query) and the second candidate sample (candidate), and a specific prompt is added to guide the ranking model to be trained to judge whether the two match. Among them, the second candidate sample can include a second candidate positive sample and a second candidate negative sample. For example, the input format is: query: <query information> candidate: <candidate information>; the prompt is "Please indicate whether the above candidate sample (candidate) matches the query sample (query), and only output yes or no".

[0130] (2) Model structure and training: The ranking model to be trained can adopt the structure of a native multimodal large language model (Multimodal Large Language Model, MLLM), and use the Low-Rank Adaptation (LoRA) method for fine-tuning. The visual encoder (CV Encoder), audio encoder (Audio Encoder), visual projection layer (CV Projector), and audio projection layer (Audio Projector) are frozen, and only the large language model (Large Language Model, LLM) is trained, and only the parameters of the large language model (Large Language Model, LLM) are updated.

[0131] (3) Model output and training objectives: The model is trained using the Next-Token Prediction method to predict whether the next token is "yes" or "no", thereby determining whether the second query sample and the second candidate sample match.

[0132] (4) Negative Sample Mining and Training Strategy: To improve the model's discriminative ability, a strategy of gradually increasing the difficulty of negative samples was adopted during training. For example, for each second query sample, one second candidate positive sample and one second candidate negative sample were selected, with a ratio of 1:1. The second candidate negative sample was randomly selected from the top-k candidate results (the top K candidate data samples) returned by the recall model. In the early stages of training, the value of k was relatively large (e.g., k=100), making the second candidate negative sample relatively easy to distinguish. As training progressed, the value of k was gradually decreased (e.g., k=20), and the difficulty of the second candidate negative sample gradually increased, forcing the model to learn finer-grained feature differences, thereby enhancing its discriminative ability. This progressively difficult negative sample mining strategy effectively improved the performance of the ranking model.

[0133] Using the above strategies, the pre-trained ranking model obtained after training can effectively distinguish between matching and non-matching samples, thereby refining the recall results and ultimately improving the overall performance of the retrieval model.

[0134] When this teacher model is applied to the inference stage, the model sequentially combines the query data with each candidate result (the top K candidate data) and feeds them into the pre-trained ranking model. The pre-trained ranking model outputs the probability value corresponding to "yes" or "no" as a score, thus obtaining the ranking score for each candidate result.

[0135] In some embodiments, the step of fine-tuning the pre-trained student model based on the teacher model through multi-head knowledge distillation to obtain the multimodal retrieval model includes: obtaining the teacher model, which is used to supervise the pre-trained student model during training to jointly optimize each output head in the multi-head output layer of the pre-trained student model so that each output head can output a feature vector corresponding to the feature dimension; obtaining third training data, which includes a third query sample and a third candidate sample corresponding to the third query sample, the third candidate sample including a third candidate positive sample and multiple third candidate negative samples; processing the third training data through the teacher model to generate a first matching score distribution between the third query sample and each of the third candidate samples; processing the third training data through the pre-trained student model to generate a second matching score distribution calculated based on the feature vector output by each output head for each output head in the multi-head output layer; calculating the difference between the second matching score distribution and the first matching score distribution for each output head as the distillation loss for each output head; and jointly updating the parameters of the pre-trained student model based on the distillation losses of all output heads to obtain the multimodal retrieval model.

[0136] The teacher model can be any of the following: a multimodal large model with a parameter size larger than the pre-trained student model; or a cascaded system consisting of a recall model and a ranking model. The output of the teacher model does not need to be aligned with the output of any output head in terms of feature dimensions. The difference between the second matching score distribution and the first matching score distribution is calculated using KL divergence.

[0137] like Figure 10 As shown, to further improve the performance of the multimodal retrieval model, the capabilities of the better-performing teacher model are distilled into the pre-trained retrieval model (pre-trained student model). A multi-head knowledge distillation method is employed, which can simultaneously distill the multi-dimensional feature heads of the retrieval model. Benefiting from the use of a contrastive learning-based distillation strategy, rather than directly distilling features, the feature dimensions of the teacher and student models do not need to be consistent. This ensures that the teacher model can output any dimension and can be simultaneously distilled into the multi-dimensional feature heads of the student model.

[0138] (1) Data preparation: First, the ranking scores of the third query sample and each third candidate sample were obtained, and then the scores were sorted from high to low (excluding positive samples) as the difficult negative samples of the model.

[0139] (2) Model training phase: The first stage of the retrieval model is a pre-trained student model. Further training is then conducted based on this model.

[0140] For each third query sample, a training batch is constructed containing third candidate positive samples and multiple third candidate negative samples, with the multiple third candidate negative samples being the hard negative samples. For example, for each third query sample, the corresponding third candidate positive sample and the top-50 third candidate negative samples are taken. Unlike the recall model, which uses random negative samples during training, hard negative samples are taken during the multi-knowledge processing to improve the distillation model's ability to distinguish hard samples.

[0141] Each third query sample, third candidate positive sample, and top-50 third candidate negative sample correspond to their respective ranking scores. The scores are normalized using softmax to obtain the result distribution of the teacher model, which generates the first matching score distribution between the third query sample and each third candidate sample. The goal is to distill the first matching score distribution into the pre-trained student model so that the pre-trained student model can mimic the teacher's first matching score distribution.

[0142] The pre-trained student model has the same structure as the initial student model (initial retrieval model). It also uses contrastive learning to obtain the cosine similarity score between the third query sample and the third candidate sample, and performs softmax normalization to obtain the result distribution of the pre-trained student model. This generates the second matching score distribution calculated based on the feature vectors output by each output head. Each output head in the multi-head output layer outputs its own result distribution, such as [S+, S-1, ..., S-50], where S+ represents the score between the third query sample and the third candidate positive sample, and S-1, ..., S-50 represent the scores between the third query sample and 50 third candidate negative samples. Since the feature dimension is independent, the output heads with multiple feature dimensions in the pre-trained student model can be directly distilled with the teacher model.

[0143] The KL divergence loss function can be used to minimize the difference between the second matching score distribution and the first matching score distribution. As shown in the figure above, the pre-trained student model outputs feature vectors from multiple output heads. The score corresponding to the feature vector output by each output head is subjected to KL divergence with the result distribution of the teacher model, and multiple output heads are distilled simultaneously.

[0144] The distillation loss function can be expressed as the following formula (2): (2); Among them, S t S represents the distribution of the first matching scores. t (i) represents the score distribution of the teacher model for the i-th third query sample; S s S represents the distribution of the second matching score. s (i) represents the score distribution of the pre-trained student model for the i-th third query sample; i represents the number of samples, such as k+1; n represents the number of output headers (FFN headers); N represents the total number of third query samples; log represents the natural logarithm function.

[0145] This application proposes an improved scheme for multimodal retrieval models. Its innovations mainly lie in dynamic feature dimension output and multi-head knowledge distillation, which bring the following significant advantages: Dynamic feature dimension output enhances model flexibility and applicability: By introducing a multi-head structure, a single model can output feature vectors with multiple different dimensions. This allows multimodal retrieval models to flexibly adapt to the needs of various downstream application scenarios, eliminating the need to train and deploy separate models for each scenario, greatly improving the practicality and deployment efficiency of multimodal retrieval models. Only one model needs to be deployed to support multiple requirements, balancing retrieval speed and performance metrics.

[0146] Multi-head knowledge distillation enhances model representation capabilities: A multi-head knowledge distillation method is employed to transfer knowledge from the Teacher model to the various output heads (FFN heads) of the Student model. This knowledge transfer effectively improves the feature representation capabilities of each FFN head, making the generated feature vectors more discriminative and expressive, thereby improving retrieval accuracy and recall.

[0147] Simplify model deployment and reduce maintenance costs: Since only one model needs to be deployed to meet the requirements of multiple feature dimensions, the model deployment process is simplified, and the subsequent maintenance costs and the complexity of updates and iterations are reduced.

[0148] Efficient resource utilization: Compared to training and maintaining multiple models with different dimensions, the multimodal retrieval model provided in this application only needs to train and maintain one model, thereby making more efficient use of computing and storage resources.

[0149] The embodiments of this application significantly improve the performance, flexibility, and applicability of multimodal retrieval models without increasing model complexity by using dynamic feature dimension output and multi-head knowledge distillation techniques, while reducing deployment and maintenance costs, thus having significant practical value.

[0150] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0151] This application embodiment acquires query data, which includes at least one modality of data: text, image, and audio. Based on the query data, a multimodal retrieval model is used to generate a query feature vector of the target feature dimension. This multimodal retrieval model is configured to output feature vectors of multiple different feature dimensions. The similarity between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset is calculated to obtain the retrieval results. This application embodiment maintains a multimodal retrieval model, supports the output of feature vectors of different feature dimensions, reduces resource consumption, improves flexibility, and ensures retrieval performance.

[0152] To facilitate better implementation of the data processing method of this application embodiment, this application embodiment also provides a data processing apparatus. Please refer to... Figure 11 , Figure 11 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application. The data processing apparatus 200 may include: The acquisition unit 210 is used to acquire query data, which includes at least one modal data selected from text modality, image modality, and audio modality; Processing unit 220 is used to generate a query feature vector of target feature dimension based on the query data through a multimodal retrieval model, wherein the multimodal retrieval model is configured to output feature vectors of multiple different feature dimensions; The retrieval unit 230 is used to calculate the similarity between the query feature vector of the target feature dimension and the candidate feature vector corresponding to each candidate data in the candidate dataset, so as to obtain the retrieval result.

[0153] In some embodiments, the multimodal retrieval model includes a feature processing layer and a multi-head output layer. The multi-head output layer includes multiple output heads for outputting feature vectors of different feature dimensions, and each output head shares the parameters of the feature processing layer. The processing unit 220 is configured to: extract features from the query data through the feature processing layer of the multimodal retrieval model to generate multimodal fusion features containing contextual semantic information; select a target output head from the multi-head output layer of the multimodal retrieval model that matches the business scenario requirements according to the business scenario requirements; and process the multimodal fusion features based on the target output head to output a query feature vector of the target feature dimension.

[0154] In some embodiments, the feature processing layer includes a visual encoder, a visual projection layer, an audio encoder, an audio projection layer, a word segmenter, and a large language model; The processing unit 220 is configured to extract features from the query data through the feature processing layer of the multimodal retrieval model to generate multimodal fusion features containing contextual semantic information. This includes: encoding and modal alignment of at least one modal data contained in the query data to generate modal embedding features within a unified semantic space, wherein the modal embedding features include at least one of text modal embedding features, image modal embedding features, and audio modal embedding features; concatenating the modal embedding features within the unified semantic space to form a concatenated feature sequence; and inputting the concatenated feature sequence into the large language model for encoding processing to generate multimodal fusion features containing contextual semantic information.

[0155] In some embodiments, the processing unit 220 is configured to encode and align at least one modal data contained in the query data to generate modal embedding features in a unified semantic space, including: when the query data contains text modal data, segmenting the text modal data using the word segmenter to obtain text modal embedding features; when the query data contains image modal data, extracting visual features of the image modal data using the visual encoder, and mapping the visual features to the same dimensional space as the text modal embedding features using the visual projection layer to obtain image modal embedding features; when the query data contains audio modal data, extracting audio features of the audio modal data using the audio encoder, and mapping the audio features to the same dimensional space as the text modal embedding features using the audio projection layer to obtain audio modal embedding features.

[0156] In some embodiments, the processing unit 220 is configured to input the concatenated feature sequence into the large language model for encoding processing to generate multimodal fusion features containing contextual semantic information, including: inputting the concatenated feature sequence into the large language model, and encoding the concatenated feature sequence through a bidirectional attention mechanism in the large language model to output multimodal fusion features containing contextual semantic information; wherein, the attention mask of the bidirectional attention mechanism is configured to allow a lexical at any position in the concatenated feature sequence to pay attention to lexicals at all other positions in the concatenated feature sequence.

[0157] In some embodiments, the processing unit 220 is configured to process the multimodal fusion features based on the target output head to output a query feature vector of the target feature dimension, including: inputting the multimodal fusion features into the target output head, and performing independent linear transformation and nonlinear activation processing on each word in the multimodal fusion features through the target output head to generate an output vector corresponding to each word; and performing mean pooling on the output vectors of all words output by the target output head to aggregate and generate a query feature vector of the target feature dimension.

[0158] In some embodiments, the data processing apparatus 200 may further include a training unit for training the multimodal retrieval model, comprising: obtaining an initial student model having the feature processing layer and the multi-head output layer; pre-training the initial student model through contrastive learning to obtain a pre-trained student model; and fine-tuning the pre-trained student model through multi-head knowledge distillation to obtain the multimodal retrieval model.

[0159] In some embodiments, the feature processing layer includes a visual encoder, a visual projection layer, an audio encoder, an audio projection layer, a word segmenter, and a large language model; the multi-head output layer includes multiple output heads; the training unit is used to pre-train the initial student model through contrastive learning to obtain a pre-trained student model, including: acquiring first training data, the first training data containing multiple first training sample pairs, each first training sample pair including a first query sample and at least one first candidate sample, the modality of the first query sample and the first candidate sample being at least one of text modality, image modality, and audio modality; extracting query feature vectors of the first query sample and candidate feature vectors of the first candidate sample from the initial student model; calculating contrastive learning loss based on the similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate sample; updating the parameters of the initial student model based on the contrastive learning loss to obtain the pre-trained student model; wherein, during the training process, the parameters of the visual encoder and the audio encoder are frozen, and the parameters of the visual projection layer, the audio projection layer, and the large language model are updated.

[0160] In some embodiments, the first candidate sample includes a first candidate positive sample and a first candidate negative sample; the training unit is configured to calculate a contrastive learning loss based on the similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate sample, including: obtaining a first similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate positive sample; obtaining a second similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate negative sample; and calculating the contrastive learning loss based on the first similarity and the second similarity.

[0161] In some embodiments, the training unit is used to train a teacher model, comprising: constructing a cascaded teacher model, the cascaded teacher model including a recall model and a ranking model; acquiring second training data, the second training data including a second query sample, and a second candidate positive sample and a second candidate negative sample corresponding to the second query sample, the second candidate positive sample and the second candidate negative sample being selected from the top K candidate data samples in the recall results returned by the recall model; encoding the second query sample, the second candidate positive sample and preset prompt words using the ranking model to be trained to obtain a first encoding feature, and encoding the second query sample, the second candidate negative sample and preset prompt words using the ranking model to be trained to obtain a first encoding feature, and encoding the second query sample, the second candidate negative sample and preset prompt words using the ranking model to be trained to obtain a first encoding feature. The prompt words are encoded to obtain a second encoded feature, which is used to guide the ranking model to be trained to determine whether the second query sample matches the second candidate positive sample or the second candidate negative sample. The ranking model to be trained decodes the first encoded feature to obtain a first ranking score sample and decodes the second encoded feature to obtain a second ranking score sample. Based on the first ranking score sample, the labeled score corresponding to the second candidate positive sample, the second ranking score sample, and the labeled score corresponding to the second candidate negative sample, a ranking loss is determined. Based on the ranking loss, the parameters of the ranking model to be trained are updated to obtain a pre-trained ranking model.

[0162] In some embodiments, the training unit is used to fine-tune the pre-trained student model based on the teacher model through multi-head knowledge distillation to obtain the multimodal retrieval model, comprising: obtaining the teacher model, which is used to supervise the pre-trained student model during training to jointly optimize each output head in the multi-head output layer of the pre-trained student model so that each output head can output a feature vector of the corresponding feature dimension; obtaining third training data, which includes a third query sample and a third candidate sample corresponding to the third query sample, the third candidate sample including a third candidate positive sample and multiple third candidate negative samples; processing the third training data through the teacher model to generate a first matching score distribution between the third query sample and each of the third candidate samples; processing the third training data through the pre-trained student model to generate a second matching score distribution calculated based on the feature vector output by each output head for each output head in the multi-head output layer; calculating the difference between the second matching score distribution and the first matching score distribution for each output head as the distillation loss of each output head; and jointly updating the parameters of the pre-trained student model based on the distillation losses of all output heads to obtain the multimodal retrieval model.

[0163] In some embodiments, the retrieval unit 230 is configured to perform similarity calculation between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset to obtain a retrieval result, including: performing similarity calculation between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset to obtain a similarity score; sorting each candidate data in the candidate dataset according to the similarity score, and selecting the candidate data with the highest similarity score as the retrieval result.

[0164] It should be noted that the functions of each module in the data processing device 200 in this application embodiment can be referred to the specific implementation of any embodiment in the above method embodiments, and will not be repeated here.

[0165] Each unit in the above-described device can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0166] For example, the data processing device 200 may be integrated into a terminal or server that has storage and a processor and thus computing power, or the data processing device 200 may be the terminal or server.

[0167] In some embodiments, this application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0168] Figure 12 A schematic diagram of the structure of the computer device provided in the embodiments of this application, such as... Figure 12 As shown, the computer device 300 may include: a communication interface 301, a memory 302, a processor 303, and a communication bus 304. The communication interface 301, memory 302, and processor 303 communicate with each other via the communication bus 304. The communication interface 301 is used for data communication between the device 300 and external devices. The memory 302 can be used to store software programs and modules, and the processor 303 runs the software programs and modules stored in the memory 302, such as the software programs for the corresponding operations in the aforementioned method embodiments.

[0169] In some embodiments, the processor 303 may invoke software programs and modules stored in the memory 302 to perform the following operations: acquiring query data, the query data including at least one modality data selected from text modality, image modality, and audio modality; generating a query feature vector of the target feature dimension based on the query data through a multimodal retrieval model, wherein the multimodal retrieval model is configured to output feature vectors of multiple different feature dimensions; and calculating the similarity between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset to obtain retrieval results.

[0170] In some embodiments, the computer device 300 may be integrated into a terminal or server that has storage and a processor and thus computing power, or the computer device 300 may be the terminal or server.

[0171] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding processes in the methods described above in the embodiments of this application; for brevity, further details are omitted here.

[0172] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described above in the embodiments of this application. For brevity, these details will not be elaborated further here.

[0173] This application also provides a computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described above in the embodiments of this application. For brevity, these details will not be elaborated further here.

[0174] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0175] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0176] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0177] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0179] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, the functional units in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0182] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0183] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain query data, which includes at least one modality of text, image, and audio. Based on the query data, a query feature vector of the target feature dimension is generated through a multimodal retrieval model, wherein the multimodal retrieval model is configured to output feature vectors of multiple different feature dimensions. The similarity between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset is calculated to obtain the retrieval results.

2. The data processing method as described in claim 1, characterized in that, The multimodal retrieval model includes a feature processing layer and a multi-head output layer. The multi-head output layer includes multiple output heads for outputting feature vectors of different feature dimensions, and each output head shares the parameters of the feature processing layer. The step of generating a query feature vector of the target feature dimension based on the query data through a multimodal retrieval model includes: The feature processing layer of the multimodal retrieval model extracts features from the query data to generate multimodal fusion features containing contextual semantic information. Based on the business scenario requirements, a target output head that matches the business scenario requirements is selected from the multi-head output layer of the multimodal retrieval model; The multimodal fusion features are processed based on the target output head to output a query feature vector of the target feature dimension.

3. The data processing method as described in claim 2, characterized in that, The feature processing layer includes a visual encoder, a visual projection layer, an audio encoder, an audio projection layer, a word segmenter, and a large language model. The step of extracting features from the query data through the feature processing layer of the multimodal retrieval model to generate multimodal fusion features containing contextual semantic information includes: The query data contains at least one modal data, which is encoded and modally aligned to generate modal embedding features in a unified semantic space. The modal embedding features include at least one of text modal embedding features, image modal embedding features, and audio modal embedding features. The modal embedding features within the unified semantic space are concatenated to form a concatenated feature sequence. The concatenated feature sequence is input into the large language model for encoding processing to generate multimodal fusion features containing contextual semantic information.

4. The data processing method as described in claim 3, characterized in that, The step of encoding and modality-aligning at least one modality data contained in the query data to generate modality embedding features in a unified semantic space includes: When the query data contains text modal data, the text modal data is segmented by the word segmenter to obtain text modal embedding features; When the query data contains image modality data, the visual features of the image modality data are extracted by the visual encoder, and the visual features are mapped to the same dimensional space as the text modality embedding features by the visual projection layer to obtain the image modality embedding features; When the query data contains audio modal data, the audio features of the audio modal data are extracted by the audio encoder, and the audio features are mapped to the same dimensional space as the text modal embedding features by the audio projection layer to obtain the audio modal embedding features.

5. The data processing method as described in claim 3, characterized in that, The step of inputting the concatenated feature sequence into the large language model for encoding processing to generate multimodal fusion features containing contextual semantic information includes: The concatenated feature sequence is input into the large language model, and the concatenated feature sequence is encoded through the bidirectional attention mechanism in the large language model to output a multimodal fusion feature containing contextual semantic information. The attention mask of the bidirectional attention mechanism is configured to allow a lexical at any position in the concatenated feature sequence to pay attention to lexicals at all other positions in the concatenated feature sequence.

6. The data processing method as described in claim 3, characterized in that, The process of processing the multimodal fusion features based on the target output header to output a query feature vector of the target feature dimension includes: The multimodal fusion features are input into the target output head, and each word in the multimodal fusion features is subjected to independent linear transformation and nonlinear activation processing through the target output head to generate an output vector corresponding to each word. The output vectors of all words output by the target output header are subjected to mean pooling to aggregate and generate the query feature vector of the target feature dimension.

7. The data processing method as described in claim 2, characterized in that, The training steps of the multimodal retrieval model include: Obtain an initial student model, wherein the initial student model has the feature processing layer and the multi-head output layer; The initial student model is pre-trained through comparative learning to obtain a pre-trained student model. The teacher model is trained, and the pre-trained student model is fine-tuned based on the teacher model through multi-head knowledge distillation to obtain the multimodal retrieval model.

8. The data processing method as described in claim 7, characterized in that, The feature processing layer includes a visual encoder, a visual projection layer, an audio encoder, an audio projection layer, a word segmenter, and a large language model; the multi-head output layer includes multiple output heads. The step of pre-training the initial student model through contrastive learning to obtain a pre-trained student model includes: Obtain first training data, which includes multiple first training sample pairs. Each first training sample pair includes a first query sample and at least one first candidate sample. The modal of the first query sample and the first candidate sample is at least one of text modal, image modal, and audio modal. Using the initial student model, the query feature vector of the first query sample and the candidate feature vector of the first candidate sample are extracted respectively. Based on the similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate sample, the contrastive learning loss is calculated. Based on the contrastive learning loss, the parameters of the initial student model are updated to obtain the pre-trained student model; During training, the parameters of the visual encoder and the audio encoder are frozen, while the parameters of the visual projection layer, the audio projection layer, and the large language model are updated.

9. The data processing method as described in claim 8, characterized in that, The first candidate sample includes a first candidate positive sample and a first candidate negative sample; the step of calculating the contrastive learning loss based on the similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate sample includes: Obtain the first similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate positive sample; Obtain the second similarity between the query feature vector of the first query sample and the candidate feature vector of the first candidate negative sample; The contrastive learning loss is calculated based on the first similarity and the second similarity.

10. The data processing method as described in claim 7, characterized in that, The training teacher model includes: Construct a cascaded teacher model, which includes a recall model and a ranking model; Obtain second training data, which includes a second query sample, a second candidate positive sample, and a second candidate negative sample corresponding to the second query sample. The second candidate positive sample and the second candidate negative sample are selected from the top K candidate data samples in the recall results returned by the recall model. The ranking model to be trained encodes the second query sample, the second candidate positive sample, and the preset prompt words to obtain the first encoding feature, and encodes the second query sample, the second candidate negative sample, and the preset prompt words to obtain the second encoding feature. The prompt words are used to guide the ranking model to be trained to determine whether the second query sample matches the second candidate positive sample or the second candidate negative sample. The first ranking score sample is obtained by decoding the first encoded feature using the ranking model to be trained, and the second ranking score sample is obtained by decoding the second encoded feature. The ranking loss is determined based on the first ranking score sample, the labeled score corresponding to the second candidate positive sample, the second ranking score sample, and the labeled score corresponding to the second candidate negative sample. Based on the ranking loss, the parameters of the ranking model to be trained are updated to obtain the pre-trained ranking model.

11. The data processing method as described in claim 7, characterized in that, The process of fine-tuning the pre-trained student model based on the teacher model through multi-head knowledge distillation to obtain the multimodal retrieval model includes: The teacher model is obtained and used to supervise the pre-trained student model during training to jointly optimize each output head in the multi-head output layer of the pre-trained student model so that each output head can output a feature vector of the corresponding feature dimension. Obtain third training data, which includes a third query sample and a third candidate sample corresponding to the third query sample. The third candidate sample includes a third candidate positive sample and multiple third candidate negative samples. The teacher model processes the third training data to generate a first matching score distribution between the third query sample and each of the third candidate samples. The third training data is processed by the pre-trained student model, and for each output head in the multi-head output layer, a second matching score distribution is generated based on the feature vector output by each output head. For each output head, the difference between the second matching fraction distribution and the first matching fraction distribution is calculated as the distillation loss for each output head; Based on the distillation loss of all output heads, the parameters of the pre-trained student model are jointly updated to obtain the multimodal retrieval model.

12. The data processing method as described in claim 1, characterized in that, The step of calculating the similarity between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset to obtain the retrieval results includes: The similarity score is obtained by comparing the query feature vector of the target feature dimension with the candidate feature vectors corresponding to each candidate data in the candidate dataset. The candidate data in the candidate dataset are sorted according to the similarity score, and the candidate data with the highest similarity score is selected as the retrieval result.

13. A data processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire query data, which includes at least one modal data selected from text modality, image modality, and audio modality. The processing unit is configured to generate a query feature vector of the target feature dimension based on the query data through a multimodal retrieval model, wherein the multimodal retrieval model is configured to output feature vectors of multiple different feature dimensions. The retrieval unit is used to calculate the similarity between the query feature vector of the target feature dimension and the candidate feature vectors corresponding to each candidate data in the candidate dataset, so as to obtain the retrieval result.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the data processing method as described in any one of claims 1-12.

15. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the data processing method as described in any one of claims 1-12 by calling the computer program stored in the memory.

16. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the data processing method according to any one of claims 1-12.

Images