Data retrieval method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a data retrieval method, electronic equipment and a computer readable storage medium, and relates to the technical fields of computers, natural language processing and information retrieval. The method comprises the following steps: acquiring to-be-retrieved target object data; performing data retrieval on the target object data by adopting the target retrieval model, and outputting a retrieval result; wherein the target retrieval model is generated after the initial retrieval model is subjected to comparative learning training by adopting a multi-modal sample, and in the comparative learning training process, the similarity calculation result of the multi-modal sample is obtained through step-by-step calculation based on the batch size corresponding to the initial retrieval model; the similarity calculation result is used for representing the semantic matching degree of different modal samples in the multi-modal samples. According to the method and the device, the technical problems of limited batch size, high computing resource demand and high computing overhead of a comparative learning method in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the fields of computer technology, natural language processing technology, and information retrieval technology, and in particular to a data retrieval method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Contrastive Learning is a machine learning method that aims to learn feature representation by comparing the similarities and differences between two or more data samples. In recent years, with the continuous development of deep learning technology, contrastive learning has shown great application potential in the fields of image-text matching, image self-representation, sentence representation, and vector retrieval.

[0003] Existing contrastive learning methods usually use contrastive loss to optimize the similarity between samples, and the batch size of contrastive learning will directly affect the effect of model training. Increasing the batch size can provide more negative samples, enhance the recognition ability of the model, and make the training process more stable. However, as the batch size increases, the computational overhead of contrastive loss will also increase. In particular, the computational overhead of the Softmax function commonly used in contrastive learning will increase sharply as the batch size increases, resulting in the application of contrastive learning methods on large-scale data sets being limited.

[0004] Currently, no effective solution has been proposed to address the problems of limited batch size, high computational resource requirements, and high computational overhead of the above contrastive learning methods. Summary of the invention

[0005] The embodiments of the present application provide a data retrieval method, an electronic device, and a computer-readable storage medium to at least solve the technical problems of limited batch size, high computing resource requirements, and high computing overhead in the comparative learning method in the related art.

[0006] According to one aspect of an embodiment of the present application, a data retrieval method is provided, comprising: obtaining target object data to be retrieved; performing data retrieval on the target object data using a target retrieval model, and outputting a retrieval result; wherein the target retrieval model is generated by performing comparative learning training on an initial retrieval model using multimodal samples, and during the comparative learning training process, similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0007] According to another aspect of an embodiment of the present application, a data retrieval method is also provided, including: obtaining product image data to be retrieved; performing data retrieval on the product image data using a target image-text retrieval model, and outputting product image retrieval results; wherein the target image-text retrieval model is generated after comparative learning and training of an initial image-text retrieval model using multimodal samples, and during the comparative learning and training process, similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial image-text retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0008] According to another aspect of an embodiment of the present application, a data retrieval method is also provided, comprising: obtaining a data retrieval request through a first application programming interface, wherein the request data carried in the data retrieval request includes: target object data to be retrieved; returning a data retrieval response through a second application programming interface, wherein the response data carried in the data retrieval response includes: retrieval results, wherein the retrieval results are generated after data retrieval of the target object data using a target retrieval model, wherein the target retrieval model is generated after comparative learning and training of an initial retrieval model using multimodal samples, and during the comparative learning and training process, similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0009] According to another aspect of an embodiment of the present application, a data retrieval method is also provided, including: obtaining a currently input data retrieval dialogue request, wherein the request data carried in the data retrieval dialogue request includes: target object data to be retrieved; in response to the data retrieval dialogue request, returning a data retrieval dialogue reply, wherein the information carried in the data retrieval dialogue reply includes: retrieval results, the retrieval results are generated after data retrieval of the target object data using a target retrieval model, the target retrieval model is generated after comparative learning and training of an initial retrieval model using multimodal samples, and during the comparative learning and training process, similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples; and the retrieval results are displayed in a graphical user interface.

[0010] According to another aspect of an embodiment of the present application, a data retrieval method is also provided, including: obtaining a multimodal sample; using the multimodal sample to perform comparative learning training on an initial retrieval model to obtain a similarity calculation result of the multimodal sample, wherein the similarity calculation result is calculated step by step based on a batch size corresponding to the initial retrieval model, and the similarity calculation result is used to characterize the degree of semantic matching between samples of different modalities in the multimodal sample; generating a target retrieval model based on the similarity calculation result, wherein the target retrieval model is used to perform data retrieval on target object data to be retrieved to output a retrieval result.

[0011] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein when the program is running, any one of the above-mentioned data retrieval methods or model training methods is executed.

[0012] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned data retrieval methods or model training methods.

[0013] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, which implements any one of the above-mentioned data retrieval methods or model training methods when executed by a processor.

[0014] In an embodiment of the present application, by acquiring the target object data to be retrieved, the target object data to be retrieved is input into the target retrieval model, and the target retrieval model is used to perform data retrieval on the target object data, so that the retrieval result corresponding to the target object data can be accurately output, and the target retrieval model is generated after contrast learning training of the initial retrieval model using multimodal samples, and in the contrast learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, wherein the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the calculation and operation efficiency, the technical effect of being able to be applied to large-scale data sets is solved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related technology.

[0015] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a schematic diagram of an application scenario of a data retrieval method according to an embodiment of the present application;

[0018] Figure 2 This is a schematic diagram of the traditional contrast loss forward propagation process;

[0019] Figure 3 is a flow chart of a data retrieval method according to an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a forward propagation process of flash memory comparison loss of a single graphics card according to an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of the forward propagation process of the ring contrast loss according to an embodiment of the present application;

[0022] Figure 6 is a flow chart of a data retrieval method according to an embodiment of the present application;

[0023] Figure 7 is a flow chart of a data retrieval method according to an embodiment of the present application;

[0024] Figure 8 is a flow chart of a data retrieval method according to an embodiment of the present application;

[0025] Fig. 9 is a flow chart of a model training method according to an embodiment of the present application;

[0026] Fig.10 is a structural schematic diagram of a data retrieval device according to an embodiment of the present application;

[0027] Fig.11 is a structural schematic diagram of a data retrieval device according to an embodiment of the present application;

[0028] Fig.12 is a structural schematic diagram of a data retrieval device according to an embodiment of the present application;

[0029] Fig.13 is a structural schematic diagram of a data retrieval device according to an embodiment of the present application;

[0030] Fig.14 is a structural schematic diagram of a model training device according to an embodiment of the present application;

[0031] Fig.15 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0035] It should be noted that when the large model is actually used, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields, and can be specifically applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, etc., and can also be widely used in text-based sentiment classification, text summary generation, machine translation and other natural language processing tasks. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, data retrieval through a picture and text retrieval model in a picture and text search scenario is used as an example for explanation.

[0036] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0037] Multimodality: refers to the use of multiple ways, means and forms to express, transmit and understand information during information processing, transmission and reception. Exemplarily, multimodality may include: visual modality, auditory modality, tactile modality, olfactory modality, taste modality, language modality, emotional modality, spatial modality and temporal modality, etc., which are not limited here. Among them, the visual modality mainly involves visual information such as images, videos and animations. This information can be observed and understood by the eyes, such as image recognition, video analysis, etc. Auditory modality mainly involves auditory information such as sound and music. This information can be received and understood by the ears, such as speech recognition, music analysis, etc. Tactile modality mainly involves tactile information, such as tactile feedback, tactile interaction, etc. This information can be received and understood by sensory organs such as skin and muscles. Olfactory modality mainly involves odor information, such as odor recognition, odor analysis, etc. This information can be received and understood by the nose. Taste modality mainly involves taste information, such as taste recognition, taste analysis, etc. This information can be received and understood by the taste buds on the tongue. Linguistic modality mainly involves information such as text, symbols and language. This information can be transmitted and understood through reading, writing and communication. Emotional modality mainly involves emotional information, such as facial expressions, body language, etc. This information can be transmitted and understood by observing and understanding the emotional reactions of others. Spatial modality mainly involves spatial information, such as geographical location, spatial layout, etc. This information can be transmitted and understood through tools such as maps and navigation. Temporal modality mainly involves time information, such as time sequence, time interval, etc. This information can be transmitted and understood through timetables and schedules.

[0038] Contrastive Learning: Contrastive learning is a machine learning method that aims to learn feature representations by comparing the similarities and differences between different instances. In contrastive learning, a loss function is usually constructed so that similar samples are mapped to close positions in the feature space, while dissimilar samples are mapped to far positions. Contrastive learning has a wide range of applications in image recognition, text matching and other fields.

[0039] Similarity matrix: The similarity matrix is ​​an N×N matrix, where N represents the batch size, and each element in the matrix represents the similarity or distance between two samples. In contrastive learning, the similarity matrix is ​​often used to calculate the similarity between sample pairs, helping to build the loss function and optimize it. The similarity matrix can be calculated based on different metrics, such as Euclidean distance, cosine similarity, etc. By comparing different elements in the similarity matrix, the similarities and differences between samples can be found.

[0040] Contrastive Loss: Contrastive loss is a loss function commonly used in unsupervised learning and self-supervised learning. It aims to learn effective feature representation by minimizing the distance between samples of the same class and maximizing the distance between samples of different classes. In contrastive learning, positive and negative sample pairs are usually used. The contrastive loss is used to push the model to form a more obvious class boundary in the feature space, thereby improving the recognition ability of the model. The performance of this loss function is often positively correlated with the batch size, but the size of the video memory grows quadratically with the number of batch samples.

[0041] Batch size: Batch size is the number of samples fed into the model at one time during model training. Larger batch sizes can speed up the training process but may cause insufficient memory or unstable gradient descent, while smaller batch sizes can lead to more stable gradient descent but longer training time.

[0042] Softmax function: Softmax function is a commonly used activation function, also known as normalized exponential function, which is usually used to convert the output of the model into a probability distribution. The Softmax function maps each element in the output to the interval (0, 1) and ensures that the sum of all elements is 1. In contrastive learning, the Softmax function is often used to calculate the similarity score between samples. Through the Softmax function, the similarity score of the sample can be mapped to a probability distribution, indicating the probability that it belongs to a positive sample.

[0043] Streaming Multiprocessor (SM): SM is the basic processing unit in the Graphics Processing Unit (GPU), similar to the core in the Central Processing Unit (CPU). Each SM contains multiple Compute Unified Device Architecture (CUDA) cores, caches, schedulers, and other resources.

[0044] Dynamic Random Access Memory (DRAM): It is the main memory component in the GPU and is used to store the data being processed, program code, and other necessary information.

[0045] Static Random Access Memory (SRAM): is a high-speed random access memory used to cache data in computers and other electronic devices. Compared with dynamic random access memory (DRAM), SRAM has faster access speeds, but is more expensive and has a smaller storage capacity. Cache space: Cache is a high-speed memory located on the GPU chip, used to temporarily store frequently accessed data to speed up data access. Hierarchical structure: GPUs usually have multiple levels of cache, including L1 cache (Level 1Cache), L2 cache (Level 2Cache), etc. Among them, L1 cache is the fastest and has the smallest capacity, while L2 cache has a larger capacity but is slower.

[0046] Contrastive learning methods have made significant progress in deep learning research in recent years, especially in the fields of CLIP model for image-text matching, MOCO model and SIMCLR model for image self-representation, SIMCSE model for sentence representation, and BGE model for vector retrieval. These models rely on contrastive loss to optimize the similarity between samples, thereby learning effective feature representations and achieving excellent performance in various tasks.

[0047] Currently, the batch size of most contrastive learning methods is limited to about 10K, which limits the application of contrastive learning methods on larger datasets. If contrastive learning methods are to be applied on larger datasets, a large amount of computing resources, such as thousands of graphics cards, are usually required. Through in-depth analysis of these phenomena, it is found that the main reason for this limitation is the calculation process of contrastive loss. In traditional contrastive learning, due to the need to calculate the similarity matrix between batch samples, the occupation of video memory is a significant challenge. Specifically, the size of the similarity matrix is ​​N×N, where N is the batch size. A large amount of video memory is required to calculate and store this matrix, especially when the batch size is large. This causes the video memory occupation to grow quadratically with the batch size, which directly increases the demand for computing resources.

[0048] For example, in the training of the CLIP model, if a batch size of 64K is used and the dimension of the feature vector is 768, then the similarity matrix and its gradient alone require about 32GB of video memory, which is very unaffordable for common GPU graphics cards.

[0049] Because traditional contrastive learning methods use the Softmax function for normalization after calculating similarity, the computational burden increases dramatically. In particular, the video memory grows quadratically with the batch size, causing the traditional contrastive learning method to be opened up to 16K size on hundreds of GPUs.

[0050] Figure 2 This is a schematic diagram of the traditional contrast loss forward propagation process, such as Figure 2 As shown in the traditional contrastive learning process, two different encoders are used to process text data (such as "a cat") and image data (such as an image of a cat) respectively. Specifically, a text encoder is used to process text data and extract a text feature tensor K, where k i Represents the feature vector of the i-th sample, k1, k2, ... N Used to represent the index of different samples in a batch. Use image encoder to process image data and extract image feature tensor Q, q i Represents the feature vector of the i-th sample in the batch, q1, q2, ... q N The feature vectors used to represent different samples in a batch. The feature tensors Q and K are matrices of shape [N, d], where the number of rows N represents the number of samples and the number of columns d represents the feature dimension.

[0051] Then, by calculating the similarity between each eigenvector in Q and each eigenvector in K, a similarity matrix S[N,N] is constructed. Each element in the similarity matrix S represents the similarity between the ith sample in Q and the jth sample in K. Then, by determining the clustering or matching relationship between the samples, a clustering matrix P[N,N] is obtained. Each element in the clustering matrix P represents whether the ith sample in Q matches the jth sample in K, that is, whether it belongs to a positive sample pair. Exemplarily, for each sample in the batch, all other samples are traversed, and the value of the clustering matrix P is set according to whether it is a positive sample pair and the Softmax function operation. The Softmax function operation is performed on each row, the similarity value is converted into a probability distribution, and the similarity of each row is exponentially calculated and normalized. If the sample q i and sample k j is a positive sample pair, then P qikj and P qjki are all set to 1 (because the matching relationship is usually symmetric). If it is not a positive sample pair, then P qikj and P qjki is set to 0.

[0052] Then, the similarity of the positive sample pairs and the negative sample pairs is compared through the contrast loss function, and the contrast loss of each sample pair is calculated to construct the loss vector Loss[N,1], where N represents the number of samples in the batch. After the loss vector is calculated, it is back-propagated to calculate the gradient of each parameter, and then these gradients are used to update the weights of the model. At the same time, the loss tensor and other intermediate tensors are stored in the GPU's video memory for fast calculation and update.

[0053] After analysis, it was found that in traditional contrastive learning, there are several memory-intensive steps in the calculation process: 1. Similarity matrix calculation, which requires calculating the similarity between all samples and generating an N×N matrix; 2. Softmax operation, which performs a Softmax operation on each row, which involves exponential calculation and normalization of the similarity of each row; 3. Gradient storage and back propagation, which requires not only storing the similarity matrix itself, but also storing the corresponding gradients for back propagation. The above calculation method causes the memory usage to increase with the square of the batch size, thus facing a memory bottleneck in large-scale training.

[0054] According to an embodiment of the present application, a data retrieval method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0055] Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above data retrieval method provided in the embodiment of the present application can be applied to the following: Figure 1 The application scenarios shown are not limited to these. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0056] In an embodiment of the present application, the system composed of a client device and a server can execute the following steps: the client device executes steps of obtaining the target object data to be retrieved input or uploaded by the user in the graphical user interface, and sending the target object data to be retrieved to the server; the server executes steps of using the target retrieval model to perform data retrieval on the acquired target object data to be retrieved, and returns the retrieval results to the client device. It should be noted that when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiment of the present application can be carried out in the client device.

[0057] Under the above operating environment, this application provides Figure 3 The data retrieval method shown. Figure 3 is a flow chart of a data retrieval method according to an embodiment of the present application. Figure 3 As shown, the method may include the following steps:

[0058] Step S31, obtaining target object data to be retrieved;

[0059] Step S32, use the target retrieval model to perform data retrieval on the target object data and output the retrieval results; wherein, the target retrieval model is generated after contrast learning training of the initial retrieval model using multimodal samples, and during the contrast learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0060] In an embodiment of the present application, the target object data to be retrieved can be understood as the data to be retrieved input or uploaded by the user. The target object data can be multimodal data. Exemplarily, the target object data can be text data, image data, audio data, odor data, emotional data, etc., or can include text data and image data at the same time, or include text data, image data and audio data at the same time, etc., which is determined according to the user's retrieval needs and is not limited here.

[0061] The target retrieval model can be a large model or other deep learning model, which is not limited here. The target retrieval model can perform data retrieval on the target object data input or uploaded by the user, thereby outputting retrieval results that match the target object data. The retrieval results can be multimodal retrieval results. Exemplarily, the retrieval results can be text retrieval results, image retrieval results, audio retrieval results, smell retrieval results, emotion retrieval results, etc., and can also include text retrieval results and image retrieval results at the same time, or include text retrieval results, image retrieval results and audio retrieval results at the same time, etc., according to the corresponding output of the target object data to be retrieved, which is not limited here.

[0062] When training the retrieval model, multimodal samples are used to compare and learn the initial retrieval model, that is, a large number of samples of different modalities are used as training data, and a large number of samples of different modalities are used to compare and learn the initial retrieval model, so as to obtain the target retrieval model. Among them, the multimodal samples can include multiple different modal pairs of samples, and any different modal pairs of samples are matched.

[0063] Exemplarily, taking the different modal pair samples as image-text pair samples as an example, the image-text pair samples include matching text samples and image samples. Among them, the text sample is, for example, "a cat", and the image sample is, for example, an image of a cat, or, the text sample is, for example, "a dog", and the image sample is, for example, an image of a dog, which is not limited here. In addition, the different modal pair samples can also be text-audio pair samples, and the text-audio pair samples include matching text samples and audio samples, wherein the text sample is, for example, a section of lyrics, and the audio sample is, for example, the song audio corresponding to the lyrics, which is not limited here. It can be understood that the above description of different modal pair samples is only an example, and the different modal pair samples can be adjusted according to actual needs, which is not limited here.

[0064] During the comparative learning and training process of the initial retrieval model, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model. The similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples, that is, the probability distribution of samples of different modal pairs belonging to positive sample pairs.

[0065] In the contrastive learning training process, considering that processing the entire similarity matrix at one time will lead to an increase in the batch size, which will in turn increase the computational overhead of the contrast loss and occupy the video memory. Therefore, this application proposes to divide the entire similarity matrix into multiple small blocks according to batches and process them step by step, and calculate the batch size step by step, so as to reduce the occupation of video memory in the traditional contrast loss calculation process by calculating the intermediate variable similarity matrix in blocks, thereby effectively reducing the computational overhead and reducing the video memory occupation.

[0066] It can be seen that the present application uses multimodal samples to perform contrastive learning training on the initial retrieval model, and in the process of contrastive learning training, the batch size corresponding to the initial retrieval model is processed in blocks, and the similarity calculation results of the multimodal samples are obtained by step-by-step calculation of the batch size, that is, the semantic matching degree of different modal samples in the multimodal samples is obtained, so as to determine the loss function of the initial retrieval model according to the similarity calculation results, and then adjust the model parameters of the initial retrieval model according to the loss function to update and optimize the initial retrieval model, and finally train to obtain the target retrieval model. Therefore, the block calculation significantly reduces the occupancy of the video memory of a single node. Compared with the traditional contrastive learning method, there is no need to process the entire similarity matrix at the same time, which avoids the quadratic growth of the video memory demand with the batch size, so that under the same hardware conditions and under the premise of ensuring the calculation and operation efficiency, the batch size can be greatly increased to a scale that is difficult to achieve with the traditional contrastive learning method.

[0067] After the target retrieval model is trained, the target object data to be retrieved is obtained, the target object data to be retrieved is input into the target retrieval model, and the target retrieval model is used to perform data retrieval on the target object data, so that the retrieval results corresponding to the target object data can be accurately output.

[0068] The above-mentioned data retrieval method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving data retrieval in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, for example: data retrieval scenarios for e-commerce services, data retrieval scenarios for educational services, data retrieval scenarios for medical services, etc., which are not limited here.

[0069] According to the embodiment of the present application, by acquiring the target object data to be retrieved, the target object data to be retrieved is input into the target retrieval model, and the target retrieval model is used to perform data retrieval on the target object data, so that the retrieval result corresponding to the target object data can be accurately output, and the target retrieval model is generated after contrast learning training of the initial retrieval model using multimodal samples, and in the contrast learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, wherein the similarity calculation results are used to characterize the degree of semantic matching of different modal samples in the multimodal samples, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the calculation and operation efficiency, the technical effect of being able to be applied to large-scale data sets is solved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related technology.

[0070] In an optional embodiment, the initial retrieval model includes: a text encoder and an image encoder, the different modality samples include text samples and image samples, and the data retrieval method further includes the following method steps:

[0071] Step S33: Use a text encoder to perform text encoding on the text sample to obtain a first encoding result.

[0072] Step S34, using an image encoder to perform image encoding on the image sample to obtain a second encoding result;

[0073] Step S35, performing step-by-step similarity calculation on the first encoding result and the second encoding result based on the processor core performance to generate a target retrieval model.

[0074] In the embodiments of the present application, for the convenience of description, a multimodal sample is a plurality of image-text pair samples, and different modal samples include text samples and image samples. It should be noted that the data retrieval method proposed in the embodiments of the present application can be applied to multimodal data retrieval, and is not limited to the image data and text data in the examples.

[0075] Under the premise of taking text samples and image samples as examples for description, the initial retrieval model includes: a text encoder and an image encoder, wherein the text encoder is used to encode text data to obtain a text feature tensor. The image encoder is used to encode image data to obtain an image feature tensor. It should be noted that if the different modal data includes audio data, the initial retrieval model includes an audio encoder, which is not limited here.

[0076] When performing contrastive learning training on the initial retrieval model, for the text samples in the multiple text-image pair samples, a text encoder is used to perform text encoding on the text samples to obtain a first encoding result, i.e., a text feature tensor corresponding to any text sample. For the image samples in the multiple text-image pair samples, an image encoder is used to perform image encoding on the image samples to obtain a second encoding result, i.e., an image feature tensor corresponding to any image sample.

[0077] After obtaining the first encoding result and the second encoding result, the first encoding result and the second encoding result are calculated in steps based on the processor core performance, thereby generating a target retrieval model. The processor may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a network processor (NP), etc., which are not limited here.

[0078] In the embodiment of the present application, for the convenience of description, the processor is taken as a graphics processor as an example for explanation. Among them, the processor core performance can be understood as the number of SMs in the GPU. It can be understood that when calculating by blocks, the block size of the similarity matrix is ​​determined by the number of SMs in the GPU chip, that is, the size of each block is determined based on the number of SMs in the GPU chip. Therefore, the first encoding result and the second encoding result are calculated in steps according to the number of SMs in the GPU, thereby generating a target retrieval model.

[0079] In an optional embodiment, in step S35, performing step-by-step similarity calculation on the first encoding result and the second encoding result based on the processor core performance, and generating the target retrieval model may include the following method steps:

[0080] Step S351, performing step-by-step similarity calculation on the first encoding result and the second encoding result based on the processor core performance to obtain a comparison loss;

[0081] Step S352, updating the model parameters of the initial retrieval model according to the contrast loss to generate a target retrieval model.

[0082] In an embodiment of the present application, when performing step-by-step similarity calculations on the first encoding result and the second encoding result based on the processor core performance to generate a target retrieval model, the first encoding result and the second encoding result can be calculated step-by-step based on the processor core performance to obtain the contrast loss of the entire similarity matrix. The contrast loss is used to represent the degree of matching between the first encoding result and the second encoding result, that is, the degree of matching between the text sample and the image sample. After obtaining the contrast loss of the entire similarity matrix, the model parameters of the initial retrieval model are updated and adjusted based on the contrast loss to generate a target retrieval model, thereby significantly reducing the video memory usage.

[0083] In an optional embodiment, in step S351, performing step-by-step similarity calculation on the first encoding result and the second encoding result based on the processor core performance to obtain the contrast loss may include the following method steps:

[0084] Step S3511, determining a target block size based on processor core performance;

[0085] Step S3512, performing step-by-step similarity calculation on the first encoding result and the second encoding result according to the target block size to obtain a comparison loss.

[0086] In the embodiment of the present application, when the first encoding result and the second encoding result are subjected to step-by-step similarity calculation based on the processor core performance to obtain the contrast loss, the target block size can be determined based on the processor core performance, that is, the size of each block matrix is ​​determined based on the number of SMs in the GPU. Then, the first encoding result and the second encoding result are subjected to step-by-step similarity calculation based on the target block size to obtain the contrast loss of the entire similarity matrix.

[0087] In an optional embodiment, in step S3511, determining the target block size based on the processor core performance may include the following method steps:

[0088] Step S35111, determining the first block size based on the number of cores of a single processor; or,

[0089] Step S35112, determining a second block size based on the number of cores of the plurality of processors; wherein the first block size is smaller than the second block size.

[0090] In the embodiment of the present application, when determining the target block size based on the processor core performance, the target block size can be determined according to the number of graphics cards, and different determination methods are used for different numbers of graphics cards.

[0091] If there is only one graphics card, that is, a single processor, the first block size may be determined based on the number of cores of the single processor, for example, the block size of the similarity matrix may be determined based on the number of SMs of a single GPU chip.

[0092] Exemplarily, the similarity matrix block calculation of a single graphics processor can extract a block containing q query samples and k key value samples from the query set (Q) and the key value set (K), and calculate the similarity matrix size of the block to be q×k. The size of the q×k matrix is ​​determined by the number of SMs of a single GPU chip. If q(64)×k(128) is put into the static random access memory (SRAM) for calculation, one block in the SRAM is calculated each time.

[0093] It should be noted that the example here assumes that each device has only one graphics card, but theoretically there can be up to eight graphics cards on a device.

[0094] If there are multiple graphics cards, that is, multiple processors, the second block size can be determined based on the number of cores of the multiple processors, for example, the block size of the similarity matrix is ​​determined according to the number of SMs of multiple GPU chips.

[0095] For example, the similarity matrix of multiple graphics processors is calculated in blocks, so that each device is responsible for calculating only a part of the similarity matrix, and key values ​​are transmitted between different devices, thereby further reducing the memory usage of a single device. For example, device 1 calculates q1, q2, k1, k2 at time T1, and device 2 calculates q3, q4, k3, k4 at time T1, which is not limited here.

[0096] It should be noted that the first block size is smaller than the second block size. It can be seen that in the multiple graphics card scenario, by distributing the calculation of the similarity matrix to different devices, the graphics memory requirements of a single device are further reduced, and the communication consumption and graphics card utilization efficiency in the multi-node training process are optimized. Since each device only needs to process the similarity calculation related to itself, without having to store the entire matrix, and the key value is transmitted during the calculation, the node communication consumption can be greatly reduced.

[0097] In an optional embodiment, in step S3512, performing step-by-step similarity calculation on the first encoding result and the second encoding result according to the target block size to obtain the contrast loss may include the following method steps:

[0098] Step S35121, obtaining a similarity matrix between the first encoding result and the second encoding result;

[0099] Step S35122, dividing the similarity matrix into a plurality of first block matrices according to the first block size, wherein each of the plurality of first block matrices includes: a first number of query samples and a second number of key value samples, the first number and the second number being determined by the first block size;

[0100] Step S35123, using a single processor to perform serial similarity calculation on multiple first block matrices to obtain contrast loss.

[0101] In an embodiment of the present application, in a single graphics card scenario, that is, a single processor, the first encoding result and the second encoding result are subjected to step-by-step similarity calculations based on the target block size. When the contrast loss is obtained, a similarity matrix between the first encoding result and the second encoding result can be obtained, that is, an intermediate variable similarity matrix (e.g., an N×N matrix) between the first encoding result and the second encoding result can be obtained.

[0102] After obtaining the similarity matrix, the similarity matrix is ​​divided into blocks according to the determined first block size, and the similarity matrix is ​​divided into multiple first block matrices. Among them, each block matrix in the multiple first block matrices includes: a first number of query samples and a second number of key value samples, and the first number and the second number are determined by the first block size. Exemplarily, each block matrix in the multiple first block matrices can be understood as a block of q query samples and k key value samples, and the number of q and k is determined according to the first block size, that is, according to the number of SMs of a single GPU chip.

[0103] After the block division is performed to obtain multiple first block matrices, a single processor is used to perform serial similarity calculation on the multiple first block matrices, thereby obtaining the contrast loss. The serial similarity calculation can be understood as performing similarity calculations on each first block matrix obtained by the division one by one in sequence. In addition, during the calculation, the similarity calculation of each first block matrix is ​​independent and does not depend on the calculation results of other first block matrices, thereby avoiding the problem of video memory occupation caused by calculating the entire matrix at the same time. This calculation method can also be called an inner loop serial similarity calculation. By using a single processor to perform serial similarity calculations on multiple first block matrices, the contrast loss of the entire similarity matrix is ​​obtained.

[0104] In an optional embodiment, in step S35123, using a single processor to perform serial similarity calculation on multiple first block matrices to obtain the contrast loss may include the following method steps:

[0105] Step S351231, batch processing is performed on a plurality of first block matrices according to a preset number of blocks to obtain a plurality of batches of block matrices;

[0106] Step S351232, using a single processor to perform step-by-step similarity calculations on multiple batches of block matrices in sequence to obtain contrast loss.

[0107] In the embodiment of the present application, when a single processor is used to perform serial similarity calculation on multiple first block matrices to obtain contrast loss, the multiple first block matrices can be processed in batches according to a preset number of blocks to obtain multiple batches of block matrices. The preset number of blocks can be, for example, a whole row of blocks, that is, the entire similarity matrix is ​​divided into multiple single-row matrices, and multiple first block matrices in each single-row matrix are processed in batches by row to obtain multiple batches of block matrices.

[0108] After dividing into multiple batches of block matrices, a single processor is used to perform step-by-step similarity calculations on the multiple batches of block matrices in sequence to obtain the contrast loss.

[0109] In an optional embodiment, in step S351232, using a single processor to sequentially perform step-by-step similarity calculations on multiple batches of block matrices to obtain the contrast loss may include the following method steps:

[0110] Step S3512321, using a single processor to perform step-by-step similarity calculations on multiple batches of block matrices in sequence to obtain a first calculation result;

[0111] Step S3512322, using a single processor to perform similarity calculation on positive samples in the multimodal samples to obtain a second calculation result, wherein the positive samples are used to characterize that the image samples are semantically related to the text samples;

[0112] Step S3512323, obtaining contrast loss based on the first calculation result and the second calculation result.

[0113] In an embodiment of the present application, when a single processor is used to perform step-by-step similarity calculations on multiple batches of block matrices in sequence to obtain contrast loss, a single processor can be used to perform step-by-step similarity calculations on multiple batches of block matrices in sequence to obtain a first calculation result. The first calculation result can be understood as the similarity calculation result of each single-row matrix obtained by calculating the similarity of each single-row matrix. For example, the similarity calculation result of the i-th row matrix can be recorded as lse i , the first calculation result represents the similarity calculation result of each single row matrix in the entire similarity matrix, which can be recorded as lse[N,1], which includes N rows and 1 column, N i Represents the i-th single row matrix.

[0114] It is understandable that the similarities of multiple single-row matrices in the entire similarity matrix can be calculated in parallel, so as to obtain the similarity calculation result of each single-row matrix. This calculation method can also be called outer loop parallel similarity calculation.

[0115] In addition, a single processor can be used to perform similarity calculation on positive samples in multimodal samples to obtain a second calculation result. Among them, the positive sample is used to characterize that the samples of different modalities are semantically related. Taking the samples of different modalities as image-text pairs as an example, the positive sample is used to characterize that the image sample and the text sample are semantically related. It can be understood that the positive sample is the sample of image-text matching. The second calculation result can be understood as the similarity calculation result obtained by performing similarity calculation on the sample of image-text matching.

[0116] It can be understood that the diagonal blocks from the upper left to the lower right in the entire similarity matrix are positive samples, such as q1k1, q2k2, ... q N k N The second calculation result can be understood as the similarity calculation result of the diagonal block from the upper left to the lower right. For example, the similarity calculation result of the positive sample in the i-th row in the entire similarity matrix can be recorded as gs i , the second calculation result represents the similarity calculation result of the positive sample in the entire similarity matrix, which can be recorded as gs[N,1], including N rows and 1 column, N i Represents the positive sample of the i-th row.

[0117] After the first calculation result and the second calculation result are calculated, that is, after the similarity calculation result of each single-row matrix and the similarity calculation result of the diagonal are calculated, the comparison loss of the entire similarity matrix can be determined according to the first calculation result and the second calculation result.

[0118] It can be understood that the first calculation result lse of each single row matrix i It can be regarded as the denominator obtained by calculating the Softmax function, and the second calculation result gs of the diagonal i It can be regarded as the numerator obtained by calculating the Softmax function, where the denominator of the Softmax function in the block can be obtained by performing exponential operations on the similarity matrix in the block and summing it up, that is, By calculating only the similarity of the positive sample pairs and performing exponential operations, the numerator of the Softmax function, gs, can be efficiently calculated. i =exp(q i k i ). Therefore, the contrast loss Loss of the entire similarity matrix can be expressed as Loss = gs i / lse i , that is, the ratio of the second calculation result to the first calculation result is the comparison loss Loss of the entire similarity matrix.

[0119] It is understandable that the negative sample pairs in the similarity matrix will be reflected in lse. lse usually refers to the least square error, also known as the L2 loss function, which is used to measure the similarity between two samples in contrast loss.

[0120] In an optional embodiment, in step S3512321, using a single processor to sequentially perform step-by-step similarity calculations on multiple batches of block matrices to obtain a first calculation result may include the following method steps:

[0121] Step S35123211, using a single processor to perform intra-block similarity calculation on each block matrix in a single batch of block matrices of the multiple batches of block matrices, to obtain a third calculation result;

[0122] Step S35123212, using a single processor to perform cross-block accumulation calculation on the third calculation results corresponding to each block matrix in a single batch of block matrices, to obtain a fourth calculation result;

[0123] Step S35123213, using a single processor to perform batch-by-batch accumulation calculations on the fourth calculation results corresponding to each batch of block matrices in the multiple batches of block matrices to obtain a first calculation result.

[0124] In an embodiment of the present application, when a single processor is used to perform step-by-step similarity calculations on multiple batches of block matrices in sequence to obtain a first calculation result, a single processor can be used to perform intra-block similarity calculations on each block matrix in a single batch of block matrices of the multiple batches of block matrices to obtain a third calculation result. It can be understood that the similarity calculation is performed on each block in each single-row matrix to obtain the third calculation result, and since the similarity calculation is only performed within each block, the video memory occupancy is greatly reduced. The third calculation result can be understood as the denominator of the Softmax function within each block.

[0125] Exemplarily, when calculating the intra-block similarity of each block matrix, the denominator of the Softmax function in the block is obtained by performing exponential operation on the similarity matrix in the block and summing them, that is,

[0126] After obtaining the third calculation result, a single processor is used to perform cross-block cumulative calculation on the third calculation results corresponding to each block matrix in a single batch of block matrices to obtain a fourth calculation result. It can be understood that after obtaining the third calculation result of each block, the third calculation results of each block matrix belonging to the same batch are cumulatively calculated to obtain the fourth calculation result corresponding to each single-row matrix.

[0127] Exemplarily, when performing cross-block cumulative calculations based on the third calculation results corresponding to each block matrix in each batch block matrix, the denominator of the Softmax function of each query sample q in all blocks is accumulated separately, that is, the denominator of the Softmax function of each row of the matrix is ​​accumulated to obtain the fourth calculation result corresponding to each single row of the matrix, thereby avoiding the problem of video memory occupation caused by calculating the entire matrix at the same time.

[0128] After obtaining the fourth calculation result, a single processor is used to perform batch-by-batch cumulative calculations on the fourth calculation results corresponding to each batch of block matrices in the multiple batches of block matrices to obtain the first calculation result. It can be understood that after obtaining the fourth calculation result of each row matrix, the fourth calculation result corresponding to each row matrix in the similarity matrix is ​​cumulatively calculated, thereby gradually completing the calculation of the entire batch and obtaining the first calculation result of the entire similarity matrix.

[0129] In an optional embodiment, in step S35123211, using a single processor to perform intra-block similarity calculation on each block matrix in a single batch of block matrices of multiple batches of block matrices to obtain the third calculation result may include the following method steps:

[0130] Step S351232111: In the cache of a single processor, intra-block similarity calculation is performed on each block matrix in a single batch of block matrices of the multiple batches of block matrices to obtain a third calculation result.

[0131] In an embodiment of the present application, when a single processor is used to perform intra-block similarity calculation on each block matrix in a single batch block matrix of multiple batch block matrices to obtain a third calculation result, intra-block similarity calculation can be performed on each block matrix in a single batch block matrix of multiple batch block matrices within the cache of a single processor to obtain a third calculation result.

[0132] The cache can be understood as a graphics card cache. For example, the cache can be SRAM, DRAM, non-volatile random access memory (NVRAM), etc., which is not limited here. Taking the cache as SRAM as an example, it can be understood that when calculating the similarity of each block, the calculation is performed in the SRAM. In the process of serial calculation, one block in the SRAM is calculated each time, and each block is calculated in the SRAM in sequence to obtain the third calculation result.

[0133] Figure 4 is a schematic diagram of the forward propagation process of flash memory comparison loss of a single graphics card according to an embodiment of the present application, such as Figure 4As shown, in the comparative learning process of the embodiment of the present application, in a single graphics card scenario, two different encoders are used to process text samples (such as "a cat") and image samples (such as an image of a cat) respectively. Specifically, the text encoder is used to encode the text sample to obtain a first encoding result, that is, to obtain a text feature tensor K, where k i Represents the feature vector of the i-th text sample. Use the image encoder to encode the image sample and obtain the second encoding result, that is, the image feature tensor Q, q i Represents the feature vector of the i-th image sample in the batch. Q and K are matrices of shape [N, d], where the number of rows N represents the number of samples and the number of columns d represents the feature dimension.

[0134] Then, the target block size is determined according to the number of SMs of a single graphics card, and the similarity between each eigenvector in Q and each eigenvector in K is calculated to construct a similarity matrix. Then, the constructed similarity matrix is ​​divided into blocks based on the determined target block size to obtain multiple first block matrices. Then, the multiple first block matrices obtained by division are processed in batches according to one batch per row. Furthermore, the similarity of each first block matrix within the block is calculated in SRAM according to the inner loop serial calculation method to obtain the third calculation result. Then, the similarities of the first block matrices in the same row are accumulated across blocks in an outer loop parallel calculation method to obtain the fourth calculation result, that is, the similarity of each row. Finally, the similarity of each row is accumulated batch by batch, and the calculation of the entire batch is gradually completed to obtain the first calculation result lse[N,1], which is used as the denominator of the Softmax function, where,

[0135] At the same time, the similarity of the positive samples in the similarity matrix is ​​calculated, that is, the similarity of the matching vector pairs is calculated, that is, the similarity of the diagonal blocks from the upper left to the lower right in the similarity matrix is ​​calculated, and the second calculation result gs[N,1] is obtained as the numerator of the Softmax function, where gs i =exp(q i k i ).

[0136] Finally, according to the first calculation result and the second calculation result, the final comparison loss of the similarity matrix Loss = gs can be determined. i / lse i .

[0137] It can be seen that this application divides the batch into multiple small blocks for step-by-step processing. The block calculation significantly reduces the single-node video memory usage, and only calculates the similarity index and of each query and key during storage, greatly reducing storage requirements. Compared with traditional methods, there is no need to process the entire similarity matrix at the same time, that is, during the calculation process, the entire similarity matrix is ​​avoided from being processed at one time, and the video memory requirement is avoided from growing quadratically with the batch size. This allows the batch size to be greatly increased under the same hardware conditions, reaching a scale that is difficult to achieve with traditional methods.

[0138] In an optional embodiment, in step S3512, performing step-by-step similarity calculation on the first encoding result and the second encoding result according to the target block size to obtain the contrast loss may include the following method steps:

[0139] Step S35124, obtaining a similarity matrix between the first encoding result and the second encoding result;

[0140] Step S35125, dividing the similarity matrix into a plurality of second block matrices according to the second block size, wherein each of the plurality of second block matrices includes: a third number of query samples and a fourth number of key value samples, the third number and the fourth number being determined by the second block size;

[0141] Step S35126, using multiple processors to perform parallel similarity calculations on multiple second block matrices to obtain contrast loss.

[0142] In an embodiment of the present application, in a scenario with multiple graphics cards, that is, with multiple processors, when the similarity between the first encoding result and the second encoding result is calculated step by step based on the target block size to obtain the contrast loss, the similarity matrix between the first encoding result and the second encoding result can be obtained, that is, the intermediate variable similarity matrix (for example, an N×N matrix) between the first encoding result and the second encoding result can be obtained.

[0143] After obtaining the similarity matrix, the similarity matrix is ​​divided into blocks according to the determined second block size, and the similarity matrix is ​​divided into multiple second block matrices. Among them, each block matrix in the multiple second block matrices includes: a third number of query samples and a fourth number of key value samples, and the third number and the fourth number are determined by the second block size. Exemplarily, each block matrix in the multiple second block matrices is generally evenly distributed among multiple devices, that is, the number of query samples and key value samples is generally evenly distributed among multiple devices, and each device is only responsible for one block at each moment. For example, the number of N is 120, and the number of graphics cards is 10, then each graphics card is allocated 12. Afterwards, multiple processors are used to perform parallel similarity calculations on multiple second block matrices, that is, the similarity results calculated by each device are respectively calculated in parallel, and finally the comparison loss is obtained.

[0144] For example, Figure 5 is a schematic diagram of the forward propagation process of the ring contrast loss according to an embodiment of the present application, such as Figure 5 As shown in the figure, in the training scenario of multiple graphics cards or multiple nodes, the calculation method is further optimized through the Ring Contrastive Loss. During the calculation, each device is only responsible for calculating part of the similarity matrix, which further reduces the memory usage of a single device. Since each device only needs to process the similarity calculation related to itself, it does not need to store the entire matrix. And the key value is transmitted during the calculation, which greatly reduces the node communication consumption.

[0145] Further, Figure 5 In the example, a device has only one graphics card, there are three devices, and N is 6, and the encoding process of the encoder is omitted. In the specific calculation process, based on the concept of the ring, device 1 calculates q1, q2, k1, k2 at time T1, device 2 calculates q3, q4, k3, k4 at time T1, and device 3 calculates q5, q6, k5, k6 at time T1. Device 1 calculates q1, q2, k5, k6 at time T2, where k5 and k6 are passed from device 3 to device 1, device 2 calculates q3, q4, k1, k2 at time T2, where k1 and k2 are passed from device 1 to device 2, and device 3 calculates q5, q6, k3, k4 at time T2, where k3 and k4 are passed from device 2 to device 3, thus forming a loop, and each subsequent moment will constitute a loop, and so on, which will not be repeated. Then, the calculation results of each device are accumulated in parallel to obtain lse1 to lse6, thereby obtaining the L2 loss function, and finally the final loss function Loss is determined according to the L2 loss function.

[0146] In the embodiment of the present application, the multiplication method is used to efficiently calculate the similarity and the Softmax function. The method optimizes the calculation process so that the batch size of contrastive learning can be increased to 1M in a single-machine environment, thereby completely breaking the bottleneck of batch size for contrastive learning.

[0147] In addition, by combining the flash contrast loss on a single graphics card and the ring contrast loss on multiple graphics cards, the batch sizes of the contrast losses can be greatly superimposed, nearly achieving an infinite effect. Therefore, a method of infinite batch size contrast loss is proposed.

[0148] It is easy to understand that the beneficial effects of the data retrieval method provided by the present application include the following points.

[0149] Beneficial effect (1): In order to solve the problem that the memory usage in contrast loss increases quadratically with the batch size and large batch training cannot be used, a block-by-block similarity matrix calculation method is proposed. By dividing the entire batch into multiple small blocks and gradually calculating the similarity, only the denominator of the Softmax function is stored, thereby significantly reducing the memory usage and solving the problem that the increase in batch size leads to a quadratic increase in memory requirements.

[0150] Beneficial effect (2): For a single graphics card environment, by efficiently utilizing the flash memory in the GPU and the cross-block accumulation mechanism, it is possible to process large amounts of data in a single graphics card environment.

[0151] Beneficial effect (3): In a multi-graphics card scenario, the ring contrast loss further reduces the graphics memory requirement of a single device by distributing the calculation of the similarity matrix to different devices, thereby optimizing the communication consumption and graphics card utilization efficiency during multi-node training.

[0152] Beneficial effect (4): This scheme can infinitely expand the batch size of contrastive learning and has great potential in big data and large model applications.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0154] In addition, it should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0155] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0156] According to the embodiment of the present application, there is also provided Figure 6 A data retrieval method shown in FIG. Figure 6 is a flow chart of a data retrieval method according to an embodiment of the present application, such as Figure 6 As shown, the method includes:

[0157] Step S61, obtaining product image data to be retrieved;

[0158] Step S62, use the target image-text retrieval model to perform data retrieval on the product image data, and output the product image retrieval results; wherein, the target image-text retrieval model is generated after comparative learning and training of the initial image-text retrieval model using multimodal samples, and during the comparative learning and training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial image-text retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0159] In an embodiment of the present application, the product image data to be retrieved can be understood as the product image data to be retrieved input or uploaded by the user. For example, if the user wants to retrieve the brand of a pen, the product image data to be retrieved can be a product image taken of the pen. If the user wants to retrieve the brand of a piece of clothing, the product image data to be retrieved can be a picture taken of the clothing. There is no limitation here.

[0160] The target image and text retrieval model can be a large model or other deep learning model, which is not limited here. The target image and text retrieval model can perform data retrieval on the product image data input or uploaded by the user, thereby outputting product image retrieval results that match the product image data. For example, if the user uploads a product image of a pen, the output product image retrieval results can be image data such as the brand of the pen and other models of the pen, and can also output text data such as the brand introduction of the pen, which is not limited here.

[0161] The target image-text retrieval model is generated by contrastive learning training of the initial image-text retrieval model using multimodal samples. During the contrastive learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial image-text retrieval model. The similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0162] For other descriptions of the target image-text retrieval model, please refer to the descriptions of the target retrieval model in step S31 and step S32, which will not be repeated here.

[0163] The above-mentioned data retrieval method provided in the embodiment of the present application can also be applied to application scenarios involving data retrieval in the fields of educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, for example: data retrieval scenarios for educational services, data retrieval scenarios for medical services, etc., which are not limited here.

[0164] According to the embodiment of the present application, by obtaining the product image data to be retrieved, the product image data to be retrieved is input into the target image-text retrieval model, and the target image-text retrieval model is used to perform data retrieval on the product image data, so that the product image retrieval result corresponding to the product image data can be accurately output, and the target image-text retrieval model is generated after contrast learning training of the initial image-text retrieval model using multimodal samples, and in the contrast learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial image-text retrieval model, wherein the similarity calculation results are used to characterize the degree of semantic matching of different modal samples in the multimodal samples, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the computing and operating efficiency, the technical effect of being able to be applied to large-scale data sets is solved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related technology.

[0165] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0166] According to the embodiment of the present application, there is also provided Figure 7 A data retrieval method is shown. Figure 7 is a flow chart of a data retrieval method according to an embodiment of the present application, such as Figure 7 As shown, the method includes:

[0167] Step S71, obtaining a data retrieval request through a first application programming interface, wherein the request data carried in the data retrieval request includes: target object data to be retrieved;

[0168] Step S72, returning a data retrieval response through a second application programming interface, wherein the response data carried in the data retrieval response includes: a retrieval result, wherein the retrieval result is generated after performing data retrieval on the target object data using a target retrieval model, wherein the target retrieval model is generated after performing comparative learning training on an initial retrieval model using multimodal samples, and during the comparative learning training process, similarity calculation results of the multimodal samples are calculated step by step based on a batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0169] The first application programming interface and the second application programming interface may be the same application programming interface or different application programming interfaces. In an optional embodiment, the interface parameters in the first application programming interface and the second application programming interface may include but are not limited to: interface global identifier, interface signature key, interface timestamp, interface request identifier, system call credential identifier, etc. The first application programming interface may use GET or POST as the interface request method to obtain the file processing request. The second application programming interface may use JSON format to feedback the file processing response.

[0170] In an embodiment of the present application, the request data carried in the data retrieval request includes: target object data to be retrieved, and the data retrieval request can be understood as a request for requesting retrieval of the target object data to be retrieved. The response data carried in the data retrieval response includes: retrieval results, and the data retrieval response is used to reply to the data retrieval request, that is, to feed back the retrieval results to the user. Among them, the retrieval results are generated after data retrieval of the target object data using the target retrieval model, and the target retrieval model is generated after comparative learning and training of the initial retrieval model using multimodal samples, and in the comparative learning and training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0171] For other descriptions of the target object data to be retrieved, reference may be made to the descriptions of the target object data to be retrieved in step S31 and step S32, which will not be described in detail here.

[0172] For other descriptions of the target retrieval model, please refer to the descriptions of the target retrieval model in step S31 and step S32, which will not be repeated here.

[0173] The above-mentioned data retrieval method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving data retrieval in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, for example: data retrieval scenarios for e-commerce services, data retrieval scenarios for educational services, data retrieval scenarios for medical services, etc., which are not limited here.

[0174] According to an embodiment of the present application, a data retrieval request is obtained through a first application programming interface, wherein the request data carried in the data retrieval request includes: target object data to be retrieved, and a data retrieval response is returned through a second application programming interface, wherein the response data carried in the data retrieval response includes: retrieval results, wherein the retrieval results are generated after data retrieval of the target object data using a target retrieval model, and the target retrieval model is generated after contrast learning training of an initial retrieval model using multimodal samples, and during the contrast learning training process, similarity calculation results of multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the semantic matching degree of different modal samples in the multimodal samples, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the computing and operating efficiency, the technical effect of being able to be applied to large-scale data sets is solved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related technology.

[0175] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0176] According to the embodiment of the present application, there is also provided Figure 8 A data retrieval method is shown. Figure 8 is a flow chart of a data retrieval method according to an embodiment of the present application, such as Figure 8 As shown, the method includes:

[0177] Step S81, obtaining the currently input data retrieval dialogue request, wherein the request data carried in the data retrieval dialogue request includes: target object data to be retrieved;

[0178] Step S82, in response to the data retrieval dialogue request, returning a data retrieval dialogue reply, wherein the information carried in the data retrieval dialogue reply includes: a retrieval result, which is generated after data retrieval of the target object data using the target retrieval model, and the target retrieval model is generated after contrast learning training of the initial retrieval model using multimodal samples, and in the contrast learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the semantic matching degree of different modal samples in the multimodal samples;

[0179] Step S83, displaying the search results in a graphical user interface.

[0180] In the embodiment of the present application, the data retrieval dialogue request can be understood as the data retrieval dialogue request input by the user when conducting a dialogue with the artificial intelligence. The request data carried in the data retrieval dialogue request includes: the target object data to be retrieved. The data retrieval dialogue request is used to request the retrieval of the target object data, thereby obtaining the corresponding retrieval result. The data retrieval dialogue reply can be understood as the reply made by the artificial intelligence based on the user's data retrieval dialogue request. The information carried in the data retrieval dialogue reply includes: the retrieval result. The retrieval result is generated after the target object data is retrieved using the target retrieval model, that is, the retrieval result corresponding to the target object data to be retrieved. After obtaining the retrieval result, the retrieval result is displayed in the graphical user interface to provide feedback to the user.

[0181] Among them, the target retrieval model is generated by contrastive learning training of the initial retrieval model using multimodal samples, and during the contrastive learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the semantic matching degree of different modal samples in the multimodal samples. For other descriptions of the target retrieval model, please refer to the description of the target retrieval model in step S31 and step S32, which will not be repeated here.

[0182] For other descriptions of the target object data to be retrieved, reference may be made to the descriptions of the target object data to be retrieved in step S31 and step S32, which will not be described in detail here.

[0183] The above-mentioned data retrieval method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving data retrieval in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, for example: data retrieval scenarios for e-commerce services, data retrieval scenarios for educational services, data retrieval scenarios for medical services, etc., which are not limited here.

[0184] According to an embodiment of the present application, by obtaining a currently input data retrieval dialogue request, wherein the request data carried in the data retrieval dialogue request includes: target object data to be retrieved, and then in response to the data retrieval dialogue request, a data retrieval dialogue reply is returned, wherein the information carried in the data retrieval dialogue reply includes: a retrieval result, wherein the retrieval result is generated after data retrieval of the target object data using a target retrieval model, and the target retrieval model is generated after contrast learning training of an initial retrieval model using multimodal samples, and in the contrast learning training process, the similarity calculation result of the multimodal sample is calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation result is used to characterize the semantic matching degree of different modal samples in the multimodal sample, and finally based on the obtained retrieval result, the retrieval result is displayed in a graphical user interface, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving the effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the computing and operating efficiency, the technical effect of being able to be applied to large-scale data sets is solved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related art.

[0185] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0186] According to the embodiment of the present application, there is also provided Fig. 9 A model training method shown. Fig. 9 is a flow chart of a model training method according to an embodiment of the present application, such as Fig. 9 As shown, the method includes:

[0187] Step S91, obtaining multimodal samples;

[0188] Step S92, using multimodal samples to perform comparative learning training on the initial retrieval model to obtain similarity calculation results of the multimodal samples, wherein the similarity calculation results are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the semantic matching degree of different modal samples in the multimodal samples;

[0189] Step S93: generating a target retrieval model based on the similarity calculation result, wherein the target retrieval model is used to perform data retrieval on the target object data to be retrieved to output a retrieval result.

[0190] In the embodiment of the present application, when training the target retrieval model, a multimodal sample is used to perform comparative learning on the initial retrieval model, that is, a large number of multimodal samples are used as training data, and a large number of multimodal samples are used to perform comparative learning on the initial retrieval model, thereby obtaining the target retrieval model. Among them, the multimodal sample may include multiple different modal pair samples, and any different modal pair samples are matched.

[0191] . Exemplarily, taking the different modal pair samples as image-text pair samples as an example, the image-text pair samples include matching text samples and image samples. Among them, the text sample is, for example, "a cat", and the image sample is, for example, an image of a cat, or, the text sample is, for example, "a dog", and the image sample is, for example, an image of a dog, which is not limited here. In addition, the different modal pair samples can also be text-audio pair samples, and the text-audio pair samples include matching text samples and audio samples, wherein the text sample is, for example, a section of lyrics, and the audio sample is, for example, the song audio corresponding to the lyrics, which is not limited here. It can be understood that the above description of different modal pair samples is only an example, and the different modal pair samples can be adjusted according to actual needs, which is not limited here.

[0192] During the comparative learning and training process of the initial retrieval model, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model. The similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples, that is, the probability distribution of samples of different modal pairs belonging to positive sample pairs.

[0193] In the contrastive learning training process, considering that processing the entire similarity matrix at one time will lead to an increase in the batch size, which will in turn increase the computational overhead of the contrast loss and occupy the video memory. Therefore, this application proposes to divide the entire similarity matrix into multiple small blocks according to batches and process them step by step, and calculate the batch size step by step, so as to reduce the occupation of video memory in the traditional contrast loss calculation process by calculating the intermediate variable similarity matrix in blocks, thereby effectively reducing the computational overhead and reducing the video memory occupation.

[0194] It can be seen that the present application uses multimodal samples to perform contrastive learning training on the initial retrieval model, and in the process of contrastive learning training, the batch size corresponding to the initial retrieval model is processed in blocks, and the similarity calculation results of the multimodal samples are obtained by step-by-step calculation of the batch size, that is, the semantic matching degree of different modal samples in the multimodal samples is obtained, so as to determine the loss function of the initial retrieval model according to the similarity calculation results, and then adjust the model parameters of the initial retrieval model according to the loss function to update and optimize the initial retrieval model, and finally train to obtain the target retrieval model. Therefore, the block calculation significantly reduces the occupancy of the video memory of a single node. Compared with the traditional contrastive learning method, there is no need to process the entire similarity matrix at the same time, which avoids the quadratic growth of the video memory demand with the batch size, so that under the same hardware conditions and under the premise of ensuring the calculation and operation efficiency, the batch size can be greatly increased to a scale that is difficult to achieve with the traditional contrastive learning method.

[0195] After the target retrieval model is trained, the target object data to be retrieved is obtained, the target object data to be retrieved is input into the target retrieval model, and the target retrieval model is used to perform data retrieval on the target object data, so that the retrieval results corresponding to the target object data can be accurately output.

[0196] For other descriptions of the target retrieval model, please refer to the descriptions of the target retrieval model in step S31 and step S32, which will not be repeated here.

[0197] The above-mentioned model training method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving data retrieval in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services and navigation services, for example: data retrieval scenarios for e-commerce services, data retrieval scenarios for educational services, data retrieval scenarios for medical services, etc., which are not limited here.

[0198] According to an embodiment of the present application, multimodal samples are obtained, and contrast learning training is performed on the initial retrieval model using the multimodal samples to obtain similarity calculation results of the multimodal samples, wherein the similarity calculation results are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples, and a target retrieval model is generated based on the similarity calculation results, wherein the target retrieval model is used to perform data retrieval on the target object data to be retrieved to output the retrieval results, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the computing and operating efficiency, the technical effect of being able to be applied to large-scale data sets is achieved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related technology.

[0199] According to an embodiment of the present application, a device embodiment for implementing the above data retrieval method is also provided. Fig.10 is a structural diagram of a data retrieval device according to an embodiment of the present application, such as Fig.10 As shown, the device comprises:

[0200] The first acquisition module 1001 is used to acquire target object data to be retrieved;

[0201] The first retrieval module 1002 is used to perform data retrieval on the target object data using a target retrieval model and output the retrieval results; wherein the target retrieval model is generated after contrast learning and training the initial retrieval model using multimodal samples, and during the contrast learning and training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0202] Optionally, the initial retrieval model includes: a text encoder and an image encoder, and the different modal samples include text samples and image samples. The device also includes: a processing module, which is used to: use a text encoder to perform text encoding on the text sample to obtain a first encoding result; use an image encoder to perform image encoding on the image sample to obtain a second encoding result; and perform step-by-step similarity calculation on the first encoding result and the second encoding result based on the processor core performance to generate a target retrieval model.

[0203] Optionally, the processing module is also used to perform step-by-step similarity calculation on the first encoding result and the second encoding result based on the processor core performance to obtain a contrast loss; update the model parameters of the initial retrieval model according to the contrast loss to generate a target retrieval model.

[0204] Optionally, the processing module is further used to determine a target block size based on processor core performance; perform step-by-step similarity calculation on the first encoding result and the second encoding result according to the target block size to obtain a comparison loss.

[0205] Optionally, the processing module is further configured to determine a first block size based on the number of cores of a single processor; or determine a second block size based on the number of cores of multiple processors; wherein the first block size is smaller than the second block size.

[0206] Optionally, the processing module is also used to obtain a similarity matrix between the first encoding result and the second encoding result; divide the similarity matrix into multiple first block matrices according to the first block size, wherein each block matrix in the multiple first block matrices includes: a first number of query samples and a second number of key value samples, and the first number and the second number are determined by the first block size; use a single processor to perform serial similarity calculation on the multiple first block matrices to obtain a comparison loss.

[0207] Optionally, the processing module is further used to perform batch processing on multiple first block matrices according to a preset number of blocks to obtain multiple batches of block matrices; and use a single processor to perform step-by-step similarity calculations on the multiple batches of block matrices in sequence to obtain contrast loss.

[0208] Optionally, the processing module is also used to use a single processor to perform step-by-step similarity calculations on multiple batches of block matrices in sequence to obtain a first calculation result; use a single processor to perform similarity calculations on positive samples in multimodal samples to obtain a second calculation result, wherein the positive samples are used to characterize that samples of different modalities are semantically related; and obtain a contrast loss based on the first calculation result and the second calculation result.

[0209] Optionally, the processing module is also used to use a single processor to perform intra-block similarity calculation on each block matrix in a single batch block matrix of multiple batch block matrices to obtain a third calculation result; use a single processor to perform cross-block cumulative calculation on the third calculation results corresponding to each block matrix in the single batch block matrix to obtain a fourth calculation result; use a single processor to perform batch-by-batch cumulative calculation on the fourth calculation results corresponding to each batch block matrix in the multiple batch block matrices to obtain the first calculation result.

[0210] Optionally, the processing module is further configured to perform intra-block similarity calculation on each block matrix in a single batch of block matrices of the multiple batches of block matrices in the cache of a single processor to obtain a third calculation result.

[0211] Optionally, the processing module is also used to obtain a similarity matrix between the first encoding result and the second encoding result; divide the similarity matrix into multiple second block matrices according to the second block size, wherein each block matrix in the multiple second block matrices includes: a third number of query samples and a fourth number of key value samples, and the third number and the fourth number are determined by the second block size; use multiple processors to perform parallel similarity calculations on the multiple second block matrices to obtain a comparison loss.

[0212] According to the embodiment of the present application, by acquiring the target object data to be retrieved, the target object data to be retrieved is input into the target retrieval model, and the target retrieval model is used to perform data retrieval on the target object data, so that the retrieval result corresponding to the target object data can be accurately output, and the target retrieval model is generated after contrast learning training of the initial retrieval model using multimodal samples, and in the contrast learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, wherein the similarity calculation results are used to characterize the degree of semantic matching of different modal samples in the multimodal samples, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the calculation and operation efficiency, the technical effect of being able to be applied to large-scale data sets is solved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related technology.

[0213] It should be noted that the first acquisition module 1001 and the first search module 1002 correspond to step S31 and step S32 in the embodiment, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiment.

[0214] According to an embodiment of the present application, another device embodiment for implementing the above data retrieval method is also provided. Fig.11 is a schematic diagram of the structure of another data retrieval device according to an embodiment of the present application. Figure 1 As shown, the device comprises:

[0215] The second acquisition module 1101 is used to acquire the product image data to be retrieved;

[0216] The second retrieval module 1102 is used to use the target image-text retrieval model to perform data retrieval on product image data and output product image retrieval results; wherein the target image-text retrieval model is generated after comparative learning and training of the initial image-text retrieval model using multimodal samples, and during the comparative learning and training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial image-text retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0217] According to the embodiment of the present application, by obtaining the product image data to be retrieved, the product image data to be retrieved is input into the target image-text retrieval model, and the target image-text retrieval model is used to perform data retrieval on the product image data, so that the product image retrieval result corresponding to the product image data can be accurately output, and the target image-text retrieval model is generated after contrast learning training of the initial image-text retrieval model using multimodal samples, and in the contrast learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial image-text retrieval model, wherein the similarity calculation results are used to characterize the degree of semantic matching of different modal samples in the multimodal samples, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the computing and operating efficiency, the technical effect of being able to be applied to large-scale data sets is solved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related technology.

[0218] It should be noted that the second acquisition module 1101 and the second retrieval module 1102 correspond to step S61 and step S62 in the embodiment, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiment.

[0219] According to an embodiment of the present application, another device embodiment for implementing the above data retrieval method is also provided. Fig.12 is a structural diagram of another data retrieval device according to an embodiment of the present application, such as Fig.12 As shown, the device comprises:

[0220] The third acquisition module 1201 is used to acquire a data retrieval request through the first application programming interface, wherein the request data carried in the data retrieval request includes: target object data to be retrieved;

[0221] The third retrieval module 1202 is used to return a data retrieval response through the second application programming interface, wherein the response data carried in the data retrieval response includes: a retrieval result, which is generated after data retrieval of the target object data using a target retrieval model, and the target retrieval model is generated after comparative learning and training of an initial retrieval model using multimodal samples, and during the comparative learning and training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0222] According to an embodiment of the present application, a data retrieval request is obtained through a first application programming interface, wherein the request data carried in the data retrieval request includes: target object data to be retrieved, and a data retrieval response is returned through a second application programming interface, wherein the response data carried in the data retrieval response includes: retrieval results, wherein the retrieval results are generated after data retrieval of the target object data using a target retrieval model, and the target retrieval model is generated after contrast learning training of an initial retrieval model using multimodal samples, and during the contrast learning training process, similarity calculation results of multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the semantic matching degree of different modal samples in the multimodal samples, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the computing and operating efficiency, the technical effect of being able to be applied to large-scale data sets is solved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related technology.

[0223] It should be noted that the third acquisition module 1201 and the third retrieval module 1202 correspond to step S71 and step S72 in the embodiment, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiment.

[0224] According to an embodiment of the present application, another device embodiment for implementing the above data retrieval method is also provided. Fig.13 is a structural diagram of another data retrieval device according to an embodiment of the present application, such as Fig.13 As shown, the device comprises:

[0225] The fourth acquisition module 1301 is used to acquire the currently input data retrieval dialogue request, wherein the request data carried in the data retrieval dialogue request includes: target object data to be retrieved;

[0226] The return module 1302 is used to respond to the data retrieval dialogue request and return a data retrieval dialogue reply, wherein the information carried in the data retrieval dialogue reply includes: a retrieval result, which is generated after the target object data is retrieved using the target retrieval model, and the target retrieval model is generated after the initial retrieval model is trained by contrastive learning using multimodal samples, and in the contrastive learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the semantic matching degree of different modal samples in the multimodal samples;

[0227] The display module 1303 is used to display the search results in a graphical user interface.

[0228] According to an embodiment of the present application, by obtaining a currently input data retrieval dialogue request, wherein the request data carried in the data retrieval dialogue request includes: target object data to be retrieved, and then in response to the data retrieval dialogue request, a data retrieval dialogue reply is returned, wherein the information carried in the data retrieval dialogue reply includes: a retrieval result, wherein the retrieval result is generated after data retrieval of the target object data using a target retrieval model, and the target retrieval model is generated after contrast learning training of an initial retrieval model using multimodal samples, and in the contrast learning training process, the similarity calculation result of the multimodal sample is calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation result is used to characterize the semantic matching degree of different modal samples in the multimodal sample, and finally based on the obtained retrieval result, the retrieval result is displayed in a graphical user interface, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving the effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the computing and operating efficiency, the technical effect of being able to be applied to large-scale data sets is solved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related art.

[0229] It should be noted that the fourth acquisition module 1301, the return module 1302 and the display module 1303 correspond to steps S81 to S83 in the embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiment.

[0230] According to an embodiment of the present application, another device embodiment for implementing the above-mentioned model training method is also provided. Fig.14 is a structural diagram of another model training device according to an embodiment of the present application, such as Fig.14 As shown, the device comprises:

[0231] The fifth acquisition module 1401 is used to acquire multimodal samples;

[0232] The training module 1402 is used to perform comparative learning training on the initial retrieval model using multimodal samples to obtain similarity calculation results of the multimodal samples, wherein the similarity calculation results are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the semantic matching degree of different modal samples in the multimodal samples;

[0233] The generating module 1403 is used to generate a target retrieval model based on the similarity calculation result, wherein the target retrieval model is used to perform data retrieval on the target object data to be retrieved to output the retrieval result.

[0234] According to an embodiment of the present application, multimodal samples are obtained, and contrast learning training is performed on the initial retrieval model using the multimodal samples to obtain similarity calculation results of the multimodal samples, wherein the similarity calculation results are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples, and a target retrieval model is generated based on the similarity calculation results, wherein the target retrieval model is used to perform data retrieval on the target object data to be retrieved to output the retrieval results, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the computing and operating efficiency, the technical effect of being able to be applied to large-scale data sets is achieved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related technology.

[0235] It should be noted that the fifth acquisition module 1401, the training module 1402 and the generation module 1403 correspond to steps S91 to S93 in the embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiment.

[0236] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the embodiments.

[0237] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal. Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices in a computer network.

[0238] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the data retrieval method: obtaining the target object data to be retrieved; using the target retrieval model to perform data retrieval on the target object data, and outputting the retrieval results; wherein the target retrieval model is generated after contrast learning and training the initial retrieval model using multimodal samples, and during the contrast learning and training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0239] Optionally, the electronic device can execute the program code of the following steps in the model training method: obtaining multimodal samples; using the multimodal samples to perform comparative learning training on the initial retrieval model to obtain similarity calculation results of the multimodal samples, wherein the similarity calculation results are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between different modal samples in the multimodal samples; generating a target retrieval model based on the similarity calculation results, wherein the target retrieval model is used to perform data retrieval on the target object data to be retrieved to output the retrieval results.

[0240] Optionally, Fig.15 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.15 As shown, taking electronic device A as an example, the electronic device A may include: one or more (only one is shown in the figure) processors 1502, a memory 1504, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0241] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data retrieval method and device in the embodiment of the present application, or the program instructions / modules corresponding to the model training method and device. The processor executes various functional applications and data processing by running the stored software programs and modules, that is, realizing the above-mentioned data retrieval method or model training method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the electronic device A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0242] According to the embodiment of the present application, by acquiring the target object data to be retrieved, the target object data to be retrieved is input into the target retrieval model, and the target retrieval model is used to perform data retrieval on the target object data, so that the retrieval result corresponding to the target object data can be accurately output, and the target retrieval model is generated after contrast learning training of the initial retrieval model using multimodal samples, and in the contrast learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, wherein the similarity calculation results are used to characterize the degree of semantic matching of different modal samples in the multimodal samples, thereby achieving the purpose of effectively reducing the video memory occupancy and computing overhead of the retrieval model, thereby achieving effective reduction of the video memory occupancy and computing overhead of the retrieval model, so that the batch size of the contrast loss is no longer limited, and under the premise of ensuring the calculation and operation efficiency, the technical effect of being able to be applied to large-scale data sets is solved, thereby solving the technical problems of limited batch size, high computing resource requirements and high computing overhead of the contrast learning method in the related technology.

[0243] It can be understood by those skilled in the art that Fig.15 The structure shown is for illustration only, and the electronic device A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.15 It does not limit the structure of the above electronic device. For example, the electronic device A may also include Fig.15 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.15 Different configurations are shown.

[0244] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0245] The embodiment of the present application also provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the data retrieval method or model training method provided in the first embodiment.

[0246] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0247] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining target object data to be retrieved; performing data retrieval on the target object data using a target retrieval model, and outputting retrieval results; wherein the target retrieval model is generated after performing comparative learning training on an initial retrieval model using multimodal samples, and during the comparative learning training process, similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0248] Optionally, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining a multimodal sample; performing comparative learning training on an initial retrieval model using the multimodal sample to obtain a similarity calculation result of the multimodal sample, wherein the similarity calculation result is calculated step by step based on a batch size corresponding to the initial retrieval model, and the similarity calculation result is used to characterize the degree of semantic matching between samples of different modalities in the multimodal sample; generating a target retrieval model based on the similarity calculation result, wherein the target retrieval model is used to perform data retrieval on target object data to be retrieved to output a retrieval result.

[0249] An embodiment of the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements any one of the above-mentioned data retrieval methods or model training methods.

[0250] Optionally, in this embodiment, the computer program product is used to perform the following steps when it is executed by the processor: obtaining target object data to be retrieved; performing data retrieval on the target object data using a target retrieval model, and outputting retrieval results; wherein the target retrieval model is generated after performing comparative learning training on an initial retrieval model using multimodal samples, and during the comparative learning training process, similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

[0251] Optionally, in this embodiment, the computer program product is used to perform the following steps when the processor executes the program code: obtaining multimodal samples; using the multimodal samples to perform comparative learning training on the initial retrieval model to obtain similarity calculation results of the multimodal samples, wherein the similarity calculation results are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between different modal samples in the multimodal samples; generating a target retrieval model based on the similarity calculation results, wherein the target retrieval model is used to perform data retrieval on the target object data to be retrieved to output the retrieval results.

[0252] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0253] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0254] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0255] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0256] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0257] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0258] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data retrieval method, characterized in that: include: Obtain the target object data to be retrieved; Using the target retrieval model to perform data retrieval on the target object data, and outputting the retrieval results; Among them, the target retrieval model is generated after contrastive learning training of the initial retrieval model using multimodal samples, and during the contrastive learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between different modal samples in the multimodal samples.

2. The data retrieval method according to claim 1, characterized in that: The initial retrieval model includes: a text encoder and an image encoder, the different modality samples include text samples and image samples, and the data retrieval method further includes: Using the text encoder to perform text encoding on the text sample to obtain a first encoding result; Using the image encoder to perform image encoding on the image sample to obtain a second encoding result; Based on the processor core performance, the first encoding result and the second encoding result are subjected to step-by-step similarity calculation to generate the target retrieval model.

3. The data retrieval method according to claim 2, characterized in that: Performing step-by-step similarity calculation on the first encoding result and the second encoding result based on the processor core performance to generate the target retrieval model includes: Performing step-by-step similarity calculation on the first encoding result and the second encoding result based on the processor core performance to obtain a comparison loss; The model parameters of the initial retrieval model are updated according to the contrast loss to generate the target retrieval model.

4. The data retrieval method according to claim 3, characterized in that: The first encoding result and the second encoding result are calculated in steps based on the processor core performance to obtain the comparison loss, which includes: Determining a target block size based on the processor core performance; The first encoding result and the second encoding result are subjected to step-by-step similarity calculation according to the target block size to obtain the contrast loss.

5. The data retrieval method according to claim 4, characterized in that: Determining the target block size based on the processor core performance includes: Determine the first block size based on the number of cores of a single processor; or, Determining a second block size based on the number of cores of the plurality of processors; The first block size is smaller than the second block size.

6. The data retrieval method according to claim 5, characterized in that: The step-by-step similarity calculation of the first encoding result and the second encoding result is performed according to the target block size to obtain the comparison loss, which includes: Obtaining a similarity matrix between the first encoding result and the second encoding result; Dividing the similarity matrix into a plurality of first block matrices according to the first block size, wherein each of the plurality of first block matrices includes: a first number of query samples and a second number of key value samples, wherein the first number and the second number are determined by the first block size; The single processor is used to perform serial similarity calculation on the multiple first block matrices to obtain the contrast loss.

7. The data retrieval method according to claim 6, characterized in that: Using the single processor to perform step-by-step similarity calculation on the multiple first block matrices to obtain the contrast loss includes: Processing the plurality of first block matrices in batches according to a preset number of blocks to obtain a plurality of batches of block matrices; The single processor is used to perform step-by-step similarity calculations on the multiple batches of block matrices in sequence to obtain the contrast loss.

8. The data retrieval method according to claim 7, characterized in that: The single processor is used to sequentially perform step-by-step similarity calculations on the multiple batches of block matrices to obtain the contrast loss, which includes: Using the single processor to perform step-by-step similarity calculations on the multiple batches of block matrices in sequence to obtain a first calculation result; Using the single processor to perform similarity calculation on positive samples in the multimodal samples to obtain a second calculation result, wherein the positive samples are used to characterize that the image sample is semantically related to the text sample; The contrast loss is obtained based on the first calculation result and the second calculation result.

9. The data retrieval method according to claim 8, characterized in that: Using the single processor to sequentially perform step-by-step similarity calculations on the multiple batches of block matrices to obtain the first calculation results includes: Using the single processor, perform intra-block similarity calculation on each block matrix in a single batch of block matrices of the multiple batches of block matrices to obtain a third calculation result; Using the single processor, perform cross-block accumulation calculation on the third calculation results respectively corresponding to each block matrix in the single batch of block matrices to obtain a fourth calculation result; The single processor is used to perform batch-by-batch cumulative calculation on the fourth calculation results corresponding to each batch of block matrices in the multiple batches of block matrices to obtain the first calculation result.

10. The data retrieval method according to claim 9, characterized in that: Using the single processor to perform intra-block similarity calculation on each block matrix in a single batch of block matrices of the multiple batches of block matrices to obtain the third calculation result includes: In the cache of the single processor, intra-block similarity calculation is performed on each block matrix in a single batch of block matrices of the multiple batches of block matrices to obtain the third calculation result.

11. The data retrieval method according to claim 5, characterized in that: The step-by-step similarity calculation of the first encoding result and the second encoding result is performed according to the target block size to obtain the comparison loss, which includes: Obtaining a similarity matrix between the first encoding result and the second encoding result; Dividing the similarity matrix into a plurality of second block matrices according to the second block size, wherein each of the plurality of second block matrices includes: a third number of query samples and a fourth number of key value samples, wherein the third number and the fourth number are determined by the second block size; The plurality of processors are used to perform parallel similarity calculations on the plurality of second block matrices to obtain the contrast loss.

12. A data retrieval method, characterized in that: include: Get the product image data to be retrieved; Using the target image-text retrieval model to perform data retrieval on the product image data, and outputting the product image retrieval result; Among them, the target image-text retrieval model is generated after contrastive learning training of the initial image-text retrieval model using multimodal samples, and during the contrastive learning training process, the similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial image-text retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

13. A data retrieval method, characterized in that: include: Obtaining a data retrieval request through a first application programming interface, wherein the request data carried in the data retrieval request includes: target object data to be retrieved; A data retrieval response is returned through a second application programming interface, wherein the response data carried in the data retrieval response includes: a retrieval result, wherein the retrieval result is generated after data retrieval of the target object data using a target retrieval model, wherein the target retrieval model is generated after contrast learning training of an initial retrieval model using multimodal samples, and during the contrast learning training process, similarity calculation results of the multimodal samples are calculated step by step based on a batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the degree of semantic matching between samples of different modalities in the multimodal samples.

14. A data retrieval method, characterized in that: include: Acquire the currently input data retrieval dialogue request, wherein the request data carried in the data retrieval dialogue request includes: target object data to be retrieved; In response to the data retrieval dialogue request, a data retrieval dialogue reply is returned, wherein the information carried in the data retrieval dialogue reply includes: a retrieval result, wherein the retrieval result is generated after data retrieval of the target object data using a target retrieval model, wherein the target retrieval model is generated after contrast learning training of an initial retrieval model using multimodal samples, and during the contrast learning training process, similarity calculation results of the multimodal samples are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the semantic matching degree of different modality samples in the multimodal samples; The search results are displayed in a graphical user interface.

15. A model training method, characterized in that: include: Obtain multimodal samples; Using the multimodal samples to perform comparative learning training on the initial retrieval model, obtaining similarity calculation results of the multimodal samples, wherein the similarity calculation results are calculated step by step based on the batch size corresponding to the initial retrieval model, and the similarity calculation results are used to characterize the semantic matching degree of different modal samples in the multimodal samples; A target retrieval model is generated based on the similarity calculation result, wherein the target retrieval model is used to perform data retrieval on the target object data to be retrieved to output a retrieval result.

16. An electronic device, characterized in that: include: A memory storing an executable program; A processor, used to run the program, wherein when the program is run, it executes the data retrieval method described in any one of claims 1 to 14 or executes the model training method described in claim 15.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein, when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the data retrieval method described in any one of claims 1 to 14 or the model training method described in claim 15.

18. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the data retrieval method described in any one of claims 1 to 14 or executes the model training method described in claim 15.

Images