Multi-modal encoder training method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The embodiment of the invention provides a multi-modal encoder training method and device, electronic equipment, a computer readable storage medium and a computer program product, and the method comprises the steps: inputting multi-modal data in a current training set of a current task into a to-be-trained multi-modal encoder, obtaining at least one multi-modal feature corresponding to the multi-modal data; according to the at least one multi-modal feature, determining a current topological structure of the current training set under the current task; calling a historical multi-mode encoder, and performing feature extraction on historical data in the historical training set to obtain a historical topological structure; determining a loss result of the current task based on the historical topological structure and the current topological structure; and correcting model parameters of the to-be-trained multi-modal encoder based on the loss result to obtain a trained multi-modal encoder. By means of the method and device, the model training effect on the current training set can be improved, and meanwhile the model training effect on the historical data set can be kept.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and relate to, but are not limited to, a training method, device, electronic device, computer-readable storage medium, and computer program product for a multimodal encoder. Background Art

[0002] With the rapid development of artificial intelligence technology, the requirements for the accuracy of artificial intelligence models are becoming increasingly higher. In the process of using artificial intelligence models, new application scenario data will continue to emerge, so artificial intelligence models need to be continuously optimized.

[0003] In related technologies, when using new data to optimize artificial intelligence models, due to the artificial intelligence models' lack of ability to accumulate knowledge, the effect will improve on new data, but decrease on historical data. Summary of the Invention

[0004] The embodiments of the present application provide a training method, device, electronic device, computer-readable storage medium, and computer program product for a multimodal encoder, which can be applied at least in the field of artificial intelligence and can improve the model training effect on the current training set while maintaining the model training effect on the historical data set.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a training method for a multimodal encoder, comprising: inputting multimodal data in a current training set of a current task into a multimodal encoder to be trained, and obtaining at least one multimodal feature corresponding to the multimodal data; determining a current topological structure of the current training set under the current task based on the at least one multimodal feature; calling a historical multimodal encoder to extract features from historical data in the historical training set to obtain a historical topological structure; wherein the historical multimodal encoder is obtained by training the multimodal encoder to be trained based on the historical training set; determining a loss result for the current task based on the historical topological structure and the current topological structure; and correcting model parameters of the multimodal encoder to be trained based on the loss result to obtain a trained multimodal encoder.

[0007] An embodiment of the present application provides a training device for a multimodal encoder, comprising: a multimodal feature acquisition module, used to input the multimodal data in the current training set of the current task into the multimodal encoder to be trained, and obtain at least one multimodal feature corresponding to the multimodal data; a current topology structure determination module, used to determine the current topology structure of the current training set under the current task based on the at least one multimodal feature; a historical topology structure determination module, used to call the historical multimodal encoder, perform feature extraction on the historical data in the historical training set, and obtain a historical topology structure; wherein the historical multimodal encoder is obtained by training the multimodal encoder to be trained based on the historical training set; a loss result determination module, used to determine the loss result of the current task based on the historical topology structure and the current topology structure; and a model parameter correction module, used to correct the model parameters of the multimodal encoder to be trained based on the loss result, and obtain a trained multimodal encoder.

[0008] In some embodiments, the current training set includes a first modal data set and a second modal data set having a preset corresponding relationship; the multimodal feature acquisition module is further used to: input the first modal data set and the second modal data set into the multimodal encoder to be trained to obtain at least one first feature vector corresponding to the first modal data set and at least one second feature vector corresponding to the second modal data set; and the first feature vector and the second feature vector having the preset corresponding relationship constitute a multimodal feature.

[0009] In some embodiments, the preset correspondence includes: a positive correlation between the first modal data in the first modal data set and the second modal data in the second modal data set, and a negative correlation between the first modal data in the first modal data set and the second modal data in the second modal data set; the current topology structure determination module is further used to: perform data pairing on the first modal data in the first modal data set and the second modal data in the second modal data set according to the positive correlation and the negative correlation to obtain positive sample pairs and negative sample pairs; determine the current cosine similarity of the positive sample pair according to the first eigenvector of the first modal data and the second eigenvector of the second modal data in the positive sample pair; determine the current cosine similarity of the negative sample pair according to the first eigenvector of the first modal data and the second eigenvector of the second modal data in the negative sample pair; and construct the current topology structure of the current training set under the current task based on the current cosine similarities of the positive sample pair and the negative sample pair.

[0010] In some embodiments, the current topology structure determination module is further used to: construct a similarity matrix based on the current cosine similarities of the positive sample pairs and the negative sample pairs; and convert the similarity matrix into the current topology structure under the current task.

[0011] In some embodiments, the device further includes a similarity matrix construction module for creating an all-zero matrix; traversing the positive sample pairs and the negative sample pairs; and filling the current cosine similarities corresponding to the positive sample pairs and the negative sample pairs into the all-zero matrix respectively to obtain the similarity matrix.

[0012] In some embodiments, the current topology structure determination module is further used to: obtain the dimension of the similarity matrix; create a topology map based on the dimension; wherein the topology map includes multiple nodes and connecting lines between every two nodes, and the number of the multiple nodes is equal to the sum of the number of first modality data in the first modality data set and the number of second modality data in the second modality data set; each node corresponds to one first modality data or one second modality data; traverse the similarity matrix to determine the cosine similarity with a value of zero in the similarity matrix; based on the first modality data and the second modality data corresponding to each cosine similarity with a value of zero, determine the corresponding two nodes in the topology map, and the connecting line between the two nodes; hide the connecting line between the two nodes to obtain the current topology structure under the current task.

[0013] In some embodiments, the historical data in the historical training set includes third modality data and fourth modality data having a preset corresponding relationship; the historical topology structure determination module is further used to: call the historical multimodal encoder to perform feature extraction on the historical data in the historical training set to obtain at least one third feature corresponding to the third modality data and at least one fourth feature corresponding to the fourth modality data; wherein each third feature has the preset corresponding relationship with a fourth feature, and the third feature and the fourth feature having the preset corresponding relationship constitute a historical positive sample pair, and the third feature and the fourth feature not having the preset corresponding relationship constitute a historical negative sample pair; based on the historical positive sample pair and the historical negative sample pair, the historical topology structure is determined.

[0014] In some embodiments, the historical topology structure determination module is further used to: calculate the historical cosine similarities of the historical positive sample pairs and the historical negative sample pairs respectively; construct a historical similarity matrix based on the historical cosine similarities; and convert the historical similarity matrix into the historical topology structure.

[0015] In some embodiments, the loss result determination module is also used to: compare the multimodal data in the current training set with the historical data in the historical training set to obtain a first type of data in the multimodal data that is repeated with the historical data and a second type of data that is not repeated with the historical data; based on the first type of data, the historical topological structure and the current topological structure, perform a first loss calculation on the current task to obtain a first loss result; based on the second type of data, the historical topological structure and the current topological structure, perform a second loss calculation on the current task to obtain a second loss result; wherein, the first loss result and the second loss result constitute the loss result of the current task.

[0016] In some embodiments, the loss result determination module is also used to: obtain the historical cosine similarity of the first category of data in the historical topological structure and the current cosine similarity of the first category of data in the current topological structure; based on the historical cosine similarity of the first category of data in the historical topological structure and the current cosine similarity of the first category of data in the current topological structure, perform loss calculation on the current task to obtain the first loss result.

[0017] In some embodiments, the loss result determination module is further used to: obtain the current cosine similarity of the second category data; perform loss calculation on the current task based on the current cosine similarity of the second category data in the current topological structure, and determine the second loss result.

[0018] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned multimodal encoder training method when executing the executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned multimodal encoder training method is implemented.

[0020] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned multimodal encoder training method.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] When training a multimodal encoder, first, the multimodal data in the current training set of the current task is input into the multimodal encoder to be trained to obtain at least one multimodal feature corresponding to the multimodal data; then, based on the at least one multimodal feature, the current topological structure of the current training set under the current task is determined; then, the historical multimodal encoder is called to extract features from the historical data in the historical training set to obtain a historical topological structure; wherein the historical multimodal encoder is obtained after training the multimodal encoder to be trained based on the historical training set; finally, based on the historical topological structure and the current topological structure, the loss result of the current task is determined, and the model parameters of the multimodal encoder to be trained are corrected based on the loss result to obtain the trained multimodal encoder. It can be seen that the training method of the multimodal encoder of the embodiment of the present application adjusts the current topological structure through the historical topological structure, so that the data in the historical topological structure can be used when training the multimodal encoder of the current task, thereby being able to improve the model training effect on the current training set while maintaining the model training effect on the historical data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of an optional architecture of a multimodal encoder training system provided in an embodiment of the present application;

[0024] Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0025] Figure 3 This is an optional flowchart of the multimodal encoder training method provided in an embodiment of the present application;

[0026] Figure 4 This is another optional flowchart of the multimodal encoder training method provided in an embodiment of the present application;

[0027] Figure 5 This is a schematic diagram of a process for determining the current topology structure provided by an embodiment of the present application;

[0028] Figure 6 This is a schematic diagram of a process for converting a similarity matrix into a current topological structure according to an embodiment of the present application;

[0029] Figure 7 This is a schematic diagram of the implementation process of converting the similarity matrix into the current topology structure provided by the real-time stream of this application;

[0030] Figure 8 This is a schematic diagram of the implementation process of determining the loss result of the current task provided by an embodiment of the present application;

[0031] Figure 9This is a flow chart of the inference of the multimodal pre-training method provided in the embodiment of the present application;

[0032] Figure 10 This is a schematic diagram of the implementation flow of the multimodal encoder training method provided in an embodiment of the present application in an actual scenario;

[0033] Figure 11 This is another implementation flowchart of the multimodal encoder training method provided in an embodiment of the present application in a practical scenario. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0035] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it will be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the art to which the embodiments of this application pertain. The terms used in the embodiments of this application are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0036] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0037] Before describing the training method of the multimodal encoder provided in the embodiment of the present application, the professional terms involved in the embodiment of the present application are first explained:

[0038] 1) Multimodal pre-training: Pre-training the multimodal encoder based on the first modal data and the second modal data with a preset correspondence.

[0039] 2) Correlation: The correlation or connection between different modal data.

[0040] 3) Self-attention mechanism: A neural network model that can automatically learn the relationship between each element in the input sequence and other elements and weight them according to the importance of the relationship.

[0041] In order to better understand the training method of the multimodal encoder provided in the embodiment of the present application, the training method of the multimodal encoder in the related art is first described below.

[0042] In related technologies, a multimodal autoencoder can be used to process multimodal data. The autoencoder includes an encoder and a decoder. The autoencoder learns the low-dimensional representation of the data by minimizing the reconstruction error. In the multimodal autoencoder, the encoder and decoder process data of different modalities respectively to learn the correlation between data of different modalities.

[0043] In related technologies, multimodal generative adversarial networks (MMGANs) can also be used to process multimodal data. MMGANs combine generative adversarial networks (GANs) with multimodal data to generate models for multimodal data. MMGANs consist of a generator and a discriminator. The generator is responsible for generating multimodal data, while the discriminator attempts to distinguish between real multimodal data and generated multimodal data. Through adversarial training, the generator can learn the characteristics of multimodal data.

[0044] In related technologies, a multimodal transformer can also be used to process multimodal data. The multimodal transformer is based on the transformer model and is used to process sequence modeling and association learning of multimodal data. The transformer model captures the associations in the sequence through the self-attention mechanism and extends it to the processing of multimodal data. The multimodal transformer can process inputs of different modalities simultaneously and learn the associations and contextual information between different modalities.

[0045] However, the multimodal encoder training methods described in the related art suffer from at least the following issues: In multimodal model usage scenarios, new application scenario data constantly emerges, necessitating continuous optimization of the multimodal model. When optimizing the multimodal model using new application scenario data, the multimodal model lacks the ability to accumulate knowledge, resulting in improved performance on new application scenario data but decreased performance on historical data.

[0046] Based on the problems existing in the related art, an embodiment of the present application provides a training method for a multimodal encoder. The method adjusts the current topological structure of the current task by constructing a historical topological structure based on a historical data set, so that the data in the historical topological structure can be used when training the multimodal encoder of the current task.

[0047] Specifically, in the training method of the multimodal encoder provided in the embodiment of the present application, first, the multimodal data in the current training set of the current task is input into the multimodal encoder to be trained to obtain at least one multimodal feature corresponding to the multimodal data; then, based on the at least one multimodal feature, the current topological structure of the current training set under the current task is determined; then, the historical multimodal encoder is called to extract features from the historical data in the historical training set to obtain a historical topological structure; wherein the historical multimodal encoder is obtained after training the multimodal encoder to be trained based on the historical training set; finally, based on the historical topological structure and the current topological structure, the loss result of the current task is determined, and the model parameters of the multimodal encoder to be trained are corrected based on the loss result to obtain the trained multimodal encoder. It can be seen that the training method of the multimodal encoder of the embodiment of the present application adjusts the current topological structure through the historical topological structure, so that the data in the historical topological structure can be used when training the multimodal encoder of the current task, thereby improving the knowledge retention ability of the multimodal encoder for data at different times, and can maintain the model training effect on the historical data set while improving the model training effect on the current training set.

[0048] Here, first, an exemplary application of the training device of the multimodal encoder of the embodiment of the present application is described, and the training device of the multimodal encoder is an electronic device for implementing the training method of the multimodal encoder. In one implementation, the training device of the multimodal encoder provided in the embodiment of the present application (i.e., electronic device) can be implemented as a terminal or a server. In one implementation, the device provided in the embodiment of the present application can be implemented as any terminal with a training function of a multimodal encoder, such as a laptop computer, a tablet computer, a desktop computer, an intelligent robot, an intelligent home appliance, and an intelligent vehicle-mounted device; in another implementation, the training device of the multimodal encoder provided in the embodiment of the present application can also be implemented as a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application. Next, an exemplary application in which the training device for the multimodal encoder is implemented as a server will be described.

[0049] See also Figure 1 , Figure 1 This is an optional architectural diagram of a multimodal encoder training system provided in an embodiment of the present application. To implement training of a multimodal encoder, a multimodal encoder training platform can be provided. The multimodal encoder training platform is deployed in a server, and the multimodal data in the current training set of the current task is input into the multimodal encoder training platform, thereby implementing the multimodal encoder training method of an embodiment of the present application.

[0050] The multimodal encoder training system 10 of the embodiment of the present application includes at least a terminal 100, a network 200 and a server 300. A multimodal encoder training application is deployed on the terminal 100, and the server 300 can be a background server of the multimodal encoder training application. The terminal 100 can constitute the multimodal encoder training device of the embodiment of the present application, that is, the multimodal encoder training method of the embodiment of the present application is implemented through the terminal 100. The terminal 100 is connected to the server 300 via the network 200. The network 200 can be a wide area network or a local area network, or a combination of the two.

[0051] See also Figure 1, the terminal 100 receives the user's multimodal encoder training operation, and generates a multimodal encoder training request in response to the multimodal encoder training operation; then, the terminal 100 sends the multimodal encoder training request to the server 300 through the network 200. After receiving the multimodal encoder training request, the server 300 responds to the multimodal encoder training request, obtains the multimodal data in the current training set, and inputs the multimodal data into the multimodal encoder to obtain at least one multimodal feature corresponding to the multimodal data; then, the server 300 determines the current topology of the current training set based on the at least one multimodal feature; and the server 300 calls the historical multimodal encoder to perform feature extraction on the historical data in the historical training set to obtain the historical topology; the server 300 determines the loss result of the current task based on the current topology and the historical topology, and the server 300 corrects the model parameters of the multimodal encoder to be trained based on the loss result to obtain the trained multimodal encoder and generates a message indicating that the multimodal encoder training is completed; finally, the server 300 sends the message indicating that the multimodal encoder training is completed to the terminal 100 via the network 200, and displays the message indicating that the multimodal encoder training is completed on the display interface of the terminal 100.

[0052] In some embodiments, the above-mentioned multimodal encoder training method can also be executed by the terminal, that is, after receiving the user's multimodal encoder training operation, the terminal 100 can respond to the multimodal encoder training operation, and the terminal 100 obtains the multimodal data in the current training set from the server 300, inputs the multimodal data into the multimodal encoder, and obtains at least one multimodal feature corresponding to the multimodal data; then, the terminal 100 determines the current topological structure of the current training set based on the at least one multimodal feature; the terminal 100 calls the historical multimodal encoder in the server 300, performs feature extraction on the historical data in the historical training set, and obtains the historical topological structure; the terminal 100 determines the loss result of the current task based on the current topological structure and the historical topological structure, and the terminal 100 corrects the model parameters of the multimodal encoder to be trained based on the loss result to obtain the trained multimodal encoder.

[0053] Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application, Figure 2The electronic device shown may be a training device for a multimodal encoder, and the training device for the multimodal encoder includes: at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. The various components in the training device for the multimodal encoder are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not shown in FIG. Figure 2 Various buses are labeled as bus system 240 .

[0054] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] The user interface 230 includes one or more output devices 231 that enable presentation of media content, and one or more input devices 232 .

[0056] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, and the like. The memory 250 may optionally include one or more storage devices physically located away from the processor 210. The memory 250 may include volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory. In some embodiments, the memory 250 may store data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.

[0057] The operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks; a network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220. Exemplary network interfaces 220 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.; an input processing module 253 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0058] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 A multimodal encoder training device 254 stored in memory 250 is shown. This multimodal encoder training device 254 can be a multimodal encoder training device in an electronic device. It can be software in the form of a program or plug-in, and includes the following software modules: a multimodal feature acquisition module 2541, a current topology determination module 2542, a historical topology determination module 2543, a loss result determination module 2544, and a model parameter correction module 2545. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0059] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the multimodal encoder provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0060] The training method of the multimodal encoder provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the training method of the multimodal encoder provided in each embodiment of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal.

[0061] Figure 3 This is an optional flow chart of the training method of the multimodal encoder provided in the embodiment of the present application. Figure 3 The steps shown are explained as Figure 3 As shown, the execution subject of the training method of the multimodal encoder is taken as an example to illustrate that the method includes the following steps S101 to S105:

[0062] Step S101: input the multimodal data in the current training set of the current task into the multimodal encoder to be trained to obtain at least one multimodal feature corresponding to the multimodal data.

[0063] In some embodiments, the current task represents an ongoing machine learning task or deep learning task; the current training set refers to the data set currently used to train the model, and the current training set contains multimodal data and corresponding labels; multimodal data refers to data containing a variety of different types of information, such as text data, image data, audio data, etc. A multimodal encoder is a model or algorithm that can process and extract features of multimodal data. The multimodal encoder can accept multiple types of data inputs and convert multiple types of data into one or more unified feature representations to perform subsequent tasks, such as classification, regression, and other tasks. Multimodal features are feature representations of multimodal data obtained after the multimodal data is processed by the multimodal encoder. These features may be one or more vectors used to characterize important information of the original multimodal data.

[0064] Step S102: determining a current topological structure of a current training set under a current task based on at least one multimodal feature.

[0065] In some embodiments, the topological structure refers to the connection mode and relationship between nodes in a data structure such as a network or a graph; the current topological structure refers to the connection relationship between the multimodal features extracted under the current training set.

[0066] Step S103 , calling a historical multimodal encoder to perform feature extraction on historical data in a historical training set to obtain a historical topological structure; wherein the historical multimodal encoder is obtained by training a multimodal encoder to be trained based on the historical training set.

[0067] In some embodiments, the historical training set refers to a data set collected within a certain period of time in the past and used to train a historical multimodal encoder; the historical topology structure refers to the connection relationship between the multimodal features extracted under the historical training set.

[0068] Step S104: determining the loss result of the current task based on the historical topology structure and the current topology structure.

[0069] Here, the loss result of the current task is calculated through the historical topology structure and the current topology structure. The smaller the loss result, the better the model performance.

[0070] Step S105 : Modify the model parameters of the multimodal encoder to be trained based on the loss result to obtain a trained multimodal encoder.

[0071] In some embodiments, model parameter correction is an optimization method based on the back-propagation algorithm, which updates parameters such as weights and biases in the model by calculating the partial derivatives of the loss function with respect to the model parameters, so that the loss function of the model can be gradually reduced, thereby improving the performance of the model.

[0072] The embodiment of the present application adjusts the current topology structure through the historical topology structure, so that the data in the historical topology structure can be used when training the multimodal encoder of the current task, thereby improving the multimodal encoder's ability to retain knowledge of data at different times, and can maintain the model training effect on the historical data set while improving the model training effect on the current training set.

[0073] Figure 4 This is another optional flow chart of the training method of the multimodal encoder provided in the embodiment of the present application, such as Figure 4 As shown, the method includes the following steps S201 to S212:

[0074] Step S201: The terminal receives a multimodal encoder training operation.

[0075] Here, a multimodal encoder training application may be running on the terminal, and the server constitutes a backend server of the multimodal encoder training application. The modal encoder training operation may be a selection operation or an input operation inputted through a client of the modal encoder training application running on the terminal. For example, the selection operation may select a storage path for the multimodal training data, or the input operation may be a user inputting the storage path for the multimodal training data on the client, or the user may directly input the multimodal training data on the client.

[0076] In some embodiments, the multimodal encoder training application may provide an input interface or input box that allows the user to select or enter the storage path of the multimodal training data. The input interface may be in the form of a form, a text box, or a drop-down menu, and the specific form is not limited in this application. The user may select the storage path of the multimodal training data from predetermined options or manually enter the storage path of the multimodal training data.

[0077] Step S202: The terminal generates a multimodal encoder training request in response to the multimodal encoder training operation.

[0078] Here, the terminal may encapsulate the storage path of the multimodal training data selected or input by the user into the multimodal encoder training request.

[0079] In some embodiments, in order to ensure that the storage path of the multimodal training data selected or input by the user is accurately included in the request data, first, it is necessary to check the integrity of the storage path information. In the process of filling the storage path information into the multimodal encoder training request, ensure that the storage path information is complete and accurate. The method of checking the integrity of the storage path includes checking whether the storage path contains the file name, file extension, and necessary characters such as slash or backslash; then, select a suitable data structure (such as JSON) to ensure that the storage path information is correctly embedded in the data structure; finally, special characters are processed. For special characters such as slash, backslash, quotation marks, etc., the special characters can be converted into a specific encoding format for correct transmission and parsing.

[0080] Step S203: The terminal sends a multimodal encoder training request to the server.

[0081] In some embodiments, the terminal sends the encapsulated multimodal encoder training request to the server and requests the server to perform a multimodal encoder training operation. The multimodal encoder training request is usually sent using protocols such as HTTP or Web Socket.

[0082] Step S204: The server obtains multimodal data in a current training set for a current task in response to the multimodal encoder training request.

[0083] Here, after receiving the multimodal encoder training request, the server parses the request. For example, for an HTTP request, the server can parse the request header and request body. The server parses the request header to obtain relevant information about the request and parses the request body to obtain the main body of the request, that is, the storage path of the multimodal training data. The server parses the specific fields or parameters in the request body that contain the storage path of the multimodal training data, extracts a specific data format from the request body, such as JSON or XML, and then parses the data format to obtain the storage path of the multimodal training data.

[0084] In step S205 , the server inputs the multimodal data in the current training set of the current task into the multimodal encoder to be trained, and obtains at least one multimodal feature corresponding to the multimodal data.

[0085] In some embodiments, the current training set includes a first modality data set and a second modality data set having a preset correspondence relationship; the first modality data set and the second modality data set are input into a multimodal encoder to be trained to obtain at least one first feature vector corresponding to the first modality data set and at least one second feature vector corresponding to the second modality data set; and the first feature vector and the second feature vector having a preset correspondence relationship constitute a multimodal feature.

[0086] Here, the first modal dataset and the second modal dataset are input into the multimodal encoder to be trained, and the multimodal encoder to be trained performs feature extraction on the first modal dataset and the second modal dataset to obtain at least one first feature vector corresponding to the first modal dataset and at least one second feature vector corresponding to the second modal dataset.

[0087] Step S206: The server determines the current topology of the current training set under the current task based on at least one multimodal feature.

[0088] In some embodiments, the preset corresponding relationship includes: a positive correlation relationship between the first modality data in the first modality data set and the second modality data in the second modality data set, and a negative correlation relationship between the first modality data in the first modality data set and the second modality data in the second modality data set. Figure 5 , Figure 5 It is shown that in step S206, the server determines the current topology of the current training set under the current task based on at least one multimodal feature, which can be achieved by the following steps S2061 to S2064:

[0089] In step S2061, the server performs data pairing on the first modality data in the first modality data set and the second modality data in the second modality data set according to the positive correlation and the negative correlation to obtain positive sample pairs and negative sample pairs.

[0090] Here, a positive correlation means that the label of the first modality data in the first modality data set is the same as the label of the second modality data in the second modality data set; a negative correlation means that the label of the first modality data in the first modality data set is different from the label of the second modality data in the second modality data set.

[0091] In some embodiments, when the first modality data in the first modality data set is positively correlated with the second modality data in the second modality data set, a positive sample pair is formed; when the first modality data in the first modality data set is negatively correlated with the second modality data in the second modality data set, a negative sample pair is formed.

[0092] In step S2062 , the server determines the current cosine similarity of the positive sample pair based on the first eigenvector of the first modality data and the second eigenvector of the second modality data in the positive sample pair.

[0093] Here, cosine similarity is a measure of the similarity between two vectors. The calculation method is to judge the similarity between the two vectors by comparing the cosine value of the angle between the two vectors.

[0094] In some embodiments, the current cosine similarity of the positive sample pair can be achieved by the following steps: multiplying each element in the first eigenvector of the first modal data in the positive sample pair with each element in the second eigenvector of the second modal data, and adding all the multiplication results to obtain the inner product of the first eigenvector of the first modal data and the second eigenvector of the second modal data; then, calculating the sum of the squares of each element in the first eigenvector of the first modal data, and taking the square root of the square sum to obtain the modulus of the first eigenvector of the first modal data; calculating the sum of the squares of each element in the second eigenvector of the second modal data, and taking the square root of the square sum to obtain the modulus of the second eigenvector of the second modal data; then, multiplying the modulus of the first eigenvector of the first modal data and the modulus of the second eigenvector of the second modal data to obtain the product result; finally, dividing the inner product of the first eigenvector of the first modal data and the second eigenvector of the second modal data by the product result to obtain the current cosine similarity of the positive sample pair.

[0095] Step S2063: The server determines the current cosine similarity of the negative sample pair based on the first eigenvector of the first modality data and the second eigenvector of the second modality data in the negative sample pair.

[0096] Here, the current cosine similarity of the negative sample pair is calculated in the same way as the current cosine similarity of the positive sample pair, which will not be described in detail here.

[0097] In step S2064 , the server constructs a current topological structure of the current training set under the current task based on the current cosine similarities of the positive sample pairs and the negative sample pairs.

[0098] In some embodiments, the server constructs a similarity matrix based on the current cosine similarities of the positive sample pairs and the negative sample pairs; and converts the similarity matrix into a current topological structure under the current task.

[0099] Here, each element in the similarity matrix represents the similarity between the two objects represented by the corresponding row and column in the similarity matrix.

[0100] In some embodiments, the server constructs a similarity matrix based on the current cosine similarities of the positive sample pairs and the negative sample pairs, which can be achieved by the following method: the server creates an all-zero matrix; the server traverses the positive sample pairs and the negative sample pairs; and the current cosine similarities corresponding to the positive sample pairs and the negative sample pairs are respectively filled into the all-zero matrix to obtain a similarity matrix.

[0101] Here, the all-zero matrix is ​​a matrix in which the values ​​of all elements in the matrix are zero.

[0102] In some embodiments, see Figure 6 , Figure 6 It shows that the server converts the similarity matrix into the current topology structure under the current task, which can be achieved by the following steps S301 to S305:

[0103] Step S301: The server obtains the dimension of the similarity matrix.

[0104] Here, dimension refers to the number of rows and columns of the similarity matrix. The dimension of a matrix is ​​usually expressed in the form of "number of rows × number of columns". For example, if a matrix A has m rows and n columns, the dimension of matrix A can be expressed as A(m,n) or m×n, where m represents the number of rows and n represents the number of columns.

[0105] Step S302: The server creates a topology map based on the dimension.

[0106] In which, the topological graph includes multiple nodes and connecting lines between every two nodes, the number of the multiple nodes is equal to the sum of the number of first modality data in the first modality data set and the number of second modality data in the second modality data set; each node corresponds to one first modality data or one second modality data.

[0107] Step S303: The server traverses the similarity matrix and determines the cosine similarity with a value of zero in the similarity matrix.

[0108] In step S304 , the server determines two corresponding nodes and a connection line between the two nodes in the topology graph based on the first modal data and the second modal data corresponding to each cosine similarity having a value of zero.

[0109] Step S305: The server hides the connection line between the two nodes to obtain the current topology structure under the current task.

[0110] Here, if the cosine similarity between the first modal data and the second modal data is zero, there is no need to display a connection line between the corresponding nodes of the first modal data and the second modal data whose cosine similarity is zero in the topological structure.

[0111] In some embodiments, see Figure 7 , there is a 3×3 similarity matrix A, where the horizontal axis of the similarity matrix A represents the three first modal data a, b, c, and the vertical axis represents the three second modal data x, y, z; first, the server creates a topology graph with 6 nodes based on the dimension of the similarity matrix A; then, the server traverses the similarity matrix and determines the cosine similarity with a value of zero in the similarity matrix, as well as the first modal data and the second modal data corresponding to the cosine similarity with a value of zero; finally, the connecting line between the first modal data and the second modal data with a cosine similarity of zero is hidden to obtain the current topology structure under the current task.

[0112] In step 207 , the server calls the historical multimodal encoder to perform feature extraction on the historical data in the historical training set to obtain a historical topological structure.

[0113] In some embodiments, the historical data in the historical training set includes third modality data and fourth modality data with a preset corresponding relationship; when obtaining the historical topological structure, the server can call the historical multimodal encoder to perform feature extraction on the historical data in the historical training set to obtain at least one third feature corresponding to the third modality data and at least one fourth feature corresponding to the fourth modality data; wherein, each third feature has a preset corresponding relationship with a fourth feature, and the third feature and the fourth feature with the preset corresponding relationship constitute a historical positive sample pair, and the third feature and the fourth feature without the preset corresponding relationship constitute a historical negative sample pair; the server determines the historical topological structure based on the historical positive sample pairs and the historical negative sample pairs.

[0114] In some embodiments, the server determines the historical topological structure based on historical positive sample pairs and historical negative sample pairs by the following method: the server calculates the historical cosine similarity of historical positive sample pairs and historical negative sample pairs respectively; the server constructs a historical similarity matrix based on the historical cosine similarity; the server converts the historical similarity matrix into a historical topological structure.

[0115] Here, the calculation method of historical cosine similarity is the same as the calculation method of current cosine similarity, the construction method of historical similarity matrix is ​​the same as the construction method of similarity matrix, and the conversion method of historical topology structure is the same as the conversion method of current topology structure, which will not be repeated here.

[0116] In step S208 , the server determines the loss result of the current task based on the historical topology structure and the current topology structure.

[0117] In some embodiments, see Figure 8 , Figure 8 It is shown that in step S208, the server determines the loss result of the current task based on the historical topology structure and the current topology structure, which can be achieved by the following steps S2081 to S2083:

[0118] In step S2081 , the server compares the multimodal data in the current training set with the historical data in the historical training set to obtain first-category data that is repeated with the historical data and second-category data that is not repeated with the historical data in the multimodal data.

[0119] Here, the multimodal data in the current training set may be repeated with the historical data in the historical training set. The multimodal data in the current training set is compared with the historical data in the historical training set, and the data in the multimodal data that is repeated with the historical data is classified as the first category of data, and the data in the multimodal data that is not repeated with the historical data is classified as the second category of data.

[0120] In step S2082, the server performs a first loss calculation on the current task based on the first type of data, the historical topology structure, and the current topology structure to obtain a first loss result.

[0121] In some embodiments, the server obtains the historical cosine similarity of the first category of data in the historical topological structure and the current cosine similarity of the first category of data in the current topological structure; the server performs a loss calculation on the current task based on the historical cosine similarity of the first category of data in the historical topological structure and the cosine similarity of the first category of data in the current topological structure to obtain a first loss result.

[0122] Here, the first loss result can be calculated by the following formula (1):

[0123] Loss_1=1-|cosine_old-cosine_current| (1)

[0124] Among them, Loss_1 represents the first loss result, cosine_old represents the historical cosine similarity of the first type of data in the historical topology structure, cosine_current represents the cosine similarity of the first type of data in the current topology structure, and |·| represents taking the absolute value.

[0125] In step S2083, the server performs a second loss calculation on the current task based on the second type of data, the historical topology structure, and the current topology structure to obtain a second loss result; wherein the first loss result and the second loss result constitute the loss result of the current task.

[0126] In some embodiments, the server obtains the current cosine similarity of the second type of data; the server performs a loss calculation on the current task based on the current cosine similarity of the second type of data in the current topology structure to determine a second loss result.

[0127] In step S209 , the server modifies the model parameters of the multimodal encoder to be trained based on the loss result of the current task to obtain a trained multimodal encoder.

[0128] In some embodiments, when both the first loss result and the second loss result no longer change, the training of the multimodal encoder is completed.

[0129] Step S210: The server generates a message indicating that the multimodal encoder training is completed.

[0130] Step S211: The server sends a message to the terminal indicating that the multimodal encoder training is completed.

[0131] In step S212, the terminal displays a message indicating that the multimodal encoder training is completed on the current interface.

[0132] A message indicating that the multimodal encoder training is completed is displayed on the current interface of the terminal to prompt the user that the multimodal encoder training is completed.

[0133] The training method of the multimodal encoder provided in the embodiment of the present application generates a current topological structure by extracting features from the multimodal data in the current training set, and calling a historical multimodal encoder to extract features from the historical data set to generate a historical topological structure; the multimodal data in the current training set is compared with the historical data set to obtain first-category data that is repeated with the historical data, and the cosine similarity of the first-category data in the historical topological structure is used to adjust the cosine similarity of the first-category data in the current topological structure, so that the multimodal encoder to be trained can continuously learn the features of the historical data during the training process, and ensure the accuracy on the historical data set on the basis of improving the accuracy of the new data (second-category data), thereby improving the ability of the multimodal encoder to accumulate knowledge.

[0134] Below, an exemplary application of the embodiment of the present application in a practical application scenario will be described.

[0135] The embodiments of the present application improve the multimodal encoder's ability to retain knowledge of data at different times by performing retention learning on the topological structure of historical data, and can maintain the effect on historical data sets while improving the performance of new data.

[0136] See also Figure 9 , Figure 9 This is an inference flowchart of the multimodal pre-training method provided in an embodiment of the present application. First, the target image and target text are input; then, the target image features are extracted using an image encoder, and the target text features are extracted using a text encoder; finally, the extracted target image features and target text features are input into the downstream task.

[0137] Figure 10 This is a flowchart of the implementation of the multimodal encoder training method provided in the embodiment of the present application in a practical scenario. Figure 10 As shown, the method includes the following steps:

[0138] Step S401: The server obtains a current training set, where the current training set includes image data and text data.

[0139] Here, the image data and text data contents in the current training set match each other, that is, the text data description content corresponds to the image data content.

[0140] In step S402 , the server inputs the image data into the image encoder to be trained to extract image features.

[0141] In some embodiments, the image encoder to be trained may use a Transformer structure to extract image features corresponding to different image data, as shown in formula (2):

[0142] v_i = F_vision(x_image) (2)

[0143] Here, vi_i represents the image features corresponding to different image data extracted by the image encoder to be trained; F_vision represents the image encoder to be trained; and x_image represents the input image data.

[0144] In step S403 , the server inputs the text data into the text encoder to be trained to extract text features.

[0145] In some embodiments, the text encoder to be trained may use a bidirectional encoder (Bert) structure, and extract text features corresponding to different text data through the Bert structure, as shown in formula (3):

[0146] w_i = F_text(x_text) (3)

[0147] Here, w_i represents the text features corresponding to different text data extracted by the text encoder to be trained; F_text represents the text encoder to be trained; and x_text represents the input text data.

[0148] In step S404, the server combines the image features and the text features into positive sample pairs and negative sample pairs according to whether the text features match the image features.

[0149] In some embodiments, the input image data and text data have corresponding labels, and whether the text features match the image features is determined based on whether the labels match; if they match, a positive sample pair is generated, and if they do not match, a negative sample pair is generated.

[0150] Step S405 : The server calculates the current cosine similarities of different positive sample pairs and negative sample pairs.

[0151] The current cosine similarity calculation method is shown in formula (4):

[0152] cosine_sim = cosine(v_i, w_i) (4)

[0153] Here, cosine_sim represents the current cosine similarity; cosine(v_i, w_i) represents the cosine operation on the corresponding image features and text features.

[0154] Step S406: The server generates a current similarity matrix according to the current cosine similarities of different positive sample pairs and negative sample pairs.

[0155] Here, the element values ​​in the current similarity matrix represent the correlation between different data.

[0156] Step S407: The server converts the current similarity matrix into the current topology structure.

[0157] In some embodiments, each value in the topological structure represents the topological correlation between different features. The calculation method for converting the current similarity matrix into the current topological structure is shown in formula (5):

[0158] current_topology =(cosine_sim1, cosine_sim2, ...) (5)

[0159] Here, current_topology represents the current topology, and cosine_sim1 and cosine_sim2 represent the current cosine similarities in the current similarity matrix.

[0160] Step S408: The server calls the historical image encoder to extract image features from the historical data.

[0161] Here, the calculation method for extracting image features from historical data through the historical image encoder is shown in formula (6):

[0162] v_i_old = F_vision_old(x_image_old) (6)

[0163] Here, vi_i_old represents the image features corresponding to different historical image data extracted by the historical image encoder; F_vision_old represents the historical image encoder; and x_image_old represents the historical input image data.

[0164] Step S409: The server calls the historical text encoder to extract text features from the historical data.

[0165] Here, the calculation method for extracting text features from historical data through the historical text encoder is shown in formula (7):

[0166] w_i_old = F_text_old(x_text_old) (7)

[0167] Here, w_i_old represents the text features corresponding to different historical text data extracted by the historical text encoder; F_text_old represents the historical text encoder; and x_text_old represents the historical input text data.

[0168] In step S410 , the server combines the text features corresponding to the historical text data and the image features corresponding to the historical image data into historical positive sample pairs and historical negative sample pairs based on whether the text features corresponding to the historical text data match the image features corresponding to the historical image data.

[0169] In step S411 , the server calculates historical cosine similarities of different historical positive sample pairs and historical negative sample pairs.

[0170] The calculation method of historical cosine similarity is shown in formula (8):

[0171] cosine_sim_old = cosine(v_i_old, w_i_old) (8)

[0172] Here, cosine_sim_old represents historical cosine similarity; cosine(v_i_old, w_i_old) represents cosine operation on text features corresponding to the corresponding historical text data and image features corresponding to the historical image data.

[0173] In step S412 , the server generates a historical similarity matrix according to the historical cosine similarities of different historical positive sample pairs and historical negative sample pairs.

[0174] Step S413: The server converts the historical similarity matrix into a historical topology structure.

[0175] Here, the calculation method for converting the historical similarity matrix into the historical topological structure is shown in formula (9):

[0176] old_topology = (cosine_sim1_old, cosine_sim2_old, ...) (9)

[0177] Here, old_topology represents the historical topology, cosine_sim1_old and cosine_sim2_old represent the historical cosine similarities in the historical similarity matrix.

[0178] In step S414, the server compares the data in the current training set with the historical data to obtain duplicate data, where the duplicate data is data in the historical data that is identical to the data in the current training set.

[0179] Step S415: The server obtains the historical cosine similarity of the duplicate data in the historical topology structure.

[0180] In step S416 , the server calculates duplicate data loss based on the historical cosine similarity of the duplicate data and the current cosine similarity of the duplicate data.

[0181] Here, the method for calculating duplicate data loss based on the historical cosine similarity of duplicate data and the current cosine similarity of duplicate data is shown in formula (10):

[0182] Loss_exist = 1-|cosine_old-cosine_current| (10)

[0183] Here, loss_exist represents the loss of duplicate data; cosine_old represents the historical cosine similarity of duplicate data in old_topology; cosine_current represents the current cosine similarity of duplicate data in current_topology; || represents the absolute value.

[0184] In step S417 , the server obtains the current cosine similarity of new data according to the current topology structure, wherein the new data is data that is different from the data in the current training set and the historical data.

[0185] Step S418: The server calculates the new data loss based on the current cosine similarity of the new data.

[0186] Here, the method for calculating the new data loss based on the current cosine similarity of the new data is shown in formula (11):

[0187] Loss_new = (1-pos_sim) + (1-neg_sim) (11)

[0188] Here, Loss_new represents the loss of new data; pos_sim represents the similarity of positive sample pairs; neg_sim represents the similarity of negative sample pairs.

[0189] In step S419 , when the server detects that the loss of duplicate data and the loss of new data are no longer decreasing, it generates a trained image encoder and text encoder.

[0190] Figure 11 This is another implementation flow diagram of the multimodal encoder training method provided in an embodiment of the present application in a practical scenario. Figure 11 As shown in the figure: For the current task, the server first obtains the input image and input text. Then, the server inputs the input image into the image encoder, which extracts image features. The server inputs the input text into the text encoder, which extracts text features. Next, the server combines the extracted image features and text features into positive and negative sample pairs, and calculates the cosine similarity of the positive and negative sample pairs. Finally, the server generates a similarity matrix based on the cosine similarity and converts the similarity matrix into the current topology. For historical tasks and historical data, the above steps are repeated to generate a historical topology. The server adjusts the current topology based on the historical topology.

[0191] In an embodiment of the present application, the current topology structure is adjusted through the historical topology structure, so that the data in the historical topology structure can be used when training the multimodal encoder of the current task, thereby improving the model training effect on the current training set while maintaining the model training effect on the historical data set.

[0192] It should be noted that the method of the embodiment of the present application, in addition to being used in the image-text multimodal task as exemplified in the above embodiment, can also achieve the same effect in the speech-text multimodal task.

[0193] It is understandable that in the embodiments of the present application, the content involving user information, such as the user's terminal IP address, the user's device information, etc., if it involves data related to user information or corporate information, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to blur this information to eliminate the correspondence between this information and the user; and the relevant data collection and processing should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0194] The following continues to describe the exemplary structure of the multimodal encoder training device 254 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the training device 254 of the multimodal encoder includes: a multimodal feature acquisition module 2541, which is used to input the multimodal data in the current training set of the current task into the multimodal encoder to be trained, and obtain at least one multimodal feature corresponding to the multimodal data; a current topology structure determination module 2542, which is used to determine the current topology structure of the current training set under the current task based on the at least one multimodal feature; a historical topology structure determination module 2543, which is used to call the historical multimodal encoder to extract features from the historical data in the historical training set to obtain a historical topology structure; wherein the historical multimodal encoder is obtained after training the multimodal encoder to be trained based on the historical training set; a loss result determination module 2544, which is used to determine the loss result of the current task based on the historical topology structure and the current topology structure; a model parameter correction module 2545, which is used to correct the model parameters of the multimodal encoder to be trained based on the loss result to obtain a trained multimodal encoder.

[0195] In some embodiments, the current training set includes a first modal data set and a second modal data set having a preset corresponding relationship; the multimodal feature acquisition module is further used to: input the first modal data set and the second modal data set into the multimodal encoder to be trained to obtain at least one first feature vector corresponding to the first modal data set and at least one second feature vector corresponding to the second modal data set; and the first feature vector and the second feature vector having the preset corresponding relationship constitute a multimodal feature.

[0196] In some embodiments, the preset correspondence includes: a positive correlation between the first modal data in the first modal data set and the second modal data in the second modal data set, and a negative correlation between the first modal data in the first modal data set and the second modal data in the second modal data set; the current topology structure determination module is further used to: perform data pairing on the first modal data in the first modal data set and the second modal data in the second modal data set according to the positive correlation and the negative correlation to obtain positive sample pairs and negative sample pairs; determine the current cosine similarity of the positive sample pair according to the first eigenvector of the first modal data and the second eigenvector of the second modal data in the positive sample pair; determine the current cosine similarity of the negative sample pair according to the first eigenvector of the first modal data and the second eigenvector of the second modal data in the negative sample pair; and construct the current topology structure of the current training set under the current task based on the current cosine similarities of the positive sample pair and the negative sample pair.

[0197] In some embodiments, the current topology structure determination module is further used to: construct a similarity matrix based on the current cosine similarities of the positive sample pairs and the negative sample pairs; and convert the similarity matrix into the current topology structure under the current task.

[0198] In some embodiments, the device further includes a similarity matrix construction module for creating an all-zero matrix; traversing the positive sample pairs and the negative sample pairs; and filling the current cosine similarities corresponding to the positive sample pairs and the negative sample pairs into the all-zero matrix respectively to obtain the similarity matrix.

[0199] In some embodiments, the current topology structure determination module is further used to: obtain the dimension of the similarity matrix; create a topology map based on the dimension; wherein the topology map includes multiple nodes and connecting lines between every two nodes, and the number of the multiple nodes is equal to the sum of the number of first modality data in the first modality data set and the number of second modality data in the second modality data set; each node corresponds to one first modality data or one second modality data; traverse the similarity matrix to determine the cosine similarity with a value of zero in the similarity matrix; based on the first modality data and the second modality data corresponding to each cosine similarity with a value of zero, determine the corresponding two nodes in the topology map, and the connecting line between the two nodes; hide the connecting line between the two nodes to obtain the current topology structure under the current task.

[0200] In some embodiments, the historical data in the historical training set includes third modality data and fourth modality data having a preset corresponding relationship; the historical topology structure determination module is further used to: call the historical multimodal encoder to perform feature extraction on the historical data in the historical training set to obtain at least one third feature corresponding to the third modality data and at least one fourth feature corresponding to the fourth modality data; wherein each third feature has the preset corresponding relationship with a fourth feature, and the third feature and the fourth feature having the preset corresponding relationship constitute a historical positive sample pair, and the third feature and the fourth feature not having the preset corresponding relationship constitute a historical negative sample pair; based on the historical positive sample pair and the historical negative sample pair, the historical topology structure is determined.

[0201] In some embodiments, the historical topology structure determination module is further used to: calculate the historical cosine similarities of the historical positive sample pairs and the historical negative sample pairs respectively; construct a historical similarity matrix based on the historical cosine similarities; and convert the historical similarity matrix into the historical topology structure.

[0202] In some embodiments, the loss result determination module is also used to: compare the multimodal data in the current training set with the historical data in the historical training set to obtain a first type of data in the multimodal data that is repeated with the historical data and a second type of data that is not repeated with the historical data; based on the first type of data, the historical topological structure and the current topological structure, perform a first loss calculation on the current task to obtain a first loss result; based on the second type of data, the historical topological structure and the current topological structure, perform a second loss calculation on the current task to obtain a second loss result; wherein, the first loss result and the second loss result constitute the loss result of the current task.

[0203] In some embodiments, the loss result determination module is also used to: obtain the historical cosine similarity of the first category of data in the historical topological structure and the current cosine similarity of the first category of data in the current topological structure; based on the historical cosine similarity of the first category of data in the historical topological structure and the current cosine similarity of the first category of data in the current topological structure, perform loss calculation on the current task to obtain the first loss result.

[0204] In some embodiments, the loss result determination module is further used to: obtain the current cosine similarity of the second category data; perform loss calculation on the current task based on the current cosine similarity of the second category data in the current topological structure, and determine the second loss result.

[0205] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0206] An embodiment of the present application provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned multimodal encoder training method when executing the executable instructions stored in the memory.

[0207] An embodiment of the present application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned multimodal encoder training method is implemented.

[0208] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided by the embodiment of the present application, for example, Figure 3 The method shown.

[0209] In some embodiments, the storage medium can be a computer-readable storage medium, such as a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPR OM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.

[0210] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0211] As an example, the executable instructions may, but need not necessarily, correspond to a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed in multiple locations and interconnected by a communication network.

[0212] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A method for training a multimodal encoder, characterized in that: The method comprises: Inputting multimodal data in a current training set of a current task into a multimodal encoder to be trained, and obtaining at least one multimodal feature corresponding to the multimodal data; Determining a current topological structure of the current training set under the current task based on the at least one multimodal feature; Calling a historical multimodal encoder to perform feature extraction on historical data in a historical training set to obtain a historical topological structure; wherein the historical multimodal encoder is obtained by training the multimodal encoder to be trained based on the historical training set; Determining a loss result of the current task based on the historical topology structure and the current topology structure; The model parameters of the multimodal encoder to be trained are modified based on the loss result to obtain a trained multimodal encoder.

2. The method according to claim 1, characterized in that The current training set includes a first modality data set and a second modality data set having a preset corresponding relationship; The step of inputting the multimodal data in the current training set of the current task into the multimodal encoder to be trained to obtain at least one multimodal feature corresponding to the multimodal data includes: The first modal data set and the second modal data set are input into the multimodal encoder to be trained to obtain at least one first feature vector corresponding to the first modal data set and at least one second feature vector corresponding to the second modal data set; and the first feature vector and the second feature vector having the preset corresponding relationship constitute a multimodal feature.

3. The method according to claim 2, characterized in that The preset corresponding relationship includes: a positive correlation relationship between the first modality data in the first modality data set and the second modality data in the second modality data set, and a negative correlation relationship between the first modality data in the first modality data set and the second modality data in the second modality data set; Determining a current topological structure of the current training set under the current task based on the at least one multimodal feature includes: According to the positive correlation and the negative correlation, pair the first modality data in the first modality data set with the second modality data in the second modality data set to obtain a positive sample pair and a negative sample pair; Determining a current cosine similarity of the positive sample pair based on a first eigenvector of the first modality data and a second eigenvector of the second modality data in the positive sample pair; Determining a current cosine similarity of the negative sample pair based on a first eigenvector of the first modality data and a second eigenvector of the second modality data in the negative sample pair; Based on the current cosine similarities of the positive sample pair and the negative sample pair, a current topological structure of the current training set under the current task is constructed.

4. The method according to claim 3, characterized in that The constructing a current topological structure of the current training set under the current task based on the current cosine similarity of the positive sample pair and the negative sample pair includes: Constructing a similarity matrix based on the current cosine similarities of the positive sample pair and the negative sample pair; The similarity matrix is ​​converted into a current topological structure under the current task.

5. The method according to claim 4, characterized in that The constructing a similarity matrix based on the current cosine similarities of the positive sample pair and the negative sample pair includes: Create an all-zero matrix; Traversing the positive sample pairs and the negative sample pairs; The current cosine similarities corresponding to the positive sample pair and the negative sample pair are respectively filled into the all-zero matrix to obtain the similarity matrix.

6. The method according to claim 4, characterized in that Converting the similarity matrix into the current topological structure under the current task includes: Obtaining the dimension of the similarity matrix; Based on the dimension, a topological graph is created; wherein the topological graph includes a plurality of nodes and a connecting line between every two nodes, the number of the plurality of nodes is equal to the sum of the number of first modality data in the first modality data set and the number of second modality data in the second modality data set; and each node corresponds to one first modality data or one second modality data; Traversing the similarity matrix, and determining the cosine similarity with a value of zero in the similarity matrix; Based on the first modal data and the second modal data corresponding to each cosine similarity having a value of zero, determining two corresponding nodes in the topological graph and a connecting line between the two nodes; The connection line between the two nodes is hidden to obtain the current topological structure under the current task.

7. The method according to claim 1, characterized in that The historical data in the historical training set includes third modality data and fourth modality data having a preset corresponding relationship; The historical multimodal encoder is called to extract features from the historical data in the historical training set to obtain a historical topological structure, including: Calling the historical multimodal encoder to perform feature extraction on the historical data in the historical training set to obtain at least one third feature corresponding to the third modal data and at least one fourth feature corresponding to the fourth modal data; wherein each third feature has the preset corresponding relationship with a fourth feature, and the third and fourth features having the preset corresponding relationship constitute a historical positive sample pair, and the third and fourth features not having the preset corresponding relationship constitute a historical negative sample pair; The historical topological structure is determined based on the historical positive sample pairs and the historical negative sample pairs.

8. The method according to claim 7, characterized in that The determining the historical topological structure based on the historical positive sample pairs and the historical negative sample pairs includes: Calculating the historical cosine similarity of the historical positive sample pair and the historical negative sample pair respectively; Constructing a historical similarity matrix based on the historical cosine similarity; The historical similarity matrix is ​​converted into the historical topology structure.

9. The method according to claim 1, characterized in that The determining the loss result of the current task based on the historical topological structure and the current topological structure includes: Comparing the multimodal data in the current training set with the historical data in the historical training set to obtain first-category data that is repeated with the historical data and second-category data that is not repeated with the historical data in the multimodal data; Performing a first loss calculation on the current task based on the first type of data, the historical topological structure, and the current topological structure to obtain a first loss result; Based on the second type of data, the historical topology structure and the current topology structure, a second loss calculation is performed on the current task to obtain a second loss result; wherein, the first loss result and the second loss result constitute the loss result of the current task.

10. The method according to claim 9, characterized in that The performing a first loss calculation on the current task based on the first type of data, the historical topological structure, and the current topological structure to obtain a first loss result includes: Obtaining historical cosine similarity of the first type of data in the historical topological structure and current cosine similarity of the first type of data in the current topological structure; Based on the historical cosine similarity of the first category of data in the historical topological structure and the current cosine similarity of the first category of data in the current topological structure, a loss calculation is performed on the current task to obtain the first loss result.

11. The method according to claim 9, characterized in that The performing a second loss calculation on the current task based on the second type of data, the historical topological structure, and the current topological structure to obtain a second loss result includes: Obtaining the current cosine similarity of the second type of data; Based on the current cosine similarity of the second type of data in the current topological structure, a loss calculation is performed on the current task to determine the second loss result.

12. A training device for a multimodal encoder, characterized in that: The device comprises: a multimodal feature acquisition module, configured to input the multimodal data in the current training set of the current task into the multimodal encoder to be trained, and obtain at least one multimodal feature corresponding to the multimodal data; a current topology structure determination module, configured to determine a current topology structure of the current training set under the current task based on the at least one multimodal feature; A historical topology structure determination module is used to call a historical multimodal encoder to perform feature extraction on historical data in a historical training set to obtain a historical topology structure; wherein the historical multimodal encoder is obtained by training the multimodal encoder to be trained based on the historical training set; a loss result determination module, configured to determine a loss result of the current task based on the historical topology structure and the current topology structure; A model parameter correction module is used to correct the model parameters of the multimodal encoder to be trained based on the loss result to obtain a trained multimodal encoder.

13. An electronic device, characterized in that: include: a memory for storing executable instructions; The processor is configured to implement the multimodal encoder training method according to any one of claims 1 to 11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to cause a processor to execute the executable instructions to implement the training method of the multimodal encoder according to any one of claims 1 to 11.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer-executable instructions or computer program are executed by a processor, the training method for the multimodal encoder according to any one of claims 1 to 11 is implemented.

Images