Collaborative training method for multi-modal general-specialized model based on federated learning

Abstract

The embodiment of the application provides a kind of based on the collaborative training method of multi-modal general-purpose model of federal learning, by receiving data to be retrieved, the data features of data to be retrieved are extracted, and multi-modal data index is established based on data features;Based on multi-modal data index, data features are processed through attention mechanism, and fusion data features are obtained;Fusion data features are input into retrieval general-purpose model, and knowledge extraction is carried out on fusion data features by knowledge extraction layer of retrieval general-purpose model, to obtain general knowledge representation;General knowledge representation is input into retrieval special-purpose model, and general knowledge representation is processed based on retrieval special-purpose model, to obtain the retrieval result corresponding to data to be retrieved, and the general model parameters of retrieval general-purpose model and the special model parameters of retrieval special-purpose model are adjusted based on retrieval result.The method effectively solves the deficiency in the aspect of difficult to effectively fuse multi-modal heterogeneous data, and significantly improves the generalization ability of model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a method for collaborative training of multimodal communication models based on federated learning. Background Art

[0002] With the development of artificial intelligence technology, the demand for processing multimodal data has increased accordingly. Multimodal data includes image data, text data, and audio data.

[0003] In the existing technology, federated learning can be used to perform joint model training while protecting data privacy, thereby processing multimodal data. However, when processing multimodal data, existing federated learning models have difficulty effectively fusing multimodal data for collaborative training because data of different modalities have different feature representations and semantic information. Moreover, in actual applications, there are general models and specialized models for specific tasks. General models and specialized models have difficulty in distinguishing the authenticity of fused features, which limits the accuracy of multimodal retrieval. Summary of the Invention

[0004] In response to the problems in the existing technology, this application provides a collaborative training method for a multimodal general-purpose model based on federated learning, which can effectively solve the shortcomings of traditional technologies in effectively integrating multimodal heterogeneous data during the federated learning process and in terms of fusion difficulties, significantly improve the generalization ability of the multimodal general-purpose model under federated learning, and better integrate the data to be retrieved.

[0005] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a method for collaborative training of a multimodal communication model based on federated learning, comprising:

[0007] receiving data to be retrieved, extracting data features of the data to be retrieved, and establishing a multimodal data index based on the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved, and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved, and audio features to be retrieved;

[0008] Based on multimodal data indexing, data features are processed through an attention mechanism in a preset cross-modal feature mapping network, and the processed data features are mapped to a preset high-dimensional feature space for fusion to obtain fused data features, thereby fusing the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved;

[0009] Input the fused data features into the general retrieval model, and extract knowledge from the fused data features through the knowledge extraction layer of the general retrieval model to obtain a general knowledge representation;

[0010] The general knowledge representation is input into the retrieval-specific model, the general knowledge representation is processed based on the retrieval-specific model to obtain retrieval results corresponding to the data to be retrieved, and the general model parameters of the retrieval general model and the special model parameters of the retrieval-specific model are adjusted based on the retrieval results.

[0011] Furthermore, the method further includes: normalizing the image data to be retrieved, and mapping the pixel values ​​corresponding to the normalized image data to be retrieved to a preset pixel interval to obtain initial image features to be retrieved;

[0012] Perform word vector encoding on the text data to be retrieved to obtain the initial features of the text to be retrieved;

[0013] Segmenting the audio data to be retrieved according to a preset frame length, and extracting features of each segmented audio data to be retrieved to obtain initial audio features to be retrieved;

[0014] The initial image features to be retrieved, the initial text features to be retrieved, and the initial audio features to be retrieved are subjected to feature normalization to convert the initial image features to be retrieved, the initial text features to be retrieved, and the initial audio features to be retrieved into a standard normal distribution, thereby obtaining the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved.

[0015] Furthermore, after mapping the processed data features to a preset high-dimensional feature space for fusion to obtain fused data features, the method further includes:

[0016] Input the fused data features into the discriminator to obtain the probability value corresponding to the fused data features;

[0017] The fused data features whose probability values ​​are greater than the preset probability threshold are determined as true fused features, and the fused data features whose probability values ​​are not greater than the preset probability threshold are determined as generated fused features;

[0018] The fusion process of data features is optimized based on the real fusion features and the generated fusion features.

[0019] Furthermore, the method further includes: inputting the data features and the general knowledge representation into a retrieval-specific model, fusing the data features and the general knowledge representation through a knowledge receiving network layer and a dedicated fusion network layer in the retrieval-specific model to obtain an initial retrieval result;

[0020] The initial retrieval results are processed by the fully connected layer and activation function in the retrieval-specific model to obtain the retrieval results corresponding to the data to be retrieved.

[0021] Furthermore, the method further includes: updating the dedicated model parameters corresponding to the retrieval dedicated model based on the retrieval results;

[0022] Receive the true label corresponding to the retrieval result, and determine the multi-task loss value between the retrieval result and the true label using the multi-task loss function in the retrieval-specific model, where the multi-task loss function includes a classification cross entropy loss function and a cross-modal contrast loss function;

[0023] The multi-task loss value is back-propagated to the retrieval general model so that the retrieval general model updates the general model parameters based on the multi-task loss value.

[0024] Furthermore, the method further includes: assigning a unique identifier to the same set of data to be retrieved, and establishing an association mapping relationship between the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved based on the unique identifier;

[0025] Based on the association mapping relationship, a graph-structured multimodal data index is constructed. The nodes in the multimodal data index represent different modal features of the same data to be retrieved, and the edges represent the semantic association strength between data features.

[0026] Furthermore, the method further includes: extracting knowledge from the fused data features by retrieving the middle hidden layer of the general model to obtain multimodal shared features;

[0027] The multimodal shared features are screened through the gating mechanism in the knowledge extraction layer to obtain the initial general knowledge representation, and the initial general knowledge representation is regularized and constrained to obtain the general knowledge representation.

[0028] In a second aspect, the present application provides a multimodal communication model collaborative training device based on federated learning, comprising:

[0029] a first processing module, configured to receive data to be retrieved, extract data features of the data to be retrieved, and establish a multimodal data index based on the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved, and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved, and audio features to be retrieved;

[0030] A second processing module is configured to process data features based on a multimodal data index through an attention mechanism in a preset cross-modal feature mapping network, and map the processed data features to a preset high-dimensional feature space for fusion to obtain fused data features, thereby fusing the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved;

[0031] The third processing module is used to input the fused data features into the retrieval general model, perform knowledge extraction on the fused data features through the knowledge extraction layer of the retrieval general model, and obtain a general knowledge representation;

[0032] The fourth processing module is configured to input the general knowledge representation into the retrieval special model, process the general knowledge representation based on the retrieval special model, obtain a retrieval result corresponding to the data to be retrieved, and adjust the general model parameters of the retrieval general model and the special model parameters of the retrieval special model based on the retrieval result.

[0033] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for collaborative training of a multi-modal general-special model based on federated learning when executing the program.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the method for collaborative training of a multi-modal general-special model based on federated learning.

[0035] In a fifth aspect, the present application provides a computer program product comprising computer programs / instructions, wherein the computer programs / instructions are executable by a processor to implement the steps of the method for collaborative training of a multi-modal general-special model based on federated learning.

[0036] According to the above technical solution, the present application provides a method for collaborative training of a multi-modal general-special model based on federated learning, which receives data to be retrieved, extracts data features of the data to be retrieved, and establishes a multi-modal data index according to the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved, and audio data to be retrieved, the data features include image features to be retrieved, text features to be retrieved, and audio features to be retrieved, the data features are processed through an attention mechanism in a preset cross-modal feature network according to the multi-modal data index, and are mapped to a high-dimensional feature space for fusion to generate fused data features, the fused data features are input into a knowledge extraction layer of a retrieval general model to obtain a general knowledge representation, the general knowledge representation is input into a retrieval special model for processing to obtain a retrieval result, and the general model parameters of the retrieval general model and the special model parameters of the retrieval special model are adjusted according to the retrieval result, thereby improving the fusion efficiency and accuracy of multi-modal data, effectively unifying the heterogeneous feature spaces of the data to be retrieved through the cross-modal feature mapping network and the attention mechanism, optimizing the collaborative training capability of the retrieval special model and the retrieval general model, realizing knowledge sharing of the retrieval special model and the retrieval general model through hierarchical knowledge extraction and parameter joint adjustment, and improving the generalization capability and convergence speed. This method effectively solves the problems of traditional technologies in the federated learning process, such as difficulty in effectively fusing multi-modal heterogeneous data and difficulty in fusion, significantly improves the generalization capability of the multi-modal general-special model under federated learning, and better fuses the data to be retrieved. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 Schematic diagram of the process of collaborative training of multimodal communication models based on federated learning in an embodiment of the present application;

[0039] Figure 2 This is a structural diagram of a multimodal communication model collaborative training device based on federated learning in an embodiment of the present application;

[0040] Figure 3 Schematic diagram of the structure of the electronic device in the embodiment of the present application.

[0041] Reference numerals:

[0042] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0043] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0044] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0045] In the prior art, the model of federated learning can be optimized by unifying the scale of multi-modal data through feature standardization, feature alignment can be performed by constructing a cross-modal mapping network, and data security can be protected by differential privacy. However, in the way of optimizing the model of federated learning by unifying the scale of multi-modal data through feature standardization, the deep semantic gap cannot be eliminated, in the process of performing feature alignment by constructing a cross-modal mapping network, the attention mechanism is not deeply combined with the federated learning framework, resulting in low knowledge transfer efficiency, and in the process of protecting data security by differential privacy, the noise adding strategy lacks dynamic adaptability and is prone to cause model convergence difficulty.

[0046] None of the prior art solves the hierarchical knowledge interaction problem in the collaborative training of general-special models, and the existing feature fusion method does not make full use of the generative adversarial network, which makes it difficult to judge the authenticity of the fused features, and restricts the improvement of multi-modal retrieval accuracy.

[0047] The role of federated learning includes providing the data required by other ends in the case of incomplete or insufficient data at one end, establishing a virtual common model without violating data privacy, protecting privacy and ensuring data compliance.

[0048] In order to effectively solve the deficiencies of traditional technology in the process of federated learning, such as difficulty in effectively fusing multi-modal heterogeneous data, and improve the generalization ability of multi-modal general-special models under federated learning, and better fuse the data to be retrieved, an embodiment of a multi-modal general-special model collaborative training method based on federated learning is provided, as shown in Figure 1 The multi-modal general-special model collaborative training method based on federated learning specifically includes the following contents:

[0049] Step S101: receiving the data to be retrieved, extracting the data features of the data to be retrieved, and establishing a multi-modal data index based on the data features.

[0050] Among them, the data to be retrieved includes image data to be retrieved, text data to be retrieved and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved and audio features to be retrieved;

[0051] Optionally, the embodiment receives the data to be retrieved, wherein the data to be retrieved includes multiple modalities, including but not limited to image data to be retrieved, text data to be retrieved and audio data to be retrieved.

[0052] For each data modality of the data to be retrieved, data features of the data to be retrieved are extracted, wherein the data features include image features to be retrieved, text features to be retrieved and audio features to be retrieved, that is, feature extraction is performed on the image data to be retrieved to obtain image features to be retrieved, feature extraction is performed on the text data to be retrieved to obtain text features to be retrieved, and feature extraction is performed on the audio data to be retrieved to obtain audio features to be retrieved.

[0053] In addition, a multimodal data index is established based on data characteristics. The role of the multimodal data index is similar to a directory that can be quickly queried, which can improve retrieval efficiency. The multimodal index can be constructed by indexing according to the corresponding samples to associate the multimodal data to be retrieved corresponding to the same data to be retrieved.

[0054] This embodiment realizes that by establishing a multimodal data index, data of different modalities corresponding to the data to be retrieved can be quickly located and associated, thereby reducing the data search time during the retrieval process and improving the retrieval efficiency.

[0055] Step S102: Based on the multimodal data index, the data features are processed through the attention mechanism in the preset cross-modal feature mapping network, and the processed data features are mapped to the preset high-dimensional feature space for fusion to obtain fused data features to fuse the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved.

[0056] Optionally, this embodiment determines the data features of different modalities of the data to be retrieved by establishing a multimodal data index, inputs the acquired data features of different modalities into a preset cross-modal feature mapping network, and processes the data features of different modalities through an attention mechanism. The processing method can be weighted fusion, and the attention mechanism can dynamically assign weights according to the importance of the data features, so that the model pays more attention to relatively critical data features, thereby improving the effect of feature fusion.

[0057] In addition, the data features processed by the attention mechanism are mapped to a preset high-dimensional feature space. In the high-dimensional feature space, the data features of different modalities can better represent the semantic information of the data to be retrieved, and the differences in data features between different modalities will be reduced, making it easier to integrate.

[0058] In the high-dimensional feature space, data features are fused to obtain fused data features. The fusion method can be simple splicing, element-by-element addition, or fusion through multimodal feature interaction networks.

[0059] Among them, the fusion feature data realizes the organic integration of the image features to be retrieved, the text features to be retrieved and the audio features to be retrieved, which can more comprehensively and accurately represent the comprehensive information of the data to be retrieved, breaking the limitations of single modal features.

[0060] This embodiment achieves highlighting key features through the attention mechanism, so that the fused data features can better reflect the essential information of the data to be retrieved. Processing in a high-dimensional feature space can narrow the differences between data features of different modalities, making the fused data features more natural and effective, and enabling the model to more accurately understand the meaning of the data to be retrieved, thereby improving the quality of the fused data features and the accuracy of retrieval.

[0061] Step S103: Input the fused data features into the general retrieval model, perform knowledge extraction on the fused data features through the knowledge extraction layer of the general retrieval model, and obtain a general knowledge representation.

[0062] Optionally, this embodiment inputs the obtained fused data features into a general retrieval model, which includes a knowledge extraction layer. The knowledge extraction layer is used to perform in-depth extraction of the fused data features to mine more general and universal knowledge representations.

[0063] Knowledge extraction is performed on the fused data features through the knowledge extraction layer, that is, a higher-level general knowledge representation can be extracted from the fused data features through operations such as nonlinear transformation. Among them, the general knowledge representation can capture the common patterns and semantic information in the fused data features and has strong versatility and transferability.

[0064] For example, for fused data features, the knowledge extraction layer can extract general knowledge representations such as data themes, emotional tendencies, and scene categories that reflect the fused data features.

[0065] Furthermore, the general retrieval model can be enhanced by pre-training it, such as one pre-trained on large-scale multimodal data, to improve its versatility and performance. Furthermore, to better extract general knowledge, multi-task learning can be employed, whereby multiple related tasks are simultaneously trained, enabling the general retrieval model to learn more general and robust feature representations.

[0066] Furthermore, for example, if the data to be retrieved is medical images, diagnostic texts, and recordings of the diagnostic process, where the general retrieval model and the special retrieval model are provided by different companies respectively, the medical images, diagnostic texts, and recordings of the diagnostic process can be jointly processed through federated learning to obtain retrieval results. This can protect the privacy of the general retrieval models and / or special retrieval models of different companies, and can also collaboratively obtain retrieval results through federated learning, thereby improving the generalization ability of model application.

[0067] This embodiment realizes that by extracting general knowledge representation, the general retrieval model can learn the general patterns and semantic information in the fusion data features to apply the general knowledge to a variety of different retrieval tasks, thereby improving the versatility, transferability and generalization capabilities.

[0068] Step S104: input the general knowledge representation into the retrieval-specific model, process the general knowledge representation based on the retrieval-specific model, obtain the retrieval results corresponding to the data to be retrieved, and adjust the general model parameters of the retrieval general model and the specific model parameters of the retrieval-specific model based on the retrieval results.

[0069] Optionally, this embodiment inputs the obtained general knowledge representation into a retrieval-specific model, and processes the general knowledge representation for a specific retrieval task according to the retrieval-specific model, wherein the retrieval-specific model may include but is not limited to the network structure and parameters of the specific retrieval task, such as a fully connected layer, an attention layer, etc. Through further processing of the general knowledge representation, a feature representation related to the retrieval task is obtained.

[0070] In addition, based on the processed feature representation, a retrieval result corresponding to the data to be retrieved is generated, wherein the retrieval result may be a document, image, or audio related to the data to be retrieved.

[0071] For example, in the scenario of image-text-audio joint retrieval of product information, the retrieval-specific model can refine and transform the general knowledge representation based on product attributes, user query intent (i.e., general knowledge representation), etc., to obtain retrieval results corresponding to the data to be retrieved.

[0072] In addition, according to the obtained retrieval results, the general model parameters of the general retrieval model and the special model parameters of the special retrieval model are adjusted so that the general retrieval model and the special retrieval model can better fit the data to be retrieved and improve the accuracy of the retrieval.

[0073] This embodiment achieves obtaining retrieval results and correcting the general model parameters and special model parameters of the general retrieval model and the special retrieval model based on the retrieval results, thereby improving retrieval performance, better adapting to future retrieval tasks, achieving continuous improvement and optimization of the model, and ensuring the accuracy and reliability of the retrieval results.

[0074] This embodiment realizes feature extraction, feature processing and fusion based on the attention mechanism, general knowledge representation extraction, retrieval-specific model processing and parameter adjustment of related models, so as to fully mine and utilize the comprehensive information of multimodal data to be retrieved, break the limitations of single-modality retrieval, improve the efficiency and quality of retrieval, and enhance the user's retrieval experience.

[0075] In some embodiments, receiving the data to be retrieved and extracting data features of the data to be retrieved include:

[0076] Normalizing the image data to be retrieved, and mapping the pixel values ​​corresponding to the normalized image data to be retrieved to a preset pixel interval to obtain initial features of the image to be retrieved;

[0077] Perform word vector encoding on the text data to be retrieved to obtain the initial features of the text to be retrieved;

[0078] Segmenting the audio data to be retrieved according to a preset frame length, and extracting features of each segmented audio data to be retrieved to obtain initial audio features to be retrieved;

[0079] The initial image features to be retrieved, the initial text features to be retrieved, and the initial audio features to be retrieved are subjected to feature normalization to convert the initial image features to be retrieved, the initial text features to be retrieved, and the initial audio features to be retrieved into a standard normal distribution, thereby obtaining the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved.

[0080] Optionally, this embodiment normalizes the image data to be retrieved, that is, maps the pixel values ​​of the image data to be retrieved to the interval [0,1]. Normalization can be achieved through linear transformation, and then the pixel values ​​corresponding to the normalized image data to be retrieved are mapped to a preset pixel interval, such as [0,255], to obtain the initial image features to be retrieved, so as to reduce the scale difference of the image data to be retrieved.

[0081] At the same time, the text data to be retrieved is encoded by word vectors, and the text data to be retrieved is converted into a numerical vector representation to obtain the initial features of the text to be retrieved. The word vector encoding can be implemented by a word vector model (such as Word2Vec, GloVe or BERT), specifically including:

[0082] Word segmentation: split the text data to be retrieved into words or subword units;

[0083] Embedding: Mapping each word or subword unit into a high-dimensional vector space;

[0084] Aggregation: The vectors of all words or subword units are averaged or weighted summed to obtain the initial text features to be retrieved.

[0085] Furthermore, the text data to be retrieved can be processed by syntactic analysis, semantic role labeling, etc. to extract the syntactic and semantic structural features of the text (i.e., the initial text features to be retrieved). The syntactic and semantic structural features can reflect the meaning and structure of the text data to be retrieved from different angles. Combined with the word vector features, they can more comprehensively represent the text data to be retrieved. For example, syntactic analysis can identify the sentence components and structure in the text, and semantic role labeling can determine the semantic relationship between each entity in the sentence.

[0086] At the same time, the audio data to be retrieved is segmented according to a preset frame length (e.g., 20-30 milliseconds). Each frame of audio data can be represented as a vector. The features of each segmented audio data to be retrieved are extracted to obtain the initial audio features to be retrieved. The extraction method can be implemented through Mel-frequency cepstral coefficients. The extraction process includes:

[0087] Converting the time domain signal corresponding to each segmented audio data to be retrieved into a frequency domain signal;

[0088] The frequency domain features corresponding to each frequency signal are extracted through the Mel filter bank, and the logarithmic energy corresponding to each filter is determined;

[0089] Perform discrete cosine transform on the logarithmic energy to obtain MFCC coefficients.

[0090] Furthermore, deep learning can be used to extract audio features. For example, convolutional neural networks (CNN) or recurrent neural networks (RNN) and their variants (such as LSTM and GRU) can be used to automatically learn the features of the audio data to be retrieved. The deep learning model can mine the time series features and spectral features in the audio signal and capture the complex patterns and regularities in the audio signal.

[0091] For example, LSTM networks, with their memory units and gating structures, can effectively process long-term dependencies in audio data to be retrieved, demonstrating excellent results for tasks such as speech recognition and music classification. Furthermore, audio data can be denoised to improve the quality and intelligibility of the audio signal. Denoising methods include, but are not limited to, spectral subtraction, wavelet transform denoising, and deep learning denoising. By removing noise interference from the audio, initial features of the audio to be retrieved can be more accurately extracted, improving the quality and usability of the audio data to be retrieved.

[0092] In addition, the mean and standard deviation of the initial image features to be retrieved, the initial text features to be retrieved, and the initial audio features to be retrieved are calculated respectively to convert the initial image features to be retrieved, the initial text features to be retrieved, and the initial audio features to be retrieved into a standard normal distribution. After standardization, the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved are obtained, so that the data to be retrieved of different modalities are comparable on the same feature scale.

[0093] This embodiment realizes the efficient and accurate extraction of multimodal data features through normalization, word vector encoding, frame processing and feature standardization. The feature standardization process improves the robustness of feature data and can maintain stable and reliable performance when facing different types of data to be retrieved and tasks to be retrieved.

[0094] In some embodiments, after the processed data features are mapped to the preset high-dimensional feature space for fusion to obtain the fused data features, the method further comprises:

[0095] inputting the fused data features into a discriminator to obtain probability values corresponding to the fused data features;

[0096] determining the fused data features with probability values greater than a preset probability threshold as real fused features, and determining the fused data features with probability values not greater than the preset probability threshold as generated fused features;

[0097] optimizing the fusion process of the data features based on the real fused features and the generated fused features.

[0098] Optionally, in the embodiment, the fused data features processed and mapped to the preset high-dimensional feature space are inputted into a discriminator, so that the discriminator processes the inputted fused data features, and obtains probability values corresponding to the fused data features through calculation and transformation of multiple layers of neurons in the network.

[0099] The discriminator is an independent neural network model, which can be a multi-layer perceptron (MLP) or other suitable network architecture. In the processing process, the discriminator learns the patterns and rules in the fused data features to determine the authenticity of the fused data features.

[0100] The probability value corresponding to the fused data features represents the probability that the inputted fused data features are real features. The range of the probability value is generally between 0 and 1, i.e., the closer the probability value is to 1, the higher the authenticity of the fused data features, and the closer the probability value is to 0, the lower the authenticity of the fused data features (which can be a generated feature). The probability value of the fused data features is positively correlated with the authenticity.

[0101] In addition, a preset probability threshold is received according to actual needs, for example, 0.5, to distinguish whether the fused data features are real features or generated features according to the preset probability threshold, i.e., if the probability value corresponding to the fused data features is greater than the preset probability threshold, the fused data features are determined as real fused features, and if the probability value corresponding to the fused data features is not greater than the preset probability threshold, the fused data features are determined as generated fused features.

[0102] In addition, a loss function of the fusion process of the fused feature data is determined based on the real fused features and the generated fused features. The loss function can be a binary cross-entropy loss or other suitable loss function, which is used to measure the difference between the fused features and the real features.

[0103] Through a back propagation algorithm, gradient information of the loss function is propagated back to the fusion process of the data features, so that the model parameters in the fusion process can be updated according to the gradient information to reduce the difference between the fused features and the real features.

[0104] Furthermore, in order to prevent overfitting, regularization techniques such as L1 or L2 regularization, dropout, etc. can be introduced into the discriminator and fusion process. Adversarial training can also be performed between the fusion process and the discriminator through the idea of ​​generative adversarial networks (GANs). That is, the goal of the generator is to generate fusion features that can "fool" the discriminator, while the goal of the discriminator is to identify the difference between real features and generated features as accurately as possible. Through the above-mentioned adversarial training mechanism, the quality of fusion features and the performance of the discriminator can be continuously improved.

[0105] This embodiment achieves the goal of effectively improving the quality of fused data features by performing discriminator evaluation on fused data features, classifying and determining feature types, and optimizing the fusion process based on different types of features, making them closer to the distribution and characteristics of real data features, improving the accuracy and recall rate of multimodal data retrieval, and making the retrieval results more in line with the actual needs of users, thereby improving retrieval performance and user experience.

[0106] In some embodiments, the general knowledge representation is input into a retrieval-specific model, and the general knowledge representation is processed based on the retrieval-specific model to obtain retrieval results corresponding to the data to be retrieved, including:

[0107] Input the data features and general knowledge representation into the retrieval-specific model, and fuse the data features and general knowledge representation through the knowledge receiving network layer and the dedicated fusion network layer in the retrieval-specific model to obtain the initial retrieval results;

[0108] The initial retrieval results are processed by the fully connected layer and activation function in the retrieval-specific model to obtain the retrieval results corresponding to the data to be retrieved.

[0109] Optionally, this embodiment uses the extracted data features (including image features to be retrieved, text features to be retrieved, and audio features to be retrieved) and the general knowledge representation output by the general model as the input of the retrieval-specific model, and receives the general knowledge representation through the knowledge receiving network layer in the retrieval-specific model, wherein the knowledge receiving network layer can be one or more neural network layers for performing preliminary processing and conversion on the general knowledge representation.

[0110] At the same time, the knowledge receiving network layer also receives data features. Data features of different modalities can be processed separately, such as feature extraction and conversion through different sub-networks.

[0111] In addition, the general knowledge representation and data features processed by the knowledge receiving network layer are input into the dedicated fusion network layer of the retrieval-specific model, wherein the dedicated fusion network layer deeply fuses the general knowledge representation and data features through a preset fusion strategy to obtain the initial retrieval results, wherein the initial retrieval results combine the comprehensive information of the general knowledge representation of the retrieval general model and the data features of the retrieval-specific model.

[0112] Among them, the preset fusion strategy may include but is not limited to feature splicing, element-by-element addition, and attention fusion. For example, through the attention mechanism, weights are dynamically assigned according to the importance of data features and general knowledge representation to highlight important feature information.

[0113] In addition, the initial retrieval results are input into the fully connected layer of the retrieval-specific model, so that the fully connected layer can perform linear transformation on the initial retrieval results through matrix multiplication and bias addition, map them to a new feature space, and integrate data features of different modalities to extract the most valuable feature combination for the retrieval task. The number of neurons and connection weights of the fully connected layer can be learned and optimized during the training process.

[0114] In addition, after the fully connected layer, by using the activation function to perform nonlinear processing on the initial retrieval results after linear transformation, the activation function can introduce nonlinear factors to obtain more complex feature relationships and patterns.

[0115] Among them, commonly used activation functions include ReLU, sigmoid, and tanh. Different activation functions have different characteristics and applicable scenarios. For example, ReLU can alleviate the gradient disappearance problem, and sigmoid can limit the output to the range of (0,1).

[0116] In addition, according to the specific requirements of the retrieval task corresponding to the data to be retrieved, the initial retrieval results after processing by the activation function can be used for classification or regression output to obtain processed retrieval results. For classification tasks, the softmax function can be used to convert the output into a probability distribution to represent the probability that the data to be retrieved belongs to different categories; for regression tasks, continuous values ​​can be directly output.

[0117] In addition, the retrieval results corresponding to the data to be retrieved are obtained based on the processed retrieval results. For example, in an image retrieval task, the retrieval results can be a list of images most similar to the image to be retrieved. In a text retrieval task, the retrieval results can be the documents most relevant to the text data to be retrieved.

[0118] In some embodiments, adjusting the general model parameters of the general retrieval model and the specific model parameters of the specific retrieval model based on the retrieval results includes:

[0119] Update the dedicated model parameters corresponding to the retrieval dedicated model based on the retrieval results;

[0120] Receive the true label corresponding to the retrieval result, and determine the multi-task loss value between the retrieval result and the true label using the multi-task loss function in the retrieval-specific model, where the multi-task loss function includes a classification cross entropy loss function and a cross-modal contrast loss function;

[0121] The multi-task loss value is back-propagated to the retrieval general model so that the retrieval general model updates the general model parameters based on the multi-task loss value.

[0122] Optionally, this embodiment inputs the data to be retrieved into a retrieval-specific model, including data features and general knowledge representation, obtains initial retrieval results through the knowledge receiving network layer and the dedicated fusion network layer in the retrieval-specific model, and processes the initial retrieval results through the fully connected layer and activation function to obtain the retrieval results.

[0123] Receive the true label corresponding to the retrieval result, determine the single-task loss function, such as the categorical cross entropy loss function, based on the retrieval result and the true label, backpropagate the gradient of the single-task loss function through the retrieval-specific model to determine the gradient of each parameter, and update the dedicated model parameters of the retrieval-specific model through an optimization algorithm (such as Adam, SGD) to minimize the loss function.

[0124] In addition, the multi-task loss functions in the retrieval-specific model include the categorical cross entropy loss function and the cross-modal contrast loss function. The categorical cross entropy loss function is used to measure the difference between the retrieval results and the true labels in the classification task, and the cross-modal contrast loss function is used to measure the similarity between the data to be detected of different modalities, ensuring that the data features of different modalities are well aligned in the high-dimensional space.

[0125] The multi-task loss is calculated by weighted summing the categorical cross-entropy loss function and the cross-modal contrast loss function. This multi-task loss is then back-propagated to the general retrieval model, which then updates the general model parameters based on the multi-task loss. This allows the general retrieval model to generate a more accurate representation of general knowledge during the next forward propagation, which in turn enables the retrieval-specific model to produce higher-quality retrieval results. This process is continuously iterated, resulting in gradual optimization and improvement of the overall multimodal data retrieval task.

[0126] Among them, the multi-task loss value is back-propagated to the general retrieval model through the retrieval-specific model to determine the gradient of the general retrieval model. The gradient is propagated to each layer of the general retrieval model, including the knowledge extraction layer and the feature mapping layer, through the chain rule. The parameters of the general retrieval model are updated through optimization algorithms (such as Adam, SGD, etc.) to minimize the multi-task loss value.

[0127] For example, in image-text retrieval tasks, it is desirable for matching pairs of image and text data to be retrieved to be close in feature space, while for mismatched pairs, they are expected to be far apart. The cross-modal contrastive loss function can calculate the loss value based on this expectation, thereby guiding both general and specialized retrieval models to correctly reflect the multimodal feature representation of semantic relationships.

[0128] Furthermore, during the training process of federated learning, an upper limit of collaborative training rounds and a termination condition can be set. The termination condition can be that the performance of the retrieval-specific model on the validation set reaches a preset threshold. When either the upper limit or the termination condition is reached, the collaborative training is stopped (i.e., the updating of the general model parameters and the dedicated model parameters is stopped).

[0129] Furthermore, the weight coefficients in the multi-task loss function can be dynamically adjusted according to the difficulty and importance of the retrieval task, so that the general retrieval model and the retrieval-specific model can better balance the learning of different tasks.

[0130] In addition, regularization terms can be added to the multi-task loss functions of different retrieval tasks to prevent overfitting of a certain task and improve the overall performance.

[0131] This embodiment realizes the adjustment of the general model parameters of the general retrieval model and the special model parameters of the special retrieval model by optimizing the special model parameter update strategy and the multi-task loss function, which can make the overall model training and optimization more efficient, stable and intelligent, and improve the model's processing ability and retrieval performance for multimodal data from multiple dimensions, making the retrieval results more accurate and reliable, and can better meet the needs of users in multimodal data retrieval scenarios.

[0132] In some embodiments, establishing a multimodal data index based on data features includes:

[0133] Assigning a unique identifier to the same set of data to be retrieved, and establishing an association mapping relationship between the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved based on the unique identifier;

[0134] Based on the association mapping relationship, a graph-structured multimodal data index is constructed. The nodes in the multimodal data index represent different modal features of the same data to be retrieved, and the edges represent the semantic association strength between data features.

[0135] Optionally, in this embodiment, a unique identifier is assigned to each group of data to be retrieved (including image data to be retrieved, text data to be retrieved, and audio data to be retrieved). The unique identifier is used to identify and associate different modal features of the same group of data.

[0136] Among them, the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved corresponding to the same data to be retrieved can be associated through a unique identifier to ensure that the data features of different modalities can accurately correspond to the same data to be retrieved in the subsequent processing process.

[0137] In addition, based on the unique identifier, an association mapping relationship is established between different modal features. For example, a dictionary or hash table is created, using the unique identifier as the key to store the corresponding image features to be retrieved, text features to be retrieved, and audio features to be retrieved.

[0138] In addition, after establishing the association mapping relationship between different modal features, a multimodal data index with a graph structure is further constructed based on the association mapping relationship.

[0139] In the graph structure, nodes represent different modal features of the same data to be retrieved. For example, one node can be the image feature to be retrieved of a certain data sample, another node is the text feature to be retrieved of the same data sample, and another node is the corresponding audio feature to be retrieved. The edges represent the semantic association strength between data features, reflecting the mutual relationship and similarity of different modal features at the semantic level.

[0140] Among them, the semantic association strength can be determined in a variety of ways, such as calculating the similarity between features based on a pre-trained semantic model, or assigning an initial value of the semantic association strength through manual labeling, and then optimizing and adjusting it in combination with a machine learning algorithm.

[0141] Furthermore, the constructed graph structure (i.e., multimodal data index) can be stored in efficient data structures, such as adjacency lists or adjacency matrices, for fast query and retrieval.

[0142] Furthermore, the graph structure can be indexed and optimized, for example, by using a graph database (such as Neo4j) or a distributed graph computing framework (such as Apache Giraph) to improve query efficiency.

[0143] This embodiment realizes the construction of an efficient, accurate, intelligent and adaptable multimodal data index by generating unique identifiers and enhancing the representation of association mapping relationships, thereby being able to effectively associate and manage different modal data features of multimodal data, and to mine and utilize their semantic association relationships, thereby providing support for the retrieval, analysis and application of multimodal data, and improving the efficiency and accuracy of data retrieval.

[0144] In some embodiments, knowledge extraction is performed on the fused data features by retrieving the knowledge extraction layer of the general model to obtain a general knowledge representation, including:

[0145] By retrieving the middle hidden layer of the general model, knowledge is extracted from the fused data features to obtain multimodal shared features;

[0146] The multimodal shared features are screened through the gating mechanism in the knowledge extraction layer to obtain the initial general knowledge representation, and the initial general knowledge representation is regularized and constrained to obtain the general knowledge representation.

[0147] Optionally, this embodiment retrieves the middle hidden layer of the general model for knowledge extraction during the process of processing the fused data features, wherein the middle hidden layer is usually composed of a multi-layer neural network, such as a multi-layer perceptron (MLP) or a convolutional neural network (CNN) structure.

[0148] After the fused data features are input into the middle hidden layer, the multimodal shared features in the fused data features are gradually extracted through linear and nonlinear transformations. The multimodal shared features are used to represent data features that can simultaneously reflect the common semantic information of the data to be detected in different modalities, that is, the intersection and common parts of the data features of different modalities at the semantic level.

[0149] For example, in a multimodal data set containing image data to be detected, text data to be detected, and audio data to be detected, the semantic information describing a "cheerful scene" may be reflected in the bright colors of the image, the positive words in the text, and the cheerful tones in the audio. Through learning and transformation, the middle hidden layer can extract features related to "cheerful scenes" from these different modalities to form multimodal shared features.

[0150] In addition, in the knowledge extraction layer, a gating mechanism can be introduced to screen the extracted multimodal shared features, where the gating mechanism can dynamically assign weights to the multimodal shared features based on the importance of the features or their relevance to the target task.

[0151] For example, a gating structure similar to that in the long short-term memory network (LSTM) is used to control the flow of information and the screening of features through the input gate, forget gate, and output gate. The input gate is used to determine new feature information that can be incorporated into the general knowledge representation, the forget gate is used to determine existing feature information that needs to be discarded, and the output gate is used to control the feature composition of the output general knowledge representation.

[0152] Through the gating mechanism, it is possible to ensure that only features that contribute significantly to the general knowledge representation are retained, while removing redundant or irrelevant feature information.

[0153] In addition, after obtaining the initial general knowledge representation, regularization constraints are further applied to it. The purpose of the regularization constraints is to prevent the model from overfitting and improve the generalization ability of the model.

[0154] Common regularization methods include L1 regularization, L2 regularization, and dropout, etc. to constrain the initial general knowledge representation through the regularization method to obtain a general knowledge representation with strong generalization ability and good robustness.

[0155] The embodiment realizes that the performance and effect of the retrieval general model in the knowledge extraction process can be improved through the structure optimization, gating mechanism and regularization constraint of the intermediate hidden layer, the intermediate hidden layer can more efficiently extract multi-modal shared features, the gating mechanism can more accurately filter features and generate an initial general knowledge representation, and the regularization constraint can further improve the quality and generalization ability of the general knowledge representation, thereby generating a more accurate, more robust and more generalizable general knowledge representation, improving the accuracy and recall rate of multi-modal data retrieval, better meeting the needs of users in complex multi-modal data retrieval scenarios, and improving user experience and overall performance.

[0156] Further, the multi-modal general-special model collaborative training method based on federated learning provided in the present application can use the same encryption technology, that is, the received general model parameters and special model parameters are encrypted, so that the server can calculate in the ciphertext state during the aggregation of the model parameters, protecting the data privacy.

[0157] A differential privacy mechanism can also be added to add noise to the general model parameters and special model parameters, so that an attacker cannot infer the information of the original data by analyzing the general model parameters and special model parameters.

[0158] A data access control mechanism can also be established to strictly manage the access rights of the client data (to-be-detected data, general model parameters and special model) participating in federated learning, and only authorized operations can access the data.

[0159] Multi-party secure computation technology can also be used to realize secure computation between any two ends of the client and the server without revealing the data privacy, such as the aggregation of the model parameters.

[0160] The privacy protection parameters can also be updated regularly, and the privacy protection parameters such as the key of homomorphic encryption and the noise level of differential privacy are adjusted according to the dynamic changes of the data and the security requirements.

[0161] In order to effectively solve the deficiencies of the traditional technology in the process of federated learning, such as the difficulty in effectively fusing multi-modal heterogeneous data, the difficulty in fusion, etc., significantly improve the generalization ability of the multi-modal general-special model under federated learning, and better fuse the to-be-retrieved data, the present application provides an embodiment of a multi-modal general-special model collaborative training system based on federated learning for implementing all or part of the contents of the multi-modal general-special model collaborative training method based on federated learning. Figure 2The multimodal communication model collaborative training device based on federated learning specifically includes the following contents:

[0162] A first processing module 10 is configured to receive data to be retrieved, extract data features of the data to be retrieved, and establish a multimodal data index based on the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved, and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved, and audio features to be retrieved;

[0163] A second processing module 20 is configured to process data features based on the multimodal data index through an attention mechanism in a preset cross-modal feature mapping network, and map the processed data features to a preset high-dimensional feature space for fusion to obtain fused data features, thereby fusing the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved;

[0164] The third processing module 30 is used to input the fused data features into the retrieval general model, perform knowledge extraction on the fused data features through the knowledge extraction layer of the retrieval general model, and obtain a general knowledge representation;

[0165] The fourth processing module 40 is used to input the general knowledge representation into the retrieval-specific model, process the general knowledge representation based on the retrieval-specific model, obtain the retrieval results corresponding to the data to be retrieved, and adjust the general model parameters of the retrieval general model and the specific model parameters of the retrieval-specific model based on the retrieval results.

[0166] From the above description, it can be seen that the multimodal communication model collaborative training device based on federated learning provided in the embodiment of the present application can innovatively receive the data to be retrieved, extract the data features of the data to be retrieved, and establish a multimodal data index based on the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved and audio features to be retrieved. According to the multimodal data index, the data features are processed by the attention mechanism in the preset cross-modal feature network, mapped to the high-order feature space for fusion, and fused data features are generated. The fused data features are input into The knowledge extraction layer of the general retrieval model obtains a general knowledge representation, which is then input into the retrieval-specific model for processing to obtain retrieval results. Based on the retrieval results, the general model parameters of the general retrieval model and the specific model parameters of the specific retrieval model are adjusted, thereby improving the efficiency and accuracy of multimodal data fusion. The cross-modal feature mapping network and attention mechanism effectively unify the heterogeneous feature space of the data to be retrieved, and optimize the collaborative training capabilities of the specific retrieval model and the general retrieval model. Through layered knowledge extraction and joint parameter adjustment, knowledge sharing between the specific retrieval model and the general retrieval model is achieved, improving generalization capabilities and convergence speed. This method effectively addresses the shortcomings of traditional technologies in effectively fusing multimodal heterogeneous data during federated learning, as well as in terms of fusion difficulties. It significantly improves the generalization capabilities of multimodal general-purpose models under federated learning and better integrates the data to be retrieved.

[0167] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in effectively integrating multimodal heterogeneous data during federated learning, significantly improve the generalization capability of the multimodal general-purpose model under federated learning, and better integrate the data to be retrieved, the present application provides an embodiment of an electronic device for implementing all or part of the content of the multimodal general-purpose model collaborative training method based on federated learning, and the electronic device specifically includes the following content:

[0168] A processor, a memory, a communications interface, and a bus; wherein the processor, the memory, and the communications interface communicate with each other via the bus; the communications interface is used to implement information transmission between a multimodal communication model collaborative training device based on federated learning and related devices such as core business systems, user terminals, and related databases; the logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., but this embodiment is not limited thereto. In this embodiment, the logic controller can be implemented with reference to the embodiment of the multimodal communication model collaborative training method based on federated learning and the embodiment of the multimodal communication model collaborative training device based on federated learning in the embodiment, and their contents are incorporated herein, and repeated parts are not repeated.

[0169] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0170] In practical applications, part of the collaborative training method for a multimodal communication model based on federated learning can be performed on the electronic device side as described above, or all operations can be completed on the client device. The specific selection can be based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed on the client device, the client device may also include a processor.

[0171] The aforementioned client device may include a communication module (i.e., a communication unit) capable of establishing a communication connection with a remote server to facilitate data transmission with the server. The server may include a server at the task scheduling center or, in other implementation scenarios, a server on an intermediate platform, such as a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may comprise a single computer device, a server cluster consisting of multiple servers, or a server structure comprising a distributed system.

[0172] Figure 3 Schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that the Figure 3 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0173] In one embodiment, the function of the multimodal communication model collaborative training method based on federated learning can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0174] Step S101: receiving data to be retrieved, extracting data features of the data to be retrieved, and establishing a multimodal data index based on the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved, and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved, and audio features to be retrieved;

[0175] Step S102: Based on the multimodal data index, the data features are processed by the attention mechanism in the preset cross-modal feature mapping network, and the processed data features are mapped to a preset high-dimensional feature space for fusion to obtain fused data features, so as to fuse the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved;

[0176] Step S103: Input the fused data features into the general retrieval model, and perform knowledge extraction on the fused data features through the knowledge extraction layer of the general retrieval model to obtain a general knowledge representation;

[0177] Step S104: input the general knowledge representation into the retrieval-specific model, process the general knowledge representation based on the retrieval-specific model, obtain the retrieval results corresponding to the data to be retrieved, and adjust the general model parameters of the retrieval general model and the specific model parameters of the retrieval-specific model based on the retrieval results.

[0178] As can be seen from the above description, the electronic device provided in the embodiment of the present application innovatively receives data to be retrieved, extracts data features of the data to be retrieved, and establishes a multimodal data index based on the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved, and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved, and audio features to be retrieved. According to the multimodal data index, the data features are processed by the attention mechanism in the preset cross-modal feature network, mapped to the high-order feature space for fusion, and fused data features are generated. The fused data features are input into the knowledge extraction layer of the general retrieval model to obtain a general knowledge representation, and the general knowledge representation is input into the retrieval-specific model for processing to obtain a retrieval result, and the general model parameters of the general retrieval model and the special model parameters of the retrieval-specific model are adjusted according to the retrieval result, thereby improving the fusion efficiency and accuracy of the multimodal data. The heterogeneous feature space of the data to be retrieved is effectively unified through the cross-modal feature mapping network and the attention mechanism, and the collaborative training capability of the retrieval-specific model and the retrieval general model is optimized. The knowledge sharing of the retrieval-specific model and the retrieval general model is realized by the hierarchical knowledge extraction and joint parameter adjustment, thereby improving the generalization ability and convergence speed. This method effectively solves the shortcomings of traditional technologies in the process of federated learning, such as the difficulty in effectively integrating multimodal heterogeneous data and the difficulty in integration. It significantly improves the generalization ability of multimodal general-purpose models under federated learning and better integrates the data to be retrieved.

[0179] In another embodiment, the collaborative training method of the multimodal communication model based on federated learning can be configured separately from the central processing unit 9100. For example, the collaborative training method of the multimodal communication model based on federated learning can be configured as a chip connected to the central processing unit 9100, and the function of the collaborative training method of the multimodal communication model based on federated learning can be realized through the control of the central processing unit.

[0180] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Figure 3 In addition, the electronic device 9600 may also include all components shown in Figure 3 For components not shown, reference may be made to the prior art.

[0181] like Figure 3 As shown, the central processing unit 9100 is sometimes also referred to as a controller or operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0182] Memory 9140 can be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It can store the aforementioned failure-related information and also store programs that execute the relevant information. The CPU 9100 can execute the programs stored in memory 9140 to implement information storage or processing.

[0183] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 may be, for example, a keypad or touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0184] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), or SIM card. Alternatively, it may be a memory that retains information even when power is off, can be selectively erased, and is capable of storing additional data. Examples of such memory are sometimes referred to as EPROMs. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs, or processes used by the central processing unit 9100 to execute operations of the electronic device 9600.

[0185] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, images, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various driver programs for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0186] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0187] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless local area network modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130, providing audio output via the speaker 9131 and receiving audio input from the microphone 9132, thereby implementing common telecommunication functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 9130 is coupled to the central processing unit 9100, enabling local recording via the microphone 9132 and playback of stored audio via the speaker 9131.

[0188] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the method for collaborative training of a multimodal communication model based on federated learning in the above-mentioned embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all steps of the method for collaborative training of a multimodal communication model based on federated learning in the above-mentioned embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0189] Step S101: receiving data to be retrieved, extracting data features of the data to be retrieved, and establishing a multimodal data index based on the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved, and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved, and audio features to be retrieved;

[0190] Step S102: Based on the multimodal data index, the data features are processed by the attention mechanism in the preset cross-modal feature mapping network, and the processed data features are mapped to a preset high-dimensional feature space for fusion to obtain fused data features, so as to fuse the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved;

[0191] Step S103: Input the fused data features into the general retrieval model, and perform knowledge extraction on the fused data features through the knowledge extraction layer of the general retrieval model to obtain a general knowledge representation;

[0192] Step S104: input the general knowledge representation into the retrieval-specific model, process the general knowledge representation based on the retrieval-specific model, obtain the retrieval results corresponding to the data to be retrieved, and adjust the general model parameters of the retrieval general model and the specific model parameters of the retrieval-specific model based on the retrieval results.

[0193] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application innovatively receives the data to be retrieved, extracts the data features of the data to be retrieved, and establishes a multimodal data index based on the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved, and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved, and audio features to be retrieved. According to the multimodal data index, the data features are processed by the attention mechanism in the preset cross-modal feature network, mapped to the high-order feature space for fusion, and fused data features are generated, and the fused data features are input into the retrieval general model The knowledge extraction layer is used to obtain a general knowledge representation, which is then input into a retrieval-specific model for processing to obtain retrieval results. The general model parameters of the general retrieval model and the special model parameters of the special retrieval model are adjusted according to the retrieval results, thereby improving the fusion efficiency and accuracy of multimodal data. The heterogeneous feature space of the data to be retrieved is effectively unified through the cross-modal feature mapping network and attention mechanism, and the collaborative training capabilities of the special retrieval model and the general retrieval model are optimized. Knowledge sharing between the special retrieval model and the general retrieval model is achieved through layered knowledge extraction and joint parameter adjustment, thereby improving generalization ability and convergence speed. This method effectively solves the shortcomings of traditional technologies in the process of federated learning, such as the difficulty of effectively fusing multimodal heterogeneous data and the difficulty of fusion. It significantly improves the generalization ability of the multimodal general-purpose model under federated learning and better integrates the data to be retrieved.

[0194] The embodiments of the present application also provide a computer program product capable of implementing all steps of the method for collaboratively training a multimodal communication model based on federated learning in the above-mentioned embodiment, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the method for collaboratively training a multimodal communication model based on federated learning are implemented. For example, the computer program / instruction implements the following steps:

[0195] Step S101: receiving data to be retrieved, extracting data features of the data to be retrieved, and establishing a multimodal data index based on the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved, and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved, and audio features to be retrieved;

[0196] Step S102: Based on the multimodal data index, the data features are processed by the attention mechanism in the preset cross-modal feature mapping network, and the processed data features are mapped to a preset high-dimensional feature space for fusion to obtain fused data features, so as to fuse the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved;

[0197] Step S103: Input the fused data features into the general retrieval model, and perform knowledge extraction on the fused data features through the knowledge extraction layer of the general retrieval model to obtain a general knowledge representation;

[0198] Step S104: input the general knowledge representation into the retrieval-specific model, process the general knowledge representation based on the retrieval-specific model, obtain the retrieval results corresponding to the data to be retrieved, and adjust the general model parameters of the retrieval general model and the specific model parameters of the retrieval-specific model based on the retrieval results.

[0199] From the above description, it can be seen that the computer program product provided in the embodiment of the present application innovatively receives the data to be retrieved, extracts the data features of the data to be retrieved, and establishes a multimodal data index based on the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved, and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved, and audio features to be retrieved. According to the multimodal data index, the data features are processed by the attention mechanism in the preset cross-modal feature network, mapped to the high-order feature space for fusion, and fused data features are generated. The fused data features are input into the retrieval general model. The knowledge extraction layer obtains a general knowledge representation, which is then input into a retrieval-specific model for processing to obtain retrieval results. Based on the retrieval results, the general model parameters of the general retrieval model and the specific model parameters of the specific retrieval model are adjusted, thereby improving the efficiency and accuracy of multimodal data fusion. The cross-modal feature mapping network and attention mechanism effectively unify the heterogeneous feature space of the data to be retrieved, and optimize the collaborative training capabilities of the specific retrieval model and the general retrieval model. Through layered knowledge extraction and joint parameter adjustment, knowledge sharing between the specific retrieval model and the general retrieval model is achieved, improving generalization capabilities and convergence speed. This method effectively addresses the shortcomings of traditional technologies in effectively fusing multimodal heterogeneous data during federated learning, as well as in terms of fusion difficulties. It significantly improves the generalization capabilities of multimodal general-purpose models under federated learning and better integrates the data to be retrieved.

[0200] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatuses, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0201] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0204] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A collaborative training method for multimodal communication models based on federated learning, characterized in that: include: receiving data to be retrieved, extracting data features of the data to be retrieved, and establishing a multimodal data index based on the data features, wherein the data to be retrieved includes image data to be retrieved, text data to be retrieved, and audio data to be retrieved, and the data features include image features to be retrieved, text features to be retrieved, and audio features to be retrieved; Based on multimodal data indexing, data features are processed through an attention mechanism in a preset cross-modal feature mapping network, and the processed data features are mapped to a preset high-dimensional feature space for fusion to obtain fused data features, thereby fusing the image features to be retrieved, the text features to be retrieved, and the audio features to be retrieved; Input the fused data features into the general retrieval model, and extract knowledge from the fused data features through the knowledge extraction layer of the general retrieval model to obtain a general knowledge representation; The general knowledge representation is input into the retrieval-specific model, the general knowledge representation is processed based on the retrieval-specific model to obtain retrieval results corresponding to the data to be retrieved, and the general model parameters of the retrieval general model and the special model parameters of the retrieval-specific model are adjusted based on the retrieval results.

2. The method according to claim 1, characterized in that The receiving of the data to be retrieved and extracting data features of the data to be retrieved includes: Normalizing the image data to be retrieved, and mapping the pixel values ​​corresponding to the normalized image data to be retrieved to a preset pixel interval to obtain initial image features to be retrieved; Performing word vector encoding on the text data to be retrieved to obtain initial features of the text to be retrieved; Segmenting the audio data to be retrieved according to a preset frame length, and extracting features of each segmented audio data to be retrieved to obtain initial audio features to be retrieved; The initial image features to be retrieved, the initial text features to be retrieved, and the initial audio features to be retrieved are subjected to feature normalization to convert the initial image features to be retrieved, the initial text features to be retrieved, and the initial audio features to be retrieved into a standard normal distribution, thereby obtaining image features to be retrieved, text features to be retrieved, and audio features to be retrieved.

3. The method according to claim 1, characterized in that After mapping the processed data features to a preset high-dimensional feature space for fusion to obtain fused data features, the method further includes: Inputting the fused data features into a discriminator to obtain a probability value corresponding to the fused data features; Determine the fused data feature whose probability value is greater than the preset probability threshold as the true fused feature, and determine the fused data feature whose probability value is not greater than the preset probability threshold as the generated fused feature; The fusion process of the data features is optimized based on the real fusion features and the generated fusion features.

4. The method according to claim 1, wherein The step of inputting the general knowledge representation into the retrieval-specific model and processing the general knowledge representation based on the retrieval-specific model to obtain retrieval results corresponding to the data to be retrieved includes: Inputting the data features and the general knowledge representation into the retrieval-specific model, and fusing the data features and the general knowledge representation through the knowledge receiving network layer and the dedicated fusion network layer in the retrieval-specific model to obtain an initial retrieval result; The initial search result is processed through the fully connected layer and activation function in the search-specific model to obtain a search result corresponding to the data to be searched.

5. The method according to claim 1, wherein The adjusting of the general model parameters of the general retrieval model and the special model parameters of the special retrieval model based on the retrieval results includes: updating the dedicated model parameters corresponding to the retrieval dedicated model based on the retrieval result; Receiving a true label corresponding to the retrieval result, and determining a multi-task loss value between the retrieval result and the true label using a multi-task loss function in the retrieval-specific model, wherein the multi-task loss function includes a classification cross entropy loss function and a cross-modal contrast loss function; The multi-task loss value is back-propagated to the retrieval general model so that the retrieval general model updates the general model parameters based on the multi-task loss value.

6. The method according to claim 1, characterized in that The multimodal data index is established based on the data features, including: Assigning a unique identifier to the same group of the to-be-retrieved data, and establishing, based on the unique identifier, an association mapping relationship among the to-be-retrieved image features, the to-be-retrieved text features, and the to-be-retrieved audio features; Based on the association mapping relationship, a multimodal data index with a graph structure is constructed, wherein the nodes in the multimodal data index represent different modal features of the same data to be retrieved, and the edges represent the semantic association strength between the data features.

7. The method according to claim 1, characterized in that The process of extracting knowledge from the fused data features by retrieving the knowledge extraction layer of the general model to obtain a general knowledge representation includes: Performing knowledge extraction on the fused data features through the middle hidden layer of the general retrieval model to obtain multimodal shared features; The multimodal shared features are filtered through a gating mechanism in the knowledge extraction layer to obtain an initial universal knowledge representation, and regularization constraints are performed on the initial universal knowledge representation to obtain a universal knowledge representation.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the multimodal communication model collaborative training method based on federated learning described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal communication model collaborative training method based on federated learning described in any one of claims 1 to 7 are implemented.

10. A computer program product for a computer program / instruction, characterized in that When the computer program / instruction is executed by a processor, the steps of the multimodal communication model collaborative training method based on federated learning described in any one of claims 1 to 7 are implemented.

Images