Universal feature extraction network training method, device and universal feature extraction network

Abstract

The present invention discloses a universal feature extraction network training method, device and universal feature extraction network, the method comprising: inputting the training data in the training data pair into the universal feature extraction network for feature extraction to obtain data features; inputting the data features into the reconstruction network of the corresponding modality for reconstruction to obtain reconstructed data, calculating the loss function between the reconstructed data output by the reconstruction network and the corresponding training data; determining the training loss function according to the loss functions corresponding to all the reconstruction networks, repeating the above steps, optimizing the model parameters of the multi-modal universal training model until the training loss function converges, and obtaining the trained universal feature extraction network. It can be seen that the universal feature extraction network trained by the scheme of the present invention can be used to extract features of data of multiple modalities to improve the versatility and efficiency of the neural network algorithm.

Description

Technical Field

[0001] The present invention relates to the field of neural network algorithms, and in particular to a general feature extraction network training method, device and a general feature extraction network. Background Art

[0002] With the development of neural network algorithm technology, neural network algorithms are increasingly used in data processing and prediction in various fields. This trend also poses challenges to the data processing modality of neural networks. How to cope with increasingly diverse data modalities more efficiently and conveniently has become one of the important issues that need to be solved in the field of neural network algorithms.

[0003] The neural network models of the existing technology can generally only solve the problem of single-modal data processing. When dealing with multi-modal data processing, they can generally only perform task classification and use different models to process data of different modalities. A relatively mature general solution has not yet been proposed. Therefore, the existing technology has defects that urgently need to be solved. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for determining the training of a universal feature extraction network, which can train a universal feature extraction network that can be used to extract features of data of multiple modalities, so as to improve the versatility and efficiency of the neural network algorithm.

[0005] In order to solve the above technical problems, the first aspect of the present invention discloses a universal feature extraction network training method, which is applicable to training a multimodal universal training model, wherein the multimodal universal training model includes a universal feature extraction network and at least two reconstruction networks for reconstructing data of different modalities; the method includes:

[0006] Determine a training data pair; the training data pair includes at least two corresponding training data of different modalities;

[0007] Inputting the training data in the training data pair into the universal feature extraction network for feature extraction to obtain data features;

[0008] Input the data features into the reconstruction network of the corresponding modality for reconstruction to obtain reconstructed data, and calculate the loss function between the reconstructed data output by the reconstruction network and the corresponding training data;

[0009] Determine the training loss function according to the loss functions corresponding to all the reconstructed networks, repeat the above steps, optimize the model parameters of the multimodal universal training model until the training loss function converges, and obtain the trained universal feature extraction network.

[0010] As an optional embodiment, in the first aspect of the present invention, the general feature extraction network includes multiple encoders; and / or the reconstruction network includes a single modality reconstruction network and / or a multimodal feature mapping network; the multimodal feature mapping network is used to map the data features of multiple modalities to the same feature space to obtain multiple mapped reconstructed data.

[0011] As an optional embodiment, in the first aspect of the present invention, the modality includes at least one of a data type modality, a data structure modality, and a data source modality; and / or the data type modality is an image modality, a numerical modality, a text modality, a symbolic modality, a video modality, an audio modality, or a time series modality; and / or the data structure modality is a data set modality, a data tree modality, or a graph data modality.

[0012] As an optional implementation, in the first aspect of the present invention, determining the training loss function according to the loss functions corresponding to all the reconstructed networks includes:

[0013] Determining a first loss function between the reconstructed data output by one or more of the single modality reconstruction networks and the corresponding training data as a training loss function;

[0014] and / or,

[0015] Determining a second loss function between the plurality of mapping reconstruction data output by one or more of the multimodal feature mapping networks as a training loss function;

[0016] and / or,

[0017] The first loss function of one or more of the single modality reconstruction networks and the second loss function of one or more of the multimodal feature mapping networks are determined as training loss functions.

[0018] As an optional implementation, in the first aspect of the present invention, the first loss function includes at least one of a cross entropy loss function, an L1 loss function, and an L2 loss function; and / or the second loss function is a similarity function.

[0019] As an optional implementation, in the first aspect of the present invention, the multimodal universal training model further includes at least two preprocessing modules for vectorizing data of different modalities; the step of inputting the training data in the training data pair into the universal feature extraction network for feature extraction to obtain data features includes:

[0020] Inputting the training data in the training data pair into the preprocessing module of the corresponding mode respectively for vectorization to obtain a data vector;

[0021] The data vector is input into the general feature extraction network for feature extraction to obtain data features.

[0022] As an optional implementation, in the first aspect of the present invention, before inputting the training data in the training data pair into the general feature extraction network for feature extraction to obtain data features, the method further includes:

[0023] The training data belonging to the priority modality in the training data pair is input into the general feature extraction network and the reconstruction network for reconstructing the priority modality data for pre-training until convergence.

[0024] A second aspect of the present invention discloses a universal feature extraction network training device, which is suitable for training a multimodal universal training model, wherein the multimodal universal training model includes a universal feature extraction network and at least two reconstruction networks for reconstructing data of different modalities; the device includes:

[0025] A first determination module is used to determine a training data pair; the training data pair includes at least two corresponding training data of different modalities;

[0026] A first input module, used for inputting the training data in the training data pair into the general feature extraction network for feature extraction to obtain data features;

[0027] A second input module is used to input the data features into the reconstruction network of the corresponding modality to reconstruct and obtain reconstructed data, and calculate the loss function between the reconstructed data output by the reconstruction network and the corresponding training data;

[0028] A second determination module is used to determine a training loss function according to the loss functions corresponding to all the reconstructed networks;

[0029] A training module is used to trigger the first determination module, the first input module and the second input module to repeatedly perform the above operations, optimize the model parameters of the multimodal universal training model until the training loss function converges, and obtain the trained universal feature extraction network.

[0030] As an optional embodiment, in the second aspect of the present invention, the general feature extraction network includes multiple encoders; and / or the reconstruction network includes a single modality reconstruction network and / or a multimodal feature mapping network; the multimodal feature mapping network is used to map the data features of multiple modalities to the same feature space to obtain multiple mapped reconstructed data.

[0031] As an optional embodiment, in the second aspect of the present invention, the modality includes at least one of a data type modality, a data structure modality, and a data source modality; and / or the data type modality is an image modality, a numerical modality, a text modality, a symbolic modality, a video modality, an audio modality, or a time series modality; and / or the data structure modality is a data set modality, a data tree modality, or a graph data modality.

[0032] As an optional implementation, in the second aspect of the present invention, the second determination module determines a specific manner of the training loss function according to the loss functions corresponding to all the reconstructed networks, including:

[0033] Determining a first loss function between the reconstructed data output by one or more of the single modality reconstruction networks and the corresponding training data as a training loss function;

[0034] and / or,

[0035] Determining a second loss function between the plurality of mapping reconstruction data output by one or more of the multimodal feature mapping networks as a training loss function;

[0036] and / or,

[0037] The first loss function of one or more of the single modality reconstruction networks and the second loss function of one or more of the multimodal feature mapping networks are determined as training loss functions.

[0038] As an optional implementation, in the second aspect of the present invention, the first loss function includes at least one of a cross entropy loss function, an L1 loss function, and an L2 loss function; and / or the second loss function is a similarity function.

[0039] As an optional implementation, in the second aspect of the present invention, the multimodal universal training model further includes at least two preprocessing modules for vectorizing data of different modalities; the first input module inputs the training data in the training data pair into the universal feature extraction network for feature extraction to obtain data features in a specific manner, including:

[0040] Inputting the training data in the training data pair into the preprocessing module of the corresponding mode respectively for vectorization to obtain a data vector;

[0041] The data vector is input into the general feature extraction network for feature extraction to obtain data features.

[0042] As an optional implementation, in the second aspect of the present invention, the device further includes:

[0043] A pre-training module is used to input the training data belonging to the priority modality in the training data pair into the general feature extraction network and the reconstruction network for reconstructing the priority modality data for pre-training until convergence before the first input module inputs the training data in the training data pair into the general feature extraction network for feature extraction to obtain data features.

[0044] The third aspect of the present invention discloses another general feature extraction network training device, the device comprising:

[0045] A memory storing executable program code;

[0046] a processor coupled to the memory;

[0047] The processor calls the executable program code stored in the memory to execute part or all of the steps in the general feature extraction network training method disclosed in the first aspect of the embodiment of the present invention.

[0048] The fourth aspect of an embodiment of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the general feature extraction network training method disclosed in the first aspect of the embodiment of the present invention.

[0049] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0050] In an embodiment of the present invention, the training data in the training data pair is input into a universal feature extraction network for feature extraction to obtain data features; the data features are input into a reconstruction network of the corresponding modality for reconstruction to obtain reconstructed data, and the loss function between the reconstructed data output by the reconstruction network and the corresponding training data is calculated; the training loss function is determined according to the loss functions corresponding to all the reconstruction networks, and the above steps are repeated to optimize the model parameters of the multimodal universal training model until the training loss function converges to obtain the trained universal feature extraction network. It can be seen that the present invention can train the model through data of multiple modalities, and determine the training loss function of the model through different loss functions corresponding to reconstruction networks of multiple different modalities, thereby training a universal feature extraction network that can be used to extract features of data of multiple modalities, so as to improve the versatility and efficiency of the neural network algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 It is a flowchart of a general feature extraction network training method disclosed in an embodiment of the present invention;

[0053] Figure 2 It is a flowchart of another general feature extraction network training method disclosed in an embodiment of the present invention;

[0054] Figure 3 It is a structural schematic diagram of a general feature extraction network training device disclosed in an embodiment of the present invention;

[0055] Figure 4 It is a structural schematic diagram of another general feature extraction network training device disclosed in an embodiment of the present invention;

[0056] Figure 5 It is a structural schematic diagram of another general feature extraction network training device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or ends.

[0059] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0060] The present invention discloses a universal feature extraction network training method, device and universal feature extraction network, which can train a model through data of multiple modalities, and determine the training loss function of the model through different loss functions corresponding to reconstruction networks of multiple different modalities, thereby training a universal feature extraction network that can be used to extract features of data of multiple modalities, so as to improve the versatility and efficiency of the neural network algorithm. The following are detailed descriptions respectively.

[0061] Embodiment 1

[0062] See also Figure 1 , Figure 1 : is a flow chart of a general feature extraction network training method disclosed in an embodiment of the present invention. Figure 1 The described method is applied to a neural network model training device, which can be a corresponding training terminal, training equipment or training server, and the server can be a local server or a cloud server, which is not limited in the embodiments of the present invention.

[0063] Specifically, the method is suitable for training a multimodal general training model, which may include a general feature extraction network and at least two reconstruction networks for reconstructing data of different modalities. The method can be applied to train the multimodal general training model until convergence to obtain a trained general feature extraction network, which can then be applied to extract features from data of various modalities and further process features. Figure 1 As shown, the general feature extraction network training method may include the following operations:

[0064] 101. Determine training data pairs.

[0065] Optionally, each training data pair may include at least two training data of different modalities corresponding to each other. Optionally, the modality described in the present invention may include at least one of a data type modality, a data structure modality, and a data source modality. For example, it can be a single type of modality, or a combination of multiple types of modalities. Optionally, when the modality refers to a data type modality, it can be an image modality, a numerical modality, a text modality, a symbolic modality, a video modality, an audio modality, or a time series modality, that is, data of different modalities can be data of the above data types. Optionally, when the modality refers to a data structure modality, it can be a data set modality, a data tree modality, or a graph data modality, that is, data of different modalities can be data of the above data structures. Optionally, when the modality refers to a data source modality, it can be data from different data acquisition devices, such as data from different sensors, or data from data acquisition devices at different locations, and the present invention is not limited here.

[0066] Optionally, the mutual correspondence described in the present invention refers to mutual correspondence in meaning, that is, the objects referred to or the meanings contained in the training data of all modalities included in the same training data pair are consistent. For example, in the same training data pair, the training data of an image modality is an image including a red apple, and the training data of another text modality can be "RedApple" or a text of "red apple" in different languages, or there is also an audio modality The training data is the artificial voice data of "this is a red apple". Optionally, the correspondence between different training data in the training data pair can be collected from some existing data with a corresponding relationship. For example, in some online articles, a description of the picture in the article is attached, and the picture and the description can be collected as a training data pair, or mined from some public large-scale image classification data sets, or can be the result of manual annotation, and the present invention is not limited.

[0067] In a specific embodiment, the method of the present invention can be used to train a feature extraction network that is commonly used for image modal data and text modal data. The training data pair should include two corresponding image training data and text training data. Those skilled in the art can train feature extraction networks of other modalities by analogy. In this specific embodiment, the advantage of the method of the present invention is that the field of NLP (natural language processing) mainly uses RNN, LSTM, BERT, etc. to process text data, while the field of image processing uses various model structures designed by convolutional neural networks to cope with visual tasks. At the same time, the semantic density of images and texts is inconsistent. Images are natural signals with many redundant features, while texts are high semantic and high information density signals generated by humans. Therefore, the training strategies of the two are inconsistent. Therefore, the differences in model structure and training strategies between images and texts in the prior art make it difficult to design a universal model structure. This specific implementation scheme can attempt to solve this problem and propose a universal feature extraction network.

[0068] 102. Input the training data in the training data pair into a general feature extraction network for feature extraction to obtain data features.

[0069] Optionally, the general feature extraction network may include multiple encoders, for example, it may be a stacked structure of multiple encoders, wherein the encoder may be an encoder block in a Transformer network structure. Specifically, the structure of the general feature extraction network here may refer to the Transformer Encoder structure in the VIT (Vision Transformer) structure.

[0070] 103. Input the data features into the reconstruction network of the corresponding modality for reconstruction to obtain reconstructed data, and calculate the loss function between the reconstructed data output by the reconstruction network and the corresponding training data.

[0071] Optionally, the role of the reconstruction network is to reconstruct data features to form reconstructed data, and its overall role is similar to the decoder structure in the existing neural network model, but corresponding to data of different modalities, its specific structure can be modified accordingly. Optionally, the reconstruction network can include a single modality reconstruction network and / or a multimodal feature mapping network.

[0072] Among them, the single modality reconstruction network is used to reconstruct the data features of a single modality. For example, when reconstructing the data features of the image modality, the single modality reconstruction network may include several decoder blocks in the Transformer network structure, a LayerNorm layer for data normalization, and a fully connected layer, and then the L2 loss function between the reconstructed image data and the training data can be calculated. When reconstructing the data features of the text modality, for example, the single modality reconstruction network can be a BERT network decoder that performs a text prediction task, wherein, optionally, the text prediction task may include an MLM (Masked Language Model) task and / or a sentence reconstruction task.

[0073] Specifically, the BERT network decoder can perform the MLM task, select the first proportion (such as 15%) of characters in the input text training data, and replace them randomly. During the random replacement process, the second proportion (such as 80%) of the selected characters are replaced with a specific mask, and the third proportion (such as 10%) of the selected characters are replaced with any word, while the fourth proportion (such as 10%) of the selected characters remain unchanged. Among them, the sum of the second proportion, the third proportion and the fourth proportion should be 1. Next, the BERT network decoder predicts the replaced characters based on the data features extracted by the general feature extraction network, uses a fully connected layer to output the predicted probability scores of different characters, and then activates through the softmax layer, and calculates the cross entropy loss function between the predicted probability scores and the original characters.

[0074] Specifically, the BERT network decoder can perform the sentence reconstruction task, using the predicted probability scores of the above different characters and then using the character with the largest predicted probability score to replace the characters replaced by a specific mask in the text training data to reconstruct a new sentence. At this time, the BERT network decoder calculates the L2 loss function between the new sentence and the original sentence in the text training data.

[0075] Specifically, the cross entropy loss function between the predicted probability score and the original character, and / or the L2 loss function between the new sentence and the original sentence in the text training data can be determined as the loss function corresponding to the BERT network decoder, a single modal reconstruction network. This loss function needs to be optimized during training. In this way, the general feature extraction network can learn to extract the contextual features, grammatical structure features, and syntactic features of the characters.

[0076] Optionally, the multimodal feature mapping network is used to map data features of multiple modalities to the same feature space to obtain multiple mapping reconstruction data. Optionally, the multimodal feature mapping network may include a decoder block, a LayerNorm layer, and a fully connected layer in a Transformer network structure. Optionally, the multimodal feature mapping network can calculate the loss function between multiple mapping reconstruction data. For example, in the case of processing feature mapping of data features of image modality and text modality, the data features of image modality and text modality output by the general feature extraction network can be respectively input into the multimodal feature mapping network for feature space transformation to obtain two mapping reconstruction data, and then the cosine similarity is used to calculate the cosine vector distance of the two mapping reconstruction data after spatial transformation as the image-text semantic loss.

[0077] Optionally, when the reconstruction network includes a single modality reconstruction network and a multimodal feature mapping network, before training, the hyperparameters of the multimodal feature mapping network should be different from the hyperparameters of the single modality reconstruction network to highlight the specificity of the two reconstruction networks during training.

[0078] 104. Determine the training loss function according to the loss functions corresponding to all the reconstructed networks, repeat the above steps, optimize the model parameters of the multimodal universal training model until the training loss function converges, and obtain the trained universal feature extraction network.

[0079] Optionally, the method for optimizing model parameters may adopt a gradient descent algorithm or other parameter optimization algorithm, which is not limited in the present invention. Optionally, the above steps repeated in step 104 refer to repeatedly executing steps 101 to 103 to implement repeated optimization training of multiple training data pairs input into the multimodal general training model.

[0080] It can be seen that the method described in the embodiment of the present invention can train the model through data of multiple modalities, and determine the training loss function of the model through different loss functions corresponding to the reconstruction networks of multiple different modalities, thereby training a universal feature extraction network that can be used to extract features of data of multiple modalities, so as to improve the versatility and efficiency of the neural network algorithm.

[0081] As an optional implementation, in the above step 104, determining the training loss function according to the loss functions corresponding to all reconstructed networks may include:

[0082] A first loss function between reconstructed data output by one or more single modality reconstruction networks and corresponding training data is determined as a training loss function.

[0083] Optionally, the number of single modality reconstruction networks can be the same as the number of modalities of the training data of the training data pair. Optionally, when the first loss function output by multiple single modality reconstruction networks is determined as the training loss function, the weighted sum function of the multiple first loss functions can be determined as the training loss function, wherein the weight of each first loss function can be positively correlated with the importance of the single modality reconstruction network, wherein the importance of the single modality reconstruction network is related to the proportion of data of the modality corresponding to the single modality reconstruction network in the data processing scenario of the subsequent application of the general feature training network, for example, in the data processing scenario of the subsequent application of the general feature training network, it is necessary to focus on processing more text modality data, then the weight of the first loss function of the reconstruction network used to reconstruct text modality data is higher.

[0084] Optionally, the first loss function may include at least one of a cross entropy loss function, an L1 loss function, and an L2 loss function. Specifically, the first loss functions corresponding to different single modality reconstruction networks can refer to the above technical details for the single modality reconstruction network, which will not be repeated here.

[0085] As an optional implementation, in the above step 104, determining the training loss function according to the loss functions corresponding to all reconstructed networks may include:

[0086] A second loss function between a plurality of mapped reconstruction data output by the one or more multimodal feature mapping networks is determined as a training loss function.

[0087] Optionally, the number of mapped reconstructed data may be the same as the number of modalities of the training data of the training data pair. Optionally, the second loss function is a similarity function, such as a similarity function based on cosine similarity, or a similarity function based on distance, or a similarity function based on correlation, and reference may also be made to the technical details of the multimodal feature mapping network described above.

[0088] As an optional implementation, in the above step 104, determining the training loss function according to the loss functions corresponding to all reconstructed networks may include:

[0089] A first loss function of one or more single-modality reconstruction networks and a second loss function of one or more multimodal feature mapping networks are determined as training loss functions.

[0090] Optionally, a weighted sum function of all first loss functions and second loss functions can be determined as a training loss function, wherein the weight of each first loss function or second loss function can be positively correlated with the importance of the single modality reconstruction network or the multimodal feature mapping network, wherein the importance is related to the proportion of data of the modality corresponding to the single modality reconstruction network or the multimodal feature mapping network in the data processing scenario of subsequent application of the general feature training network. For example, if the data processing scenario of subsequent application of the general feature training network needs to focus on processing more text modality data, then the first loss function of the reconstruction network used to reconstruct text modality data has a higher weight. For example, if the data processing scenario of subsequent application of the general feature training network needs to focus on processing more text modality and image modality data, then the second loss function of the multimodal feature mapping network used to map text modality data and image modality data has a higher weight. Optionally, the sum of the weights of the second loss function of the multimodal feature mapping network should be higher than the sum of the weights of the first loss function of the single modality reconstruction network. This is because the second loss function of the multimodal feature mapping network is used to characterize the multimodal data processing capability of the general feature extraction network, which should be given special attention during training.

[0091] As an optional implementation, the multimodal general training model may also include at least two preprocessing modules for vectorizing data of different modalities, which can be used to perform embedding and vectorization processing on the input training data to obtain data vectors.

[0092] Accordingly, in the above step 102, inputting the training data in the training data pair into the general feature extraction network for feature extraction to obtain data features may include:

[0093] The training data in the training data pair are respectively input into the preprocessing module of the corresponding mode for vectorization to obtain a data vector;

[0094] The data vector is input into the general feature extraction network for feature extraction to obtain data features.

[0095] Optionally, the training data processed in step 102, such as character processing or text sentence processing in the training data, may be processing of data vectors corresponding to the training data.

[0096] Optionally, the preprocessing module may include an embedding processing unit, a vectorization unit, and a marking unit, wherein the embedding processing unit is used to embed the data, the vectorization unit is used to vectorize the data, and the marking unit is used to mark the position or importance of the data. Optionally, the dimensions of the data vectors finally processed by all preprocessing modules should be the same.

[0097] For example, when it is necessary to perform embedding vectorization processing on the training data of the image modality, a convolution layer can be used to implement the embedding processing unit, wherein the size of the convolution kernel and the step size in the convolution layer are equal to the size of the image block to be processed. For example, for an image data of an image modality, the image size is 224×224. If you want to divide the image data into 196 non-overlapping blocks, each block is 16×16 in size, then the size of the convolution kernel in the convolution layer is 16×16, the step size is 16, and the number of channels can be selected as 768. Then the training data of the image modality is input into the above convolution layer to obtain the embedded features, and then the embedded features are vectorized by the vectorization unit. Specifically, the vectorization is to stretch the tensor into a vector in the length and width dimensions. Furthermore, the order of the channel dimension and the vector dimension can be exchanged, and an initialization mark can be added to them for subsequent migration training on the classification task. Furthermore, a position encoding vector with the same dimension as the vectorized embedded feature can be initialized through the marking unit, and finally the values ​​of the two are added. The position encoding vector is used to distinguish the spatial positions of the image blocks corresponding to different vectors in the original image.

[0098] Furthermore, in order to reduce the amount of calculation, a certain proportion of data vectors can be randomly discarded based on uniform distribution when inputting data vectors into the general feature extraction network. If data vectors are discarded in this step, it is necessary to re-input these discarded data vectors when inputting data features into the reconstruction network for image reconstruction.

[0099] For another example, when it is necessary to perform embedding vectorization processing on the training data of the text modality, an embedding processing unit for the text modality can be used to first perform a character segmentation (tokenization) operation on the training data of the text modality, and add a first identifier and a second identifier to the beginning and end of each text sentence in the training data through a marking unit, wherein, like the initialization mark in the above-mentioned image modality case, the first identifier has no obvious semantic information compared with other words already in the text, and it can more fairly integrate the semantic information of each word in the text, and the vector corresponding to the identifier in the subsequent steps can be used as the semantic representation of the whole sentence, so as to be used for downstream classification tasks. Then, each word or phrase in the input text (corresponding to the case where the text is Chinese or English) is converted into a 768-dimensional vector (used to correspond to the dimension of the processing result of the preprocessing module of the above-mentioned image modality) through a vectorization unit. Further, a position encoding vector with the same dimension as the converted vector can be initialized by the marking unit at the same time, and finally the converted vector is summed with the position encoding vector to obtain the final data vector.

[0100] Embodiment 2

[0101] See also Figure 2 , Figure 2 is a flow chart of another general feature extraction network training method disclosed in an embodiment of the present invention. Figure 2 The method described is applied to a neural network model training device, which can be a corresponding training terminal, training equipment or training server, and the server can be a local server or a cloud server, which is not limited in the embodiment of the present invention. Specifically, the method is also applicable to training a multimodal general training model as described in the first embodiment, such as Figure 2 As shown, the general feature extraction network training method may include the following operations:

[0102] 201. Determine training data pairs.

[0103] 202. Input the training data belonging to the priority modality in the training data pair into the general feature extraction network and the reconstruction network for reconstructing the priority modality data for pre-training until convergence.

[0104] 203. Input the training data in the training data pair into a general feature extraction network for feature extraction to obtain data features.

[0105] 204. Input the data features into a reconstruction network of the corresponding modality to reconstruct the data and obtain the reconstructed data, and calculate the loss function between the reconstructed data output by the reconstruction network and the corresponding training data.

[0106] 205. Determine the training loss function according to the loss functions corresponding to all the reconstructed networks, repeat the above steps, optimize the model parameters of the multimodal universal training model until the training loss function converges, and obtain a trained universal feature extraction network.

[0107] The specific technical details and technical terminology of the above steps 201, 203-205 can be referred to the description of steps 101-104 in the first implementation, and will not be repeated here.

[0108] Optionally, in the pre-training in step 202, the loss function used should only include the loss function corresponding to the reconstruction network used to reconstruct the priority modality data, wherein the details of the loss function can refer to the technical details of the loss function corresponding to the reconstruction network corresponding to different modalities in Example 1, which will not be repeated here.

[0109] Optionally, the training data belonging to the priority modality in step 202 may be data of the image modality, especially when the general feature extraction network adopts the Transformer Encoder structure as described in Example 1, then correspondingly, the reconstruction network trained in step 202 is a reconstruction network for reconstructing the image modality, and the loss function adopted in the pre-training may be the L2 loss function corresponding to the image modality. Through this setting, the training data of the image modality and the reconstruction network for reconstructing the image modality can be first used to train the general feature extraction network until convergence. This is because the training difficulty of training data of different modalities is different, and in order to avoid the instability caused by multimodal data in the early stage of training, the image data set is first used to train the general feature extraction network. At this time, the output of the general feature extraction network is only connected to the image reconstruction network, and the optimized loss also only uses the image reconstruction loss function. When the pre-training of the general feature extraction network and the image reconstruction network are basically converged, subsequent formal training is performed based on the image-text data set or the image-other modality training data pair. At this time, the image reconstruction module, the text prediction module and the feature mapping module will be used at the same time, and the optimized loss is the above-mentioned joint loss.

[0110] It can be seen that the method described in the embodiment of the present invention can pre-train the general feature extraction network and the reconstruction network for reconstructing the priority modality data through the training data belonging to the priority modality in the training data pair, so as to avoid the instability caused by direct training of multimodal data, thereby improving the stability of the general feature network training.

[0111] Embodiment 3

[0112] See also Figure 3 , Figure 3 Schematic diagram of a general feature extraction network training device disclosed in an embodiment of the present invention. Figure 3 The described device can be applied to a corresponding neural network model training device, which can be a corresponding training terminal, training equipment or training server, and the server can be a local server or a cloud server, which is not limited in the embodiment of the present invention. The device is suitable for training a multimodal general training model, which can include a general feature extraction network and at least two reconstruction networks for reconstructing different modal data. The method can be applied to train the multimodal general training model until convergence to obtain a trained general feature extraction network, which can then be applied to realize feature extraction of data of various modalities and further feature processing.

[0113] like Figure 3 As shown, the device may include:

[0114] The first determination module 301 is used to determine a training data pair; the training data pair includes at least two corresponding training data of different modalities.

[0115] Optionally, each training data pair may include at least two training data of different modalities corresponding to each other. Optionally, the modality described in the present invention may include at least one of a data type modality, a data structure modality, and a data source modality. For example, it can be a single type of modality, or a combination of multiple types of modalities. Optionally, when the modality refers to a data type modality, it can be an image modality, a numerical modality, a text modality, a symbolic modality, a video modality, an audio modality, or a time series modality, that is, data of different modalities can be data of the above data types. Optionally, when the modality refers to a data structure modality, it can be a data set modality, a data tree modality, or a graph data modality, that is, data of different modalities can be data of the above data structures. Optionally, when the modality refers to a data source modality, it can be data from different data acquisition devices, such as data from different sensors, or data from data acquisition devices at different locations, and the present invention is not limited here.

[0116] Optionally, the mutual correspondence described in the present invention refers to mutual correspondence in meaning, that is, the objects referred to or the meanings contained in the training data of all modalities included in the same training data pair are consistent. For example, in the same training data pair, the training data of an image modality is an image including a red apple, and the training data of another text modality can be "RedApple" or a text of "red apple" in different languages, or there is also an audio modality The training data is the artificial voice data of "this is a red apple". Optionally, the correspondence between different training data in the training data pair can be collected from some existing data with a corresponding relationship. For example, in some online articles, a description of the picture in the article is attached, and the picture and the description can be collected as a training data pair, or mined from some public large-scale image classification data sets, or can be the result of manual annotation, and the present invention is not limited.

[0117] The first input module 302 is used to input the training data in the training data pair into the general feature extraction network for feature extraction to obtain data features.

[0118] Optionally, the general feature extraction network may include multiple encoders, for example, it may be a stacked structure of multiple encoders, wherein the encoder may be an encoder block in a Transformer network structure. Specifically, the structure of the general feature extraction network here may refer to the Transformer Encoder structure in the VIT (Vision Transformer) structure.

[0119] The second input module 303 is used to input the data features into the reconstruction network of the corresponding modality to reconstruct the data to obtain the reconstructed data, and calculate the loss function between the reconstructed data output by the reconstruction network and the corresponding training data.

[0120] Optionally, the role of the reconstruction network is to reconstruct data features to form reconstructed data, and its overall role is similar to the decoder structure in the existing neural network model, but corresponding to data of different modalities, its specific structure can be modified accordingly. Optionally, the reconstruction network can include a single modality reconstruction network and / or a multimodal feature mapping network.

[0121] Among them, the single modality reconstruction network is used to reconstruct the data features of a single modality, and optionally, the multimodal feature mapping network is used to map the data features of multiple modalities to the same feature space to obtain multiple mapping reconstruction data. Optionally, the multimodal feature mapping network may include a decoder block, a LayerNorm layer, and a fully connected layer in the Transformer network structure. Optionally, the multimodal feature mapping network may calculate a loss function between multiple mapping reconstruction data.

[0122] Optionally, when the reconstruction network includes a single modality reconstruction network and a multimodal feature mapping network, before training, the hyperparameters of the multimodal feature mapping network should be different from the hyperparameters of the single modality reconstruction network to highlight the specificity of the two reconstruction networks during training.

[0123] The second determination module 304 is used to determine the training loss function according to the loss functions corresponding to all the reconstructed networks.

[0124] The training module 305 is used to trigger the first determination module 301, the first input module 302 and the second input module 303 to repeatedly perform the above operations, optimize the model parameters of the multimodal general training model until the training loss function converges, and obtain a trained general feature extraction network.

[0125] It can be seen that the device described in the embodiment of the present invention is able to train the model through data of multiple modalities, and determine the training loss function of the model through different loss functions corresponding to reconstruction networks of multiple different modalities, thereby training a universal feature extraction network that can be used to extract features of data of multiple modalities, so as to improve the versatility and efficiency of the neural network algorithm.

[0126] As an optional implementation, the second determination module 304 determines a specific method of the training loss function according to the loss functions corresponding to all reconstructed networks, including:

[0127] A first loss function between reconstructed data output by one or more single modality reconstruction networks and corresponding training data is determined as a training loss function.

[0128] Optionally, the number of single modality reconstruction networks can be the same as the number of modalities of the training data of the training data pair. Optionally, when the first loss function output by multiple single modality reconstruction networks is determined as the training loss function, the weighted sum function of the multiple first loss functions can be determined as the training loss function, wherein the weight of each first loss function can be positively correlated with the importance of the single modality reconstruction network, wherein the importance of the single modality reconstruction network is related to the proportion of data of the modality corresponding to the single modality reconstruction network in the data processing scenario of the subsequent application of the general feature training network, for example, in the data processing scenario of the subsequent application of the general feature training network, it is necessary to focus on processing more text modality data, then the weight of the first loss function of the reconstruction network used to reconstruct text modality data is higher.

[0129] Optionally, the first loss function may include at least one of a cross entropy loss function, an L1 loss function, and an L2 loss function. Specifically, the first loss functions corresponding to different single modality reconstruction networks can refer to the above technical details for the single modality reconstruction network, which will not be repeated here.

[0130] As an optional implementation, the second determination module 304 determines a specific method of the training loss function according to the loss functions corresponding to all reconstructed networks, including:

[0131] A second loss function between a plurality of mapped reconstruction data output by the one or more multimodal feature mapping networks is determined as a training loss function.

[0132] Optionally, the number of mapped reconstructed data may be the same as the number of modalities of the training data of the training data pair. Optionally, the second loss function is a similarity function, such as a similarity function based on cosine similarity, or a similarity function based on distance, or a similarity function based on correlation, and reference may also be made to the technical details of the multimodal feature mapping network described above.

[0133] As an optional implementation, the second determination module 304 determines a specific method of the training loss function according to the loss functions corresponding to all reconstructed networks, including:

[0134] A first loss function of one or more single-modality reconstruction networks and a second loss function of one or more multimodal feature mapping networks are determined as training loss functions.

[0135] Optionally, a weighted sum function of all first loss functions and second loss functions can be determined as the training loss function, wherein the weight of each first loss function or second loss function can be positively correlated with the importance of the single modal reconstruction network or the multimodal feature mapping network, wherein the importance is related to the proportion of data of the modality corresponding to the single modal reconstruction network or the multimodal feature mapping network in the data processing scenario of subsequent application of the general feature training network. Optionally, the sum of the weights of the second loss function of the multimodal feature mapping network should be higher than the sum of the weights of the first loss function of the single modal reconstruction network. This is because the second loss function of the multimodal feature mapping network is used to characterize the multimodal data processing capability of the general feature extraction network, and it should be given special attention in training.

[0136] As an optional implementation, the multimodal universal training model also includes at least two preprocessing modules for vectorizing data of different modalities, which can be used to embed and vectorize the input training data to obtain data vectors.

[0137] Accordingly, the first input module 302 inputs the training data in the training data pair into the general feature extraction network for feature extraction to obtain data features in a specific manner, including:

[0138] The training data in the training data pair are respectively input into the preprocessing module of the corresponding mode for vectorization to obtain a data vector;

[0139] The data vector is input into the general feature extraction network for feature extraction to obtain data features.

[0140] Optionally, the above description of the first input module 302 processing training data, such as character processing in the training data, or text sentence processing, can be regarded as processing of the data vector corresponding to the training data. Optionally, the preprocessing module may include an embedding processing unit, a vectorization unit and a marking unit, wherein the embedding processing unit is used to embed the data, the vectorization unit is used to vectorize the data, and the marking unit is used to position or importance mark the data. Optionally, the dimensions of the data vectors finally processed by all preprocessing modules should be the same.

[0141] As an optional implementation, Figure 4 As shown, the device also includes:

[0142] The pre-training module 306 is used to input the training data belonging to the priority modality in the training data pair into the general feature extraction network and the reconstruction network for reconstructing the priority modality data for pre-training until convergence before the first input module 302 inputs the training data in the training data pair into the general feature extraction network for feature extraction to obtain data features.

[0143] Optionally, the loss function used in the pre-training of the pre-training module 306 should only include the loss function corresponding to the reconstruction network used to reconstruct the preferred modality data, wherein the details of the loss function can refer to the technical details of the loss function corresponding to the reconstruction network corresponding to different modalities in Example 1, which will not be repeated here.

[0144] Optionally, the training data belonging to the preferred mode may be image mode data, especially when the general feature extraction network adopts the Transformer as described in the first embodiment. Encoder structure, then correspondingly, the reconstruction network trained by the pre-training module 306 is a reconstruction network for reconstructing the image modality, and the loss function adopted in the pre-training of the pre-training module 306 can be the L2 loss function corresponding to the image modality. Through this setting, the training data of the image modality and the reconstruction network for reconstructing the image modality can be used to train the general feature extraction network until convergence. This is because the training difficulty of training data of different modalities is different, and in order to avoid the instability caused by multimodal data in the early stage of training, the image data set is first used to train the general feature extraction network. At this time, the output of the general feature extraction network is only connected to the image reconstruction network, and the optimized loss also only uses the image reconstruction loss function. When the pre-training of the general feature extraction network and the image reconstruction network are basically converged, subsequent formal training is carried out based on the image-text data set or the image-other modality training data pair. At this time, the image reconstruction module, the text prediction module and the feature mapping module will be used at the same time, and the optimized loss is the above-mentioned joint loss.

[0145] Embodiment 4

[0146] See also Figure 5 , Figure 5 FIG. 1 is a schematic diagram of the structure of another general feature extraction network training device disclosed in an embodiment of the present invention. Figure 5 As shown, the device may include:

[0147] A memory 401 storing executable program codes;

[0148] a processor 402 coupled to the memory 401;

[0149] The processor 402 calls the executable program code stored in the memory 401 to execute part or all of the steps in the general feature extraction network training method disclosed in the first or second embodiment of the present invention.

[0150] Embodiment 5

[0151] An embodiment of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the general feature extraction network training method disclosed in Embodiment 1 or Embodiment 2 of the present invention.

[0152] The above describes specific embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0153] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0154] The apparatus, device, non-volatile computer-readable storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus, device, and non-volatile computer storage medium will not be repeated here.

[0155] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0156] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0157] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0158] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0159] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may be in the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of this specification may be in the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0160] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0164] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0165] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0166] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0167] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0168] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0169] Finally, it should be noted that the general feature extraction network training method, device and general feature extraction network disclosed in the embodiments of the present invention are only preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A general feature extraction network training method, characterized in that: The method is applicable to training a multimodal general training model, wherein the multimodal general training model comprises a general feature extraction network and at least two reconstruction networks for reconstructing data of different modalities; the reconstruction network comprises a single modality reconstruction network and / or a multimodal feature mapping network; the method comprises: Determine a training data pair; the training data pair includes at least two corresponding training data of different modalities; the training data pair includes at least two of image modality training data, text modality training data and audio modality training data; Inputting the training data in the training data pair into the universal feature extraction network for feature extraction to obtain data features; the data features include at least two of the data features corresponding to the image modality training data, the data features corresponding to the text modality training data, and the data features corresponding to the audio modality training data; Input the data features into the reconstruction network of the corresponding modality for reconstruction to obtain reconstructed data, and calculate the loss function between the reconstructed data output by the reconstruction network and the corresponding training data; Determine a training loss function according to the loss functions corresponding to all the reconstruction networks, repeat the above steps, optimize the model parameters of the multimodal universal training model until the training loss function converges, and obtain the trained universal feature extraction network; the trained universal feature extraction network can be generally used to extract data features of image modality data and / or text modality data and / or audio modality data; And, determining the training loss function according to the loss functions corresponding to all the reconstructed networks includes: Determining a first loss function between the reconstructed data output by one or more of the single modality reconstruction networks and the corresponding training data as a training loss function; and / or, Determining a second loss function between a plurality of mapping reconstruction data output by one or more of the multimodal feature mapping networks as a training loss function; and / or, The first loss function of one or more of the single modality reconstruction networks and the second loss function of one or more of the multimodal feature mapping networks are determined as training loss functions.

2. The general feature extraction network training method according to claim 1, characterized in that: The universal feature extraction network includes multiple encoders; and / or the multimodal feature mapping network is used to map the data features of multiple modalities to the same feature space to obtain multiple mapping reconstruction data.

3. The general feature extraction network training method according to claim 1, characterized in that: The modality includes at least one of a data type modality, a data structure modality, and a data source modality; and / or the data type modality is an image modality, a numerical modality, a text modality, a symbolic modality, a video modality, an audio modality, or a time series modality; and / or the data structure modality is a data set modality, a data tree modality, or a graph data modality.

4. The general feature extraction network training method according to claim 1, characterized in that: The first loss function includes at least one of a cross entropy loss function, an L1 loss function, and an L2 loss function; and / or the second loss function is a similarity function.

5. The general feature extraction network training method according to claim 1, characterized in that: The multimodal universal training model also includes at least two preprocessing modules for vectorizing data of different modalities; The step of inputting the training data in the training data pair into the general feature extraction network for feature extraction to obtain data features includes: Inputting the training data in the training data pair into the preprocessing module of the corresponding mode respectively for vectorization to obtain a data vector; The data vector is input into the general feature extraction network for feature extraction to obtain data features.

6. The general feature extraction network training method according to claim 1, characterized in that: Before inputting the training data in the training data pair into the universal feature extraction network for feature extraction to obtain data features, the method further includes: The training data belonging to the priority modality in the training data pair is input into the general feature extraction network and the reconstruction network for reconstructing the priority modality data for pre-training until convergence.

7. A general feature extraction network training device, characterized in that: The device is suitable for training a multimodal universal training model, wherein the multimodal universal training model comprises a universal feature extraction network and at least two reconstruction networks for reconstructing data of different modalities; the reconstruction network comprises a single modality reconstruction network and / or a multimodal feature mapping network; The device comprises: A first determination module is used to determine a training data pair; the training data pair includes at least two mutually corresponding training data of different modalities; the training data pair includes at least two of image modality training data, text modality training data and audio modality training data; A first input module, used for inputting the training data in the training data pair into the universal feature extraction network for feature extraction to obtain data features; the data features include at least two of the data features corresponding to the image modality training data, the data features corresponding to the text modality training data, and the data features corresponding to the audio modality training data; A second input module is used to input the data features into the reconstruction network of the corresponding modality to reconstruct and obtain reconstructed data, and calculate the loss function between the reconstructed data output by the reconstruction network and the corresponding training data; A second determination module is used to determine a training loss function according to the loss functions corresponding to all the reconstructed networks; A training module, used to trigger the first determination module, the first input module and the second input module to repeatedly perform the above operations, optimize the model parameters of the multimodal universal training model until the training loss function converges, and obtain the trained universal feature extraction network; the trained universal feature extraction network can be generally used to extract data features of image modality data and / or text modality data and / or audio modality data; And, the second determination module determines a specific method of the training loss function according to the loss functions corresponding to all the reconstructed networks, including: Determining a first loss function between the reconstructed data output by one or more of the single modality reconstruction networks and the corresponding training data as a training loss function; and / or, Determining a second loss function between a plurality of mapping reconstruction data output by one or more of the multimodal feature mapping networks as a training loss function; and / or, The first loss function of one or more of the single modality reconstruction networks and the second loss function of one or more of the multimodal feature mapping networks are determined as training loss functions.

8. A general feature extraction network training device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the general feature extraction network training method as described in any one of claims 1-6.

9. A general feature extraction network, characterized in that: It is obtained by training by executing the general feature extraction network training method as described in any one of claims 1-6.

Images