Multimodal data processing method, electronic device, and storage medium

Abstract

The embodiment of the application discloses a kind of multimodal data processing methods, comprising: determining first feature vector and second feature vector according to current input data in neural network model;Data includes image modal data and text modal data;First loss value and second loss value are respectively determined based on first feature vector, second feature vector;First loss value represents the semantic category prediction error of current input data;Second loss value represents the degree of association between semantic relevance and distance between current input data and historical input data;Semantic relevance indicates whether data belongs to the same semantic category;Third loss value is determined based on the feature vector corresponding to all data of current input and historical input in neural network model;Third loss value represents the difference between image modal and text modal;Neural network model is trained based on first loss value, second loss value and third loss value.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, and in particular to a multi-modal data processing method, an electronic device and a storage medium. BACKGROUND

[0002] In the prior art, for retrieval of different modal data, a target feature extraction model is often trained by one or two constraints according to sample data features corresponding to training sample data of different modalities, such as intra-class low-rank loss constraint and semantic consistency constraint. After obtaining target data and data to be retrieved of different modalities, a cross-modal retrieval operation is performed by calling the target feature extraction model to obtain a corresponding retrieval result. However, the prior art has the problem of insufficient constraint in generating a common feature space of different modalities, which leads to lack of robustness of the model and thus low retrieval accuracy. SUMMARY

[0003] Therefore, the embodiments of the present application provide a multi-modal data processing method, an electronic device and a storage medium.

[0004] The technical solution of the present application is implemented as follows:

[0005] In a first aspect, the embodiments of the present application provide a multi-modal data processing method, comprising:

[0006] determining a first feature vector and a second feature vector in a neural network model according to currently input data; the data includes image modal data and text modal data;

[0007] determining a first loss value and a second loss value based on the first feature vector and the second feature vector respectively; the first loss value represents a semantic category prediction error of the currently input data; the second loss value represents an association degree of semantic relevance and distance between the currently input data and historical input data; the semantic relevance represents whether the data belongs to the same semantic category;

[0008] determining a third loss value based on feature vectors corresponding to all data of the currently input and the historical input in the neural network model; the third loss value represents a difference degree between the image modal and the text modal;

[0009] training the neural network model based on the first loss value, the second loss value and the third loss value.

[0010] Further, the determining of the first loss value and the second loss value based on the first feature vector and the second feature vector respectively comprises:

[0011] determine a first loss value based on an iteration result of the first feature vector and the second feature vector in iterative calculation of the neural network model; the iteration result includes probabilities of the first feature vector and the second feature vector corresponding to a plurality of semantic categories respectively;

[0012] determine a second loss value based on distances between the first feature vector, the second feature vector and other feature vectors corresponding to historical input data.

[0013] Further, the determining, in the neural network model, the first feature vector and the second feature vector according to the current input data includes:

[0014] processing, based on an image feature extraction network in the neural network model, the current input image modal data to obtain the first feature vector;

[0015] processing, based on a text feature extraction network in the neural network model, the current input text modal data to obtain the second feature vector;

[0016] The method further includes:

[0017] determining a fourth loss value representing a fitting degree of the neural network model based on a number of layers and neuron weights of the image feature extraction network and a number of layers and neuron weights of the text feature extraction network;

[0018] The training, based on the first loss value, the second loss value and the third loss value, the neural network model includes:

[0019] updating parameters of the neural network model based on the first loss value, the second loss value, the third loss value and the fourth loss value.

[0020] Further, the processing, based on the image feature extraction network in the neural network model, the current input image modal data to obtain the first feature vector includes:

[0021] segmenting the current input image modal data into a preset number of image modal sub-data;

[0022] processing, based on the image feature extraction network in the neural network model, the preset number of image modal sub-data to obtain the first feature vector.

[0023] Further, the determining, based on the iteration result of the first feature vector and the second feature vector in the iterative calculation of the neural network model, the first loss value includes:

[0024] obtaining first network parameters corresponding to a network used for calculating the first loss value;

[0025] determine a first loss value based on the first network parameter and iterative results of the first feature vector and the second feature vector in iterative calculation of the neural network model;

[0026] The second loss value is determined based on distances between the first feature vector, the second feature vector and other feature vectors corresponding to historical input data, and the second loss value comprises:

[0027] Obtain image modal parameters and text modal parameters in the neural network model;

[0028] In the other feature vectors corresponding to the historical input data, reference feature vectors of the first feature vector and the second feature vector are determined respectively; the reference feature vectors include semantic related feature vectors and semantic independent feature vectors;

[0029] Based on the image modal parameters and the text modal parameters, distances between the first feature vector, the second feature vector and the reference feature vectors are calculated respectively;

[0030] The second loss value is determined based on the distances.

[0031] Further, the third loss value is determined based on feature vectors corresponding to the current input and the historical input data in the neural network model, and the third loss value comprises:

[0032] Obtain second network parameters corresponding to a network for calculating the third loss value;

[0033] Determine the third loss value based on the second network parameters and the feature vectors corresponding to the current input and the historical input data in the neural network model;

[0034] The parameters of the neural network model are updated based on the first loss value, the second loss value, the third loss value and the fourth loss value, and the updating comprises:

[0035] The first network parameter, the image modal parameter, the text modal parameter and the second network parameter are updated based on the first loss value, the second loss value, the third loss value and the fourth loss value.

[0036] Further, the first network parameter, the image modal parameter, the text modal parameter and the second network parameter are updated based on the first loss value, the second loss value, the third loss value and the fourth loss value, and the updating comprises:

[0037] Calculate a loss parameter based on the first loss value, the second loss value and the fourth loss value;

[0038] update the first network parameter, the image modality parameter, the text modality parameter and the second network parameter respectively based on a first network parameter, an image modality parameter, a text modality parameter and a second network parameter corresponding to a minimum difference between the loss parameter and the third loss value and a maximum difference between the loss parameter and the third loss value.

[0039] Further, the method further comprises:

[0040] inputting the image modality data to be retrieved into the trained neural network model;

[0041] determining the text modality data corresponding to the image modality data to be retrieved through the trained neural network model.

[0042] In a second aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory for storing a computer program capable of running on the processor;

[0043] When the processor runs the computer program, the steps of the method according to the foregoing one or more technical solutions are executed.

[0044] In a third aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions; after the computer executable instructions are executed by a processor, the method according to the foregoing one or more technical solutions can be implemented.

[0045] The multi-modal data processing method provided by the present application comprises: determining a first loss value and a second loss value based on a first feature vector and a second feature vector respectively; the first loss value represents a semantic category prediction error of the current input data; the second loss value represents an association degree of semantic relevance and distance between the current input data and the historical input data; the semantic relevance represents whether the data belongs to the same semantic category; determining a third loss value based on feature vectors corresponding to all data of the current input and the historical input in the neural network model; the third loss value represents a difference degree between the image modality and the text modality; and training the neural network model based on the first loss value, the second loss value and the third loss value. In this way, when training the neural network model, three loss values calculated based on three loss functions can comprehensively realize three constraints of semantic category distinction, feature vector distance arrangement based on semantic relevance and difference between different modalities, so as to make the constraints in the neural network model more robust and rich, and provide more accurate results for multi-modal data retrieval. Moreover, based on the third loss value, the difference between different modalities can be reduced, so as to facilitate querying related data of another modality based on data of one modality, and facilitate cross-modal data retrieval and query through the neural network model. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A flowchart of a multi-modal data processing method provided by an embodiment of the present application is shown in FIG. 1.

[0047] Figure 2 A flowchart of a multi-modal data processing method provided by an embodiment of the present application is shown in FIG. 1.

[0048] Figure 3 A structural diagram of a multi-modal data processing device provided by an embodiment of the present application is shown in FIG. 2.

[0049] Figure 4 A multi-modal data retrieval framework provided by an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0050] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0051] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0052] In the following description, the terms "first\second\third" are only to distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0054] As shown in FIG. 1, the present application provides a multi-modal data processing method, which comprises: Figure 1

[0055] S110: determining a first feature vector and a second feature vector in a neural network model according to the current input data; the data comprises image modal data and text modal data;

[0056] ​S120: determining a first loss value and a second loss value based on the first feature vector and the second feature vector respectively; the first loss value represents a semantic category prediction error of the current input data; the second loss value represents a degree of association between semantic relevance and distance between the current input data and the historical input data; the semantic relevance represents whether the data belongs to the same semantic category;

[0057] S130: determining a third loss value based on feature vectors corresponding to all data of the current input and the historical input in the neural network model; the third loss value represents a degree of difference between the image modal and the text modal;

[0058] S140: training the neural network model based on the first loss value, the second loss value and the third loss value.

[0059] Here, the neural network model is used for training and learning based on multi-modal sample data, to retrieve data of other modal corresponding to a certain modal data to be retrieved. The modal can represent the form, mode, etc. of the data, for example, the data can include image modal data and text modal data. The image modal data can be image form data, and the text modal data can be text form data, for example, text form data used to describe the semantic content represented by the corresponding image modal data.

[0060] In an embodiment of the present application, the current input data is sample data used for training the neural network model, which can be a pair of image modal data and text modal data, i.e. the text modal data represents the text content corresponding to the image content of the image modal data, or a group of unrelated image modal data and text modal data.

[0061] In one embodiment, the neural network model can include a feature generation module and a modal classification module, wherein the feature generation module can be used to perform steps S110 to S120, and the modal classification module can be used to perform step S130. Determining the first feature vector and the second feature vector can be performed by one or more neural networks in the neural network model, for example, determining the first feature vector by an image feature extraction network and determining the second feature vector by a text feature extraction network. The neural network in the neural network model can be a long short-term memory (LSTM) network or the like. The first feature vector is used to represent the feature space of the image modal data, and the second feature vector is used to represent the feature space of the text modal data.

[0062] In another embodiment, determining the first loss value can be performed by one of the neural networks in the neural network model, for example, the first loss value can be determined by a 2-layer fully connected network based on a first loss function. Here, the first loss function can be a loss function for label prediction to make a prediction of the semantic class (i.e., label) of the data. In addition, the neural network for determining the first loss value can be connected to the output of the neural network for determining the first feature vector and the second feature vector.

[0063] Exemplarily, the first loss value based on the first loss function can be a cross-entropy loss representing the semantic class prediction of the current input image modality data and text modality data, or a cross-entropy loss representing the semantic class of all data of the current input and historical input in the neural network model. The smaller the cross-entropy loss, i.e., the first loss value, the smaller the semantic class prediction error, i.e., the predicted semantic class is closer to the true semantic class. Here, the semantic class can be a class to which the semantic information of the data belongs, for example, when the semantic information of the text modality data is “two people running”, the corresponding semantic class can be “exercise” or “sports”, etc. In this way, the calculation based on the first loss value can realize the distinction of different semantic information.

[0064] In one embodiment, determining the second loss value can be performed by a second loss function in the neural network model, for example, the second loss function can be a triplet loss function. Based on the triplet constraint, the semantic correlation and distance between the feature vectors in the neural network model can have a correlation, for example, the closer the distance between the sample data with higher semantic correlation in the feature space, the farther the distance between the sample data with lower semantic correlation in the feature space. The semantic correlation can refer to the same or different semantic classes of the data, or the correlation of the semantic classes of the data, for example, the correlation of “basketball” and “soccer” is higher than the correlation of “basketball” and “key”.

[0065] In another embodiment, determining the third loss value can be performed by one of the neural networks in the neural network model, for example, the third loss value can be determined by a 2-layer fully connected network based on a third loss function. Here, the neural network for determining the third loss value is different from the neural network for determining the first loss value. The third loss value based on the third loss function can be used to represent the cross-entropy loss of the modality class of all data of the current input and historical input in the neural network model, where the modality class can include image modality and text modality. The smaller the cross-entropy loss, i.e., the third loss value, the smaller the difference between the image modality and the text modality.

[0066] Thus, based on the loss values obtained by the three loss functions, the multi-modal sample data can be fully optimized. When training the neural network model based on the sample data, based on the three constraints of semantic category prediction error, semantic correlation and vector correlation, and difference between different modalities, the data semantic category can be distinguished, the distance between the feature vectors arranged based on the semantic correlation can be arranged, and the difference between the image modalities and the text modalities can be reduced. Moreover, based on the third loss value and the loss function thereof, the difference between the image modalities and the text modalities is reduced, which can greatly reduce the independence of the image feature space and the text feature space, thereby facilitating cross-modal data retrieval query through the neural network model. Based on this, the constraints in the neural network model are more robust and rich, and more conducive to accurate retrieval of multi-modal data.

[0067] In some embodiments, the S120 can include:

[0068] determining a first loss value based on an iterative result of the first feature vector and the second feature vector in the iterative calculation of the neural network model; the iterative result includes probabilities of the first feature vector and the second feature vector corresponding to a plurality of semantic categories, respectively;

[0069] determining a second loss value based on a distance between the first feature vector, the second feature vector and other feature vectors corresponding to the historical input data;

[0070] The first loss value represents the semantic category difference degree between the current input image modal data and text modal data; the second loss value represents the association degree of semantic correlation and distance between the current input data and the historical input data; and the semantic correlation represents whether the data belongs to the same semantic category.

[0071] In the embodiments of the present application, the first feature vector and the second feature vector are input into a neural network for determining the first loss value, such as a 2-layer fully connected network. The first feature vector and the second feature vector are iteratively calculated in the neural network model, and based on the iterative result obtained by each iteration calculation, the first loss value is determined by the first loss function for label prediction.

[0072] In one embodiment, a certain number of semantic categories are preset in the neural network model, and each iteration calculation will obtain a probability distribution of the first feature vector or the second feature vector belonging to each preset semantic category, for example, one iteration calculation can obtain that the probability of the image modal data corresponding to the first feature vector belonging to semantic category A is 40%, the probability of belonging to semantic category B is 20%, and so on.

[0073] In another embodiment, a first loss value can be calculated based on the iteration results and the ground truth semantic labels corresponding to the first and second feature vectors. For example, the first and second feature vectors exist in pairs, and the text modal data represents the text information represented by the image modal data. The ground truth semantic label of the corresponding text information is (0,0,0,1,0,0), representing the fourth semantic category corresponding to the first and second feature vectors. Based on this, the learning ability of the neural network model is trained and optimized using the first loss function, based on the ground truth semantic labels and the probability distribution of the semantic categories obtained through iterative calculation. Thus, the obtained first loss value can represent the cross-entropy loss of the semantic categories of each image instance and text instance (i.e., the existence form of image modal data and text modal data in the neural network model), thereby effectively training the neural network model.

[0074] In one embodiment, the semantic relevance and distance between other feature vectors and the feature vectors corresponding to the current input data can be determined by obtaining other feature vectors corresponding to the sample data of the historical input neural network model. For example, the first feature vector corresponding to the current input data is v. i The second eigenvector is t i From the historical input data, you can select data related to v. i A semantically related text feature vector t j , and with v i A semantically irrelevant text feature vector t k Based on these three feature vectors, v can be determined respectively. i With t j and t k Distance between: Know in, Representative image v i With text t j Semantic relevance Representative image v i With text t k Semantic irrelevant. Based on and Knowing the second loss function allows us to calculate the second loss value. The second loss function can be a triplet loss function.

[0075] In another embodiment, besides selecting v i A semantically related text feature vector t j , and with v i A semantically irrelevant text feature vector t k In addition, you can also select t iAn image feature vector v related to semantics j , and t i An image feature vector v unrelated to semantics k Similar to the previous embodiment, the distance between the two vectors is calculated as The The hyperparameters λ and μ related to the second loss function in the neural network model are obtained, where the hyperparameters are preset parameter values in the neural network model.

[0076] Based on The Based on the image second loss value and the hyperparameters, a text second loss value of the text modal data can be calculated as Here, θ V represents a preset image modal parameter in the neural network model, represents a value greater than 0.

[0077] Based on and Based on the text second loss value and the hyperparameters, a text second loss value of the text modal data can be calculated as Here, θ T represents a preset text modal parameter in the neural network model.

[0078] The second loss value L imi (θ V , θ T ) can be the sum of the image second loss value and the text second loss value:

[0079]

[0080] In yet another embodiment, when calculating the distance between the two vectors, the following formula can be used:

[0081] l2(v, t) = ||f V (v; θ V ) - f T (t; θ T )||2

[0082] Here, f V (v; θ V ) represents the first feature vector output by the image feature extraction network based on the image modal parameter, f T (t; θ T ) represents the second feature vector output by the text feature extraction network based on the text modal parameter, and ||||2 represents the 2-norm.

[0083] ​Thus, based on the training of the first loss function, the neural network model can effectively distinguish the semantic categories of different sample data, so as to identify the semantic information of the to-be-retrieved data. In addition, based on the training of the second loss function, the neural network model can arrange the distance of the feature space based on the sample semantic correlation, so as to preferentially determine one or more feature vectors with the closest distance when retrieving data, and then output the corresponding modal data.

[0084] In some embodiments, the S110 can include:

[0085] processing the current input image modal data based on an image feature extraction network in the neural network model to obtain a first feature vector;

[0086] processing the current input text modal data based on a text feature extraction network in the neural network model to obtain a second feature vector;

[0087] The method further includes:

[0088] determining a fourth loss value representing the fitting degree of the neural network model based on the number of layers and neuron weights of the image feature extraction network and the number of layers and neuron weights of the text feature extraction network;

[0089] The S140 can include:

[0090] updating the parameters of the neural network model based on the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0091] In the embodiments of the present application, the image feature extraction network can be a neural network for outputting the first feature vector of the image modal data. For example, it can be a 5-layer neural network, in which the first 2 layers are convolutional layers for convolution operation on the image modal data, the third layer is a pooling layer for ROI (Region of Interest) pooling on the feature map obtained after convolution, and the last 2 layers are LSTM layers for processing the pooled feature map to obtain the final first feature vector.

[0092] The text feature extraction network can be a neural network for outputting the second feature vector of the text modal data, for example, it can be a 2-layer LSTM network.

[0093] In one embodiment, the text modal data can be first processed by word segmentation, and the segmented text modal data is converted into a vector by a word to vector (Word2Vec) model. The obtained vector is input into the text feature extraction network to obtain the second feature vector.

[0094] In another embodiment, the fourth loss value can be calculated by a fourth loss function, representing the fitting degree of the neural network model, so as to suppress the overfitting of the network training. Here, the fourth loss function can be a regularization loss function, for example, the fourth loss value wherein, represents the neuron weight of the lth layer in the image feature extraction network, represents the neuron weight of the lth layer in the text feature extraction network. l = 1, 2…, L, and l is a positive integer. L is the maximum value of the number of layers of the image feature extraction network and the number of layers of the text feature extraction network, for example, the image feature extraction network is 5 layers, and the text feature extraction network is 2 layers, then L takes 5. F represents the F-norm.

[0095] In one embodiment, based on the first loss value, the second loss value, the third loss value and the fourth loss value, the parameters of the neural network model can be updated by back propagation to optimize the ability of the neural network model. The parameters of the neural network model can include the preset functional parameters in the neural network model, such as image modal parameters, text modal parameters, etc. In the neural network model, the image modal parameters participate in the processing of the image modal data and the first feature vector, and the text modal parameters participate in the processing of the text modal data and the second feature vector.

[0096] In this way, based on the fourth loss function, the fitting degree of the neural network model can be converged on the basis of sufficient optimization of the learning ability of the neural network model, and the overfitting of the neural network can be suppressed, thereby further improving the reliability of the neural network model.

[0097] In some embodiments, the processing of the current input image modal data based on the image feature extraction network in the neural network model to obtain the first feature vector can include:

[0098] segmenting the current input image modal data into a preset number of image modal sub-data;

[0099] processing the preset number of image modal sub-data based on the image feature extraction network in the neural network model to obtain the first feature vector.

[0100] In the embodiment of the application, before the image modal data is input into the image feature extraction network for processing, the image modal data can be segmented into a preset number of image modal sub-data. For example, the image can be cut into K equal sub-pictures, for example, for image modal data with a size of 360*240, when L = 3, H = 2, and K = 6, the size of each sub-picture is 360 / 3 = 120, 240 / 2 = 120, and a total of 6 sub-pictures of 120*120 are cut.

[0101] In one embodiment, when the image feature extraction network is a 5-layer neural network, after the image modality data is segmented into a preset number of image modality sub-data, the image modality sub-data is convolved by the first 2 convolutional layers of the image feature extraction network. Since the input image modality data can be of different sizes, the sizes of the segmented image modality sub-data are also different, resulting in different sizes of the feature maps after convolution. Therefore, the feature maps obtained after convolution are scaled to a predefined size by the third pooling layer. The feature maps after convolution are input into the last 2 LSTM layers, and the first feature vector can be obtained.

[0102] In some embodiments, determining the first loss value based on the iterative results of the first feature vector and the second feature vector obtained in the iterative calculation of the neural network model comprises:

[0103] obtaining first network parameters corresponding to the network used to calculate the first loss value;

[0104] determining the first loss value based on the first network parameters, and the iterative results of the first feature vector and the second feature vector obtained in the iterative calculation of the neural network model;

[0105] determining the second loss value based on the distances between the first feature vector, the second feature vector, and other feature vectors corresponding to historical input data comprises:

[0106] obtaining image modality parameters and text modality parameters in the neural network model;

[0107] determining reference feature vectors of the first feature vector and the second feature vector among the other feature vectors corresponding to the historical input data; the reference feature vectors include semantically related feature vectors and semantically unrelated feature vectors;

[0108] calculating the distances between the first feature vector, the second feature vector, and the reference feature vectors based on the image modality parameters and the text modality parameters;

[0109] determining the second loss value based on the distances.

[0110] In the embodiments of the present application, the network used to calculate the first loss value can be a 2-layer fully connected network, and the first network parameters can be network parameters corresponding to the network used to calculate the first loss value in the neural network model, for example, the first network parameters can be classifier parameters corresponding to the network.

[0111] In one embodiment, the groundtruth corresponding to the first feature vector and the second feature vector is obtained before the training starts, so as to modify and learn the generated probability distribution by the iterative calculation through the first loss function. Based on the first network parameter θ imd and the iterative result of the first feature vector and the second feature vector, the first loss value is determined:

[0112]

[0113] Here, the first loss value is L imd (θ imd ), y i is the groundtruth corresponding to each set of the first feature vector and the second feature vector, representing the generated probability distribution of each iteration calculation.

[0114] In another embodiment, the reference feature vector of the first feature vector v i may include one text feature vector t i related to the semantic of v j and one text feature vector t i irrelevant to the semantic of v k . The reference feature vector of the second feature vector t i may include one image feature vector v i related to the semantic of t j and one image feature vector v i irrelevant to the semantic of t k . In this way, based on the determined distance know The second loss value can be determined

[0115]

[0116] wherein V(θ V ) represents the image modal data trained by the image modal parameter θ V , and T(θ T ) represents the text modal data trained by the text modal parameter θ T .

[0117]

[0118]

[0119] In this way, the image modality data closest in distance can be the text modality data with high semantic correlation with the image modality data, and the text modality data closest in distance can be the image modality data with high semantic correlation with the text modality data. Based on this, when cross-modal data retrieval is performed through the neural network model, the other modality data with the highest correlation with the to-be-retrieved modality data can be quickly found according to the distance in the feature space.

[0120] In some embodiments, the S130 can include:

[0121] obtaining second network parameters of a network for calculating a third loss value;

[0122] determining a third loss value based on the second network parameters and feature vectors corresponding to all data of current input and historical input in the neural network model; the third loss value representing a difference degree between the image modality and the text modality;

[0123] The updating of the parameters of the neural network model based on the first loss value, the second loss value, the third loss value, and the fourth loss value includes:

[0124] updating the first network parameters, the image modality parameters, the text modality parameters, and the second network parameters based on the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0125] In the embodiments of the present application, the third loss value can be calculated by a third loss function in the modality classification module of the neural network model, and is used to eliminate the difference between the first feature vector and the second feature vector based on the difference degree between the image modality and the text modality, that is, to eliminate the difference between the image feature space and the text feature space. The network for calculating the third loss value can be a network in the neural network model, and is different from the networks for feature extraction, calculation of the first loss value, the second loss value, and the fourth loss value, for example, can be a 2-layer fully connected network. The second network parameters represent the network parameters of the network, for example, can be classifier parameters. The third loss value can have a corresponding relationship with the difference degree between the image modality and the text modality, for example, the smaller the third loss value, the smaller the difference degree between the image modality and the text modality.

[0126] In an embodiment, a groundtruth modality label of each data instance can also be obtained, where the data instance refers to an image modality data or a text modality data in the form of an instance in the neural network model. Similar to the aforementioned groundtruth semantic label, the groundtruth modality label represents the modality of each data instance. For example, if the groundtruth modality label of a data instance is (0, 1), it represents that the data is a text modality data; if it is (1, 0), it represents that the data is an image modality data. A third loss value L adv (θ D ) can be calculated by a third loss function:

[0127]

[0128] Here, θ D represents the second network parameter, m i represents the groundtruth modality label of each data instance, n represents the total number of data instances contained in the neural network model, and D represents the generated modality probability of each data instance after calculation, for example, the probability of a data instance belonging to an image modality is 60%, the probability of a text modality is 40%, etc.

[0129] In this way, the difference between the image modality and the text modality is reduced based on the third loss function, thereby gradually eliminating the difference between the image feature space and the text feature space, which is more conducive to the use of the neural network model for cross-modality data retrieval.

[0130] In another embodiment, after the image modality data and the text modality data as training samples are input into the neural network model, the parameters of the neural network model are updated by back propagation based on the first loss function, the second loss function, the third loss function, and the fourth loss function. By updating the first network parameter, the parameter update of the network of the first loss function for label prediction can be realized; by updating the image modality parameter and the text modality parameter, the parameter update of the second loss function can be realized; by updating the second network parameter, the parameter update of the network of the third loss function for modality classification can be realized. In this way, the optimization training of the neural network model can be realized.

[0131] In some embodiments, the updating of the first network parameter, the image modality parameter, the text modality parameter, and the second network parameter based on the first loss value, the second loss value, the third loss value, and the fourth loss value comprises:

[0132] calculating a loss parameter based on the first loss value, the second loss value, and the fourth loss value;

[0133] update the first network parameter, the image modality parameter, the text modality parameter, and the second network parameter respectively based on a first network parameter, an image modality parameter, a text modality parameter, and a second network parameter corresponding to a minimum difference value between the loss parameter and the third loss value and a second network parameter corresponding to a maximum difference value between the loss parameter and the third loss value.

[0134] In the embodiments of the present application, the calculation of the first loss value, the second loss value, and the fourth loss value all belong to the feature generation module of the neural network model, and the calculation of the third loss value belongs to the modality classification module. Therefore, first, the pre-set hyperparameters α and β in the feature generation module can be obtained, and the loss parameter L is calculated:

[0135] L emb (θ V , θ T , θ imd )=α·L imi +β·L imd +L reg

[0136] determine the value of θ V , θ T , θ imd when the difference value between the loss parameter and the third loss value is minimum, and update the image modality parameter, the text modality parameter, and the first network parameter in the neural network model to these three values respectively:

[0137]

[0138] determine the value of θ D when the difference value between the loss parameter and the third loss value is maximum, and update the second network parameter in the neural network model to this value:

[0139]

[0140] Here, the minimum difference value and the maximum difference value can be the minimum and maximum of the difference values generated after a certain number of iterations. In this way, since the smaller the first loss value in the feature generation module, the smaller the semantic category prediction error, the more accurate the distinction of the semantic information of the data; the smaller the second loss value, the closer the other modality data with high correlation to one modality data; the smaller the fourth loss value, the lower the overfitting probability of the model. Therefore, the smaller the difference value between the loss parameter and the third loss value, the more the corresponding first network parameter, image modality parameter, and text modality parameter deviate from the optimal parameters of the model training and optimization. Since the smaller the third loss value in the modality classification module, the smaller the difference between the image modality and the text modality, therefore, the larger the difference value between the loss parameter and the third loss value, the more the corresponding second network parameter deviates from the optimal parameters of the model training and optimization.

[0141] In some embodiments, as shown in Figure 2 The method further comprises:

[0142] S150: inputting the image modality data to be retrieved into the trained neural network model;

[0143] S160: determining the text modality data corresponding to the image modality data to be retrieved by the trained neural network model.

[0144] In the embodiments of the present application, the neural network model trained based on the sample data can be used for data retrieval, for example, when it is needed to retrieve the text modality data corresponding to the current image modality data, the image modality data to be retrieved can be input into the model. In this way, based on the neural network model that has been prevented from overfitting and the difference between the cross-modal data has been eliminated, the data retrieval of distinguishing the semantic categories and identifying the text modality data closest to the image modality data as the cross-modal data with the highest relevance can more accurately obtain the text modality data with the highest semantic information relevance to the image modality data.

[0145] In one embodiment, S160 can include: determining a preset number of text modality data corresponding to the image modality data to be retrieved by the trained neural network model. For example, a preset number of text modality data closest to the image modality data to be retrieved can be determined. Illustratively, according to the distance from small to large, the first N text modality data are selected, that is, the text modality data corresponding to the image modality data to be retrieved, as the Top-N output result of the neural network model cross-modal data retrieval. Here, N is a positive integer greater than 0.

[0146] In another embodiment, after the feature vector corresponding to the image modality data to be retrieved is determined, the corresponding feature vector is input into the trained neural network model to determine the corresponding text modality data.

[0147] As shown in Figure 3 The embodiments of the present application provide a multi-modal data processing apparatus, which comprises:

[0148] A first determination unit 110 is configured to determine a first feature vector and a second feature vector in a neural network model according to currently input data; the data comprises image modality data and text modality data;

[0149] The second determining unit 120 is configured to determine a first loss value and a second loss value based on the first feature vector and the second feature vector respectively, wherein the first loss value represents a semantic category prediction error of the current input data, and the second loss value represents a correlation degree between semantic relevance and distance between the current input data and the historical input data, and the semantic relevance represents whether the data belongs to the same semantic category.

[0150] The third determining unit 130 is configured to determine a third loss value based on feature vectors corresponding to all data of the current input and the historical input in the neural network model, wherein the third loss value represents a difference degree between the image modal and the text modal.

[0151] The training unit 140 is configured to train the neural network model based on the first loss value, the second loss value and the third loss value.

[0152] In some embodiments, the second determining unit 120 is specifically configured to:

[0153] determine the first loss value based on iteration results of the first feature vector and the second feature vector in iterative calculation of the neural network model, wherein the iteration results include probabilities of the first feature vector and the second feature vector in a plurality of semantic categories respectively;

[0154] determine the second loss value based on distances between the first feature vector, the second feature vector and other feature vectors corresponding to the historical input data.

[0155] In some embodiments, the first determining unit 110 is specifically configured to:

[0156] obtain the first feature vector by processing image modal data of the current input based on an image feature extraction network in the neural network model;

[0157] obtain the second feature vector by processing text modal data of the current input based on a text feature extraction network in the neural network model.

[0158] The apparatus further includes:

[0159] A fourth determining unit is configured to determine a fourth loss value representing a fitting degree of the neural network model based on a number of layers and neuron weights of the image feature extraction network and a number of layers and neuron weights of the text feature extraction network.

[0160] The training unit 140 is specifically configured to:

[0161] update parameters of the neural network model based on the first loss value, the second loss value, the third loss value and the fourth loss value.

[0162] In some embodiments, the first determining unit 110 is specifically configured to:

[0163] segmenting the current input image modality data into a preset number of image modality sub-data;

[0164] processing the preset number of image modality sub-data based on an image feature extraction network in the neural network model to obtain a first feature vector.

[0165] In some embodiments, the second determining unit 120 is specifically configured to:

[0166] obtaining first network parameters corresponding to a network used for calculating a first loss value;

[0167] determining the first loss value based on the first network parameters, and iterative results of the first feature vector and the second feature vector obtained in iterative calculation of the neural network model;

[0168] obtaining image modality parameters and text modality parameters in the neural network model;

[0169] determining reference feature vectors of the first feature vector and the second feature vector respectively among other feature vectors corresponding to historical input data; the reference feature vectors include semantic related feature vectors and semantic irrelevant feature vectors;

[0170] calculating distances between the first feature vector, the second feature vector and the reference feature vectors respectively based on the image modality parameters and the text modality parameters;

[0171] determining a second loss value based on the distances.

[0172] In some embodiments, the third determining unit 130 is specifically configured to:

[0173] obtaining second network parameters corresponding to a network used for calculating a third loss value;

[0174] determining the third loss value based on the second network parameters and feature vectors corresponding to all data of current input and historical input in the neural network model;

[0175] The training unit 140 is specifically configured to:

[0176] updating the first network parameters, the image modality parameters, the text modality parameters and the second network parameters based on the first loss value, the second loss value, the third loss value and the fourth loss value.

[0177] In some embodiments, the training unit 140 is specifically configured to:

[0178] compute a loss parameter based on the first loss value, the second loss value, and the fourth loss value;

[0179] update the first network parameter, the image modality parameter, the text modality parameter, and the second network parameter based on a first network parameter corresponding to a minimum difference between the loss parameter and the third loss value, an image modality parameter corresponding to the minimum difference between the loss parameter and the third loss value, a text modality parameter corresponding to the minimum difference between the loss parameter and the third loss value, and a second network parameter corresponding to a maximum difference between the loss parameter and the third loss value.

[0180] In some embodiments, the apparatus further comprises:

[0181] an input unit configured to input image modality data to be retrieved into the trained neural network model;

[0182] a retrieval unit configured to determine text modality data corresponding to the image modality data to be retrieved by the trained neural network model.

[0183] The following provides a specific example in combination with any of the above embodiments:

[0184] The embodiment of the present application provides a multi-constraint cross-modal retrieval framework. In the framework, label prediction constraint, triple constraint, and regularization constraint constitute a feature generation module; and modality classification constraint constitutes a modality classification module. The two modules restrict each other, finally generate a public subspace of images and texts, and complete retrieval in the public subspace. A framework structure diagram is shown in Figure 4 .

[0185] S1. Model construction:

[0186] (1) Image feature extraction:

[0187] (a) Cut the image into L*H, a total of K equal subgraphs. For example, input a 360*240 picture, when L=3, H=2, K=6, 360 / 3=120, 240 / 2=120, cut into 6 subgraphs of 120*120.

[0188] (b) The cut subgraph is input into a neural network. The neural network is composed of 5 layers, the first 2 layers are convolution layers, convolution operation is performed, the third layer is ROI pooling, and the convolution feature map is scaled to a certain predefined size (since the input picture may be inconsistent in size, the size of the cut subgraph is also different, and the size of the convolution feature map is also different, therefore, ROI pooling is needed to scale), and the last 2 layers are LSTM units, which input the pooled features to obtain the final feature vector of the image. The feature vector is the feature space of the image.

[0189] (2) Text feature extraction:

[0190] (a) First, the text is segmented, and then converted into a vector by Word2Vec;

[0191] (b) Input into a 2-layer LSTM network to obtain its feature vector, which is the feature space of the text.

[0192] (3) Feature generation module:

[0193] The modal classification module accepts the image and text feature vectors output by steps (1) and (2) during training and inputs them into the following structure:

[0194] (a) Label prediction: a 2-layer fully connected network is used for semantic label prediction. The purpose of label prediction is to train the feature space to distinguish different semantic information. The loss function of label prediction is:

[0195]

[0196] where L imd (θ imd ) represents the cross-entropy loss of the semantic categories of all image and text instances O i , i = 1,..., n, and n is the total number of instances in the model. θ imd represents the classifier parameters of the label prediction part, v i represents the image, t i represents the text, y i represents the ground truth of each sample, is the generation probability distribution of each iteration of sample O i . Figure 4 C1, C2,... Cn in the above formula represent the results obtained after label prediction.

[0197] (b) Triplet constraint: a triplet loss function is used to make the distance between semantic related samples in the feature space as close as possible, and the distance between semantic unrelated samples in the feature space as far as possible. The triplet constraint loss function is:

[0198]

[0199] where θ V is the image modal parameter, and θ T is the text modal parameter. The definitions of C1, C2,... Cn are as follows:

[0200]

[0201]

[0202] ​ representative image v i with text t j semantically related, representative image v i with text t k semantically unrelated. λ and μ are hyperparameters. l2(v, t) represents the distance between image and text, which is defined as follows:

[0203] l2(v, t) = ||f V (v; θ V ) - f T (t; θ T )||2

[0204] (c) Regularization constraint: a regularization term is used to avoid network training overfitting, and the specific function is as follows:

[0205]

[0206] where W v represents the neuron weight in the image feature extraction network, W t represents the neuron weight in the text feature extraction network. l represents the number of layers of the network, l = 1,..., L. ||| | F represents the F norm.

[0207] In summary, the final loss function of the modal classification module is as follows, where α and β are hyperparameters:

[0208] L emb (θ V , θ T , θ imd ) = α · L imi + β · L imd + L reg

[0209] (4) Modal classification module:

[0210] In module (3), although the feature space we get has the label classification function and meets the triple constraint, it is still two independent spaces. In the modal classification module, we need to use a modal classifier to eliminate the difference between the two spaces. When the modal classifier cannot distinguish whether the input feature vector comes from the text space or the image space, it can be considered that the difference between the two feature spaces has been eliminated.

[0211] The modal classifier is also composed of 2 layers of fully connected layers, and the specific loss function is as follows:

[0212]

[0213] L advCross-entropy loss of modal classification representing all samples, θ D Network parameters representing the modal classification part, mi represents the ground truth modal label of each instance, and D represents the generated modal probability of each instance.

[0214] S2 model training: input the image and text samples in the training set into the model for training, and update the parameters by back propagation.

[0215] The overall model training is as follows:

[0216]

[0217]

[0218] S3 cross-modal retrieval: after training, the model can be used for retrieval. Taking image searching text as an example, input the image, map it to the feature space, and retrieve N text samples closest to the image sample in the feature space. These texts are the top N of the retrieval results.

[0219] The embodiment of the application also provides an electronic device, which comprises a processor and a memory for storing a computer program capable of running on the processor, and the processor executes the computer program to perform the steps of the method of the one or more technical solutions.

[0220] The embodiment of the application also provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are executed by the processor to implement the method of the one or more technical solutions.

[0221] The computer storage medium provided by the embodiment can be a non-transient storage medium.

[0222] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interface, indirect coupling or communication connection between devices or units, which can be electrical, mechanical or other forms.

[0223] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place or distributed to multiple network units; part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0224] In addition, each functional unit in each embodiment of the application can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0225] In some cases, any two of the above technical features can be combined into a new method technical solution without conflict.

[0226] In some cases, any two of the above technical features can be combined into a new device technical solution without conflict.

[0227] Those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps including the above method embodiments when executed; and the foregoing storage medium includes mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic discs or optical discs and various storage medium capable of storing program codes.

[0228] The above is only a specific embodiment of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A multi-modal data processing method, characterized by, The method comprises: determining a first feature vector and a second feature vector in a neural network model according to current input data; the data comprises image modal data and text modal data; determining a first loss value and a second loss value based on the first feature vector and the second feature vector respectively; the first loss value represents a semantic category prediction error of the current input data; the second loss value represents a degree of association between semantic relevance and distance between the current input data and historical input data; the semantic relevance indicates whether the data belongs to the same semantic category; determining a third loss value based on feature vectors corresponding to all data of the current input and the historical input in the neural network model; the third loss value represents a degree of difference between the image modal and the text modal; calculating a loss parameter based on the first loss value, the second loss value and the fourth loss value; wherein the fourth loss value represents a fitting degree of the neural network model; the calculation of the first loss value, the second loss value and the fourth loss value all belong to a feature generation module of the neural network model, and the calculation of the third loss value belongs to a modal classification module of the neural network model; after completing a predetermined number of iteration calculations for each loss value, calculating the minimum difference and the maximum difference between the loss parameter and the third loss value; updating the first network parameter, the image modal parameter and the text modal parameter of the neural network model based on the first network parameter, the image modal parameter and the text modal parameter corresponding to the minimum difference between the loss parameter and the third loss value; wherein the first network parameter is a network parameter corresponding to a network in the feature generation module for calculating the first loss value; updating the second network parameter of the neural network model based on the second network parameter corresponding to the maximum difference between the loss parameter and the third loss value; wherein the second network parameter is a network parameter corresponding to a network in the modal classification module for calculating the third loss value.

2. The method of claim 1, wherein, The method further comprises: determining a first loss value based on the first feature vector and the second feature vector in the iteration calculation of the neural network model; the iteration result comprises probabilities of the first feature vector and the second feature vector corresponding to a plurality of semantic categories respectively; determining a second loss value based on the distance between the first feature vector, the second feature vector and other feature vectors corresponding to the historical input data.

3. The method of claim 2, wherein, The method further comprises: processing the current input image modal data based on an image feature extraction network in the neural network model to obtain the first feature vector; processing the current input text modal data based on a text feature extraction network in the neural network model to obtain the second feature vector; The method further comprises: Determine a fourth loss value representing a fitting degree of the neural network model based on the number of layers and neuron weights of the image feature extraction network and the number of layers and neuron weights of the text feature extraction network.

4. The method of claim 3, wherein, The image feature extraction network in the neural network model processes the current input image modality data to obtain a first feature vector, including: Segmenting the current input image modality data into a preset number of image modality sub-data; The image feature extraction network in the neural network model processes the preset number of image modality sub-data to obtain a first feature vector.

5. The method of claim 3, wherein, Determine a first loss value based on the iterative results of the first feature vector and the second feature vector in the iterative calculation of the neural network model, including: Obtain the first network parameters corresponding to the network used to calculate the first loss value; Determine the first loss value based on the first network parameters and the iterative results of the first feature vector and the second feature vector in the iterative calculation of the neural network model; Determine a second loss value based on the distance between the first feature vector, the second feature vector and other feature vectors corresponding to the historical input data, including: Obtain the image modality parameters and text modality parameters in the neural network model; Determine the reference feature vectors of the first feature vector and the second feature vector in the historical input data corresponding other feature vectors; The reference feature vectors include semantic related feature vectors and semantic independent feature vectors; Calculate the distance between the first feature vector, the second feature vector and the reference feature vectors based on the image modality parameters and text modality parameters; Determine the second loss value based on the distance.

6. The method of claim 5, wherein, Determine a third loss value based on the feature vectors corresponding to the current input and historical input data in the neural network model, including: Obtain the second network parameters corresponding to the network used to calculate the third loss value; Determine the third loss value based on the second network parameters and the feature vectors corresponding to the current input and historical input data in the neural network model.

7. The method of claim 1, wherein, The method further includes: Input the image modality data to be searched into the trained neural network model; Determine the text modality data corresponding to the image modality data to be searched through the trained neural network model.

8. An electronic device, comprising: The electronic device includes a processor and a memory for storing a computer program capable of running on the processor; wherein, The processor executes the computer program to perform the steps of the multi-modal data processing method of any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions; the computer executable instructions are executed by the processor to realize the multi-modal data processing method of any one of claims 1 to 7.

Images