A method and system for training multiple models combined with multimodal data

Abstract

This invention provides a method and system for multimodal data multi-model combination training, relating to the field of computer technology. The method includes: acquiring a multimodal sample dataset containing multiple samples; extracting features from the multimodal sample dataset to obtain multimodal feature vectors; aligning the multimodal feature vectors to obtain a joint feature vector; inputting the joint feature vector into a multimodal combination model to obtain a multimodal training dataset; determining a multimodal combination loss function based on the multimodal sample dataset and the multimodal training dataset; and performing combined training on the multimodal combination model based on the multimodal combination loss function to obtain a trained multimodal combination model. According to this invention, the combined training can be optimized based on the alignment of the multimodal feature vectors, thereby obtaining a trained multimodal combination model that can simultaneously handle multiple different types of modal data and achieves better training results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a multi-modal data multi-model combined training method and system. BACKGROUND

[0002] In the prior art, although a multi-modal model capable of processing multi-modal data is obtained through module combination training, the influence of the multi-modal feature vector alignment operation on the combined training result is not considered, that is, the combined training cannot be optimized according to the multi-modal feature vector alignment.

[0003] The information disclosed in the background section of this application is only intended to deepen the understanding of the general background of the application and should not be regarded as acknowledging or implying in any form that the information constitutes prior art known to those skilled in the art. SUMMARY

[0004] The present application provides a multi-modal data multi-model combined training method and system, which can solve the technical problem that the combined training cannot be optimized according to the multi-modal feature vector alignment in the related art.

[0005] According to a first aspect of the present application, a multi-modal data multi-model combined training method is provided, comprising: obtaining a multi-modal sample data set of a plurality of samples, wherein the multi-modal sample data set comprises image data, text data and time series data; performing feature extraction on the multi-modal sample data set respectively to obtain multi-modal feature vectors, wherein the multi-modal feature vectors comprise image feature vectors, text feature vectors and time series feature vectors; performing multi-modal alignment on the multi-modal feature vectors to obtain joint feature vectors; inputting the joint feature vectors into a multi-model combined model to obtain a multi-modal training data set, wherein the multi-model combined model comprises an image generation model, a text reconstruction model and a time series prediction model, and the multi-modal training data set comprises generated image data, reconstructed text data and predicted time series data; determining a multi-model combined loss function according to the multi-modal sample data set and the multi-modal training data set; performing combined training on the multi-model combined model according to the multi-model combined loss function to obtain a trained multi-model combined model; wherein determining the multi-model combined loss function according to the multi-modal sample data set and the multi-modal training data set comprises: obtaining a generated gradient amplitude of each pixel point of the generated image data; obtaining a real gradient amplitude of each pixel point of the image data; determining an image consistency loss function according to the generated image data, the image data, the generated gradient amplitude and the real gradient amplitude; performing word segmentation on the reconstructed text data and the text data respectively; performing feature extraction on each word to obtain semantic vectors; determining a text reconstruction loss function according to the semantic vectors; determining a time series pattern loss function according to the predicted time series data and the time series data; and determining the multi-model combined loss function according to the image consistency loss function, the text reconstruction loss function and the time series pattern loss function.

[0006] Further, the multi-modal feature vectors are aligned to obtain joint feature vectors, comprising: mapping the multi-modal feature vectors to a shared D-dimensional hidden space through a fully connected layer to obtain multi-modal mapping feature vectors, wherein the multi-modal mapping feature vectors comprise image mapping feature vectors, text mapping feature vectors and time series mapping feature vectors, and D is a positive integer; determining an alignment loss function according to the multi-modal mapping feature vectors; training mapping parameters of the fully connected layer using the alignment loss function until the alignment loss function converges; after the mapping parameter training is completed, mapping the multi-modal feature vectors through the trained fully connected layer to obtain multi-modal aligned feature vectors, wherein the multi-modal aligned feature vectors comprise image aligned feature vectors, text aligned feature vectors and time series aligned feature vectors; and performing weighted splicing on the multi-modal aligned feature vectors to obtain joint feature vectors.

[0007] Further, according to the multi-modal mapping feature vector, an alignment loss function is determined, including: according to the formula: determining an alignment loss function , wherein U is a set of pairs of multi-modal mapping feature vectors of the same sample, and is a pair of vectors in the set U, , is the Pearson correlation coefficient between and , is the L1 norm of the vector, is the first preset weight.

[0008] Further, according to the generated image data, the image data, the generated gradient amplitude and the real gradient amplitude, an image consistency loss function is determined, including: according to the formula: determining an image consistency loss function , wherein is the pixel value of the generated image data of the i-th sample at coordinates (x, y), is the pixel value of the image data of the i-th sample at coordinates (x, y), is the gradient amplitude of the generated image data of the i-th sample in the x and y directions, is the gradient amplitude of the image data of the i-th sample in the x and y directions, is the second preset weight, N is the number of samples in a training batch, H is the length of the generated image data, W is the width of the generated image data, i≤N, x≤H, y≤W, and i, x, y, N, H and W are all positive integers.

[0009] Further, according to the semantic vector, a text reconstruction loss function is determined, including: according to the formula: determining a text reconstruction loss function , wherein is the semantic vector of the j-th word in the i-th sample reconstructed text data, is the semantic vector of the j-th word in the i-th sample text data, is the L2 norm of the vector, N is the number of samples in a training batch, M is the number of words in the text, i≤N, j≤M, and i, j, N and M are all positive integers.

[0010] Further, according to the predicted time series data and the time series data, a time series pattern loss function is determined, including: according to the formula: determining a time series pattern loss function , wherein is the value of the predicted time series data of the i-th sample at time point t, is a value of the time series data of the i-th sample at a time point t, is a first-order derivative with respect to time, is a second-order derivative with respect to time, is a weight factor of the first-order derivative error, is a weight factor of the second-order derivative error, N is a number of samples in a training batch, K is a length of predicted time series data, i≤N, t≤K, and i, t, N and K are all positive integers.

[0011] Further, according to the image consistency loss function, the text reconstruction loss function and the time series pattern loss function, a multi-model combination loss function is determined, including: according to the formula: determining a multi-model combination loss function , wherein, is an image consistency loss function, is a text reconstruction loss function, is a time series pattern loss function, is a preset basic weight of is a preset basic weight of is a preset basic weight of

[0012] According to the second aspect of the present application, a multi-modal data multi-model combination training system is provided, comprising: a multi-modal sample data set module for obtaining a multi-modal sample data set of a plurality of samples, wherein the multi-modal sample data set comprises image data, text data and time series data; a multi-modal feature vector module for respectively performing feature extraction on the multi-modal sample data set to obtain a multi-modal feature vector, wherein the multi-modal feature vector comprises an image feature vector, a text feature vector and a time series feature vector; a joint feature vector module for multi-modal alignment of the multi-modal feature vector to obtain a joint feature vector; a multi-modal training data set module for inputting the joint feature vector into a multi-model combination model to obtain a multi-modal training data set, wherein the multi-model combination model comprises an image generation model, a text reconstruction model and a time series prediction model, and the multi-modal training data set comprises generated image data, reconstructed text data and predicted time series data; a multi-model combination loss function module for determining a multi-model combination loss function according to the multi-modal sample data set and the multi-modal training data set; a trained multi-model combination model module for combination training of the multi-model combination model according to the multi-model combination loss function to obtain a trained multi-model combination model.

[0013] Technical effects: According to the present application, by processing the three different types of data of image, text and time sequence in the multi-modal sample data set, first feature extraction, multi-modal alignment to obtain joint feature vector, then input multi-model combination model, and according to the comprehensive and detailed processing mode of multi-model combination loss function, the trained multi-model combination model which can process multiple different types of modal data and has better training effect is obtained. When determining the alignment loss function, the alignment degree between the multi-modal mapping feature vectors can be measured from the linear correlation and the numerical proximity by combining the Pearson correlation coefficient and the L1 norm loss of the vector element level operation. In the mapping parameter training process, minimizing the alignment loss function can promote the better alignment of the multi-modal mapping feature vectors in the shared space, thereby improving the processing ability and performance of the multi-model combination model for different modal data. When determining the image consistency loss function, the image consistency loss function can be determined by generating the pixel value difference and gradient amplitude difference between the image data and the image data, thereby improving the similarity between the generated image data and the image data, improving the quality and performance of the image generation model, and effectively generating images with sharper vision and clearer texture. When determining the text reconstruction loss function, the text reconstruction loss function can be determined by calculating the L2 norm difference between the reconstructed text data and the semantic vector of each word in the text data, so that the reconstructed text data is semantically close to the text data, thereby improving the performance and reconstruction quality of the text reconstruction model in the text processing task. When determining the time sequence form loss function, the time sequence form loss function can be determined based on the square error of the value, the first derivative and the second derivative, so that the time sequence prediction model learns and reproduces the dynamic change pattern of the time sequence data, thereby improving the similarity between the predicted time sequence data and the time sequence data in value, change trend and change trend, and improving the performance of the time sequence prediction model. When determining the multi-model combination loss function, the multi-model combination loss function can be determined based on the product of the adaptive weight of each loss function and the loss function, thereby realizing the stability and balance of multi-task joint training, and improving the performance of the multi-model combination model in image processing, text processing and time sequence prediction.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory, but not limiting the present application. Other features and aspects of the present application will be more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other embodiments from these drawings without creative labor.

[0016] Figure 1 An exemplary flowchart of a multi-modal data multi-model combination training method according to an embodiment of the present application is shown;

[0017] Figure 2 An exemplary flowchart of calculating a joint feature vector according to an embodiment of the present application is shown;

[0018] Figure 3 An exemplary flowchart of calculating a multi-model combination loss function according to an embodiment of the present application is shown;

[0019] Figure 4 An exemplary block diagram of a multi-modal data multi-model combination training system according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] In order to make the objects, technical solutions and advantages of embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0021] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and some embodiments may not be described again for the same or similar concepts or processes.

[0022] Figure 1An exemplary flowchart of a multi-modal data multi-model combined training method according to an embodiment of the present application is shown. The method includes: step S1, obtaining a multi-modal sample data set of a plurality of samples, wherein the multi-modal sample data set includes image data, text data, and time series data; step S2, respectively performing feature extraction on the multi-modal sample data set to obtain a multi-modal feature vector, wherein the multi-modal feature vector includes an image feature vector, a text feature vector, and a time series feature vector; step S3, performing multi-modal alignment on the multi-modal feature vector to obtain a joint feature vector; step S4, inputting the joint feature vector into a multi-model combined model to obtain a multi-modal training data set, wherein the multi-model combined model includes an image generation model, a text reconstruction model, and a time series prediction model, and the multi-modal training data set includes generated image data, reconstructed text data, and predicted time series data; step S5, determining a multi-model combined loss function according to the multi-modal sample data set and the multi-modal training data set; and step S6, performing combined training on the multi-model combined model according to the multi-model combined loss function to obtain a trained multi-model combined model.

[0023] The multi-modal data multi-model combined training method according to the embodiment of the present application can comprehensively and carefully process the image, text, and time series data in the multi-modal sample data set by respectively performing feature extraction, multi-modal alignment to obtain a joint feature vector, inputting the joint feature vector into a multi-model combined model, and performing combined training according to a multi-model combined loss function, thereby obtaining a trained multi-model combined model that can simultaneously process a plurality of different types of modal data and has better training effect.

[0024] According to one embodiment of the present application, in step S1, the specific application scenario to which the multi-modal sample data set is applied is obtained, for example, if applied to the field of intelligent transportation, the traffic scene images (image data) of traffic monitoring cameras in different areas of the city, the text records (text data) of the traffic management department, and the traffic flow sensors, speed sensors, etc. (time series data) installed on the road are focused on, if used in the field of medical health, the medical images (image data) of different patients, medical records (text data), and the changes of vital signs over time (time series data) are developed around. The integrated multi-modal sample data set is checked for quality to check if there are missing, incorrect, abnormal values, etc. For example, check if the image data is clear and complete, if the text data has garbled characters, unclear semantics, etc., if the time series data has data jumps, unreasonable fluctuations, etc. If there is a quality problem, the sample data is removed. According to the needs of subsequent model training, the multi-modal sample data set is subjected to necessary preprocessing operations, for example, image data is subjected to cropping, scaling, normalization, etc., text data is subjected to word segmentation, removal of stop words, normalization, etc., time series data is subjected to smoothing filtering, segmentation, normalization, etc.

[0025] According to one embodiment of the present application, in step S2, feature extraction is performed on the image data (for example, a deep learning model), obtaining an image feature vector, semantic encoding is performed on the text data (for example, a word embedding method), obtaining a text feature vector, and segmentation filtering and feature fusion are performed on the time series data, obtaining a time series feature vector, that is, the time series data is divided into multiple data segments (the last segment can be left) according to the time window, adaptive median filtering is performed on the first few data segments to remove noise interference, time domain features and frequency domain features are extracted to form local feature vectors, and attention weighted fusion is performed on all local feature vectors to obtain a time series feature vector.

[0026] According to one embodiment of the present application, in step S3, the multi-modal feature vectors are aligned to obtain a joint feature vector.

[0027] Figure 2 An exemplary flowchart of calculating a joint feature vector according to an embodiment of the present application is shown.

[0028] According to one embodiment of the present application, step S3 comprises: step S31, mapping the multi-modal feature vectors to a shared D-dimensional hidden space through a fully connected layer to obtain multi-modal mapping feature vectors, wherein the multi-modal mapping feature vectors comprise image mapping feature vectors, text mapping feature vectors and time sequence mapping feature vectors, and D is a positive integer; step S32, determining an alignment loss function according to the multi-modal mapping feature vectors; step S33, training mapping parameters of the fully connected layer using the alignment loss function until the alignment loss function converges; step S34, after the training of the mapping parameters is completed, obtaining multi-modal alignment feature vectors by passing the multi-modal feature vectors through the trained fully connected layer, wherein the multi-modal alignment feature vectors comprise image alignment feature vectors, text alignment feature vectors and time sequence alignment feature vectors; and step S35, obtaining a joint feature vector by weightedly splicing the multi-modal alignment feature vectors.

[0029] According to one embodiment of the present application, the multi-modal feature vectors are respectively input into a fully connected layer, wherein each kind of feature vector corresponds to one fully connected layer. The fully connected layer is a kind of neural network structure, in which each neuron is connected with all neurons of the previous layer. Through the linear transformation of the fully connected layer and the action of the non-linear activation function (such as ReLU, etc.), the feature vectors of different modalities are mapped from their original feature space to a pre-set shared D-dimensional hidden space. After the mapping of the fully connected layer, the multi-modal mapping feature vectors in the shared D-dimensional hidden space are obtained. By introducing an explicit similarity-based alignment loss function in the early training stage, the cross-modal semantic alignment is forced to be realized directly from the root of the feature representation, which is a kind of “hard alignment” or “representation layer alignment”, and provides a semantic consistent and structure clear joint feature basis for the subsequent generation and prediction tasks. The purpose of the alignment loss function is to measure the alignment degree between the multi-modal mapping feature vectors, that is, the multi-modal mapping feature vectors are as close as possible in the shared hidden space to realize the information alignment between modalities. The alignment loss function is used as the optimization objective of training the fully connected layer. In the training process, the gradient of the loss function with respect to the mapping parameters such as weights and biases in the fully connected layer is calculated through the back propagation algorithm, and then the mapping parameters are updated according to the gradient information using the optimization algorithm (such as stochastic gradient descent, Adam, etc.) to gradually reduce the value of the alignment loss function. The above mapping parameter updating process is repeatedly performed until the alignment loss function converges to meet the pre-set convergence condition, that is, the alignment loss function decreases by less than the convergence threshold (for example, 1x10 -5 for example, when the alignment loss function decreases by less than the convergence threshold in continuous 5 training periods, the specific definition is: wherein, is the current training period, for the E-th training cycle, for the current training cycle, for the E-th training cycle, Q = 5, is a convergence threshold, it is determined that the alignment loss function has converged. Convergence indicates that the multi-modal mapping feature vectors have reached a good alignment state in the shared hidden space, at which time the mapping parameters of the fully connected layer have learned the ability to effectively map multi-modal feature vectors to the shared space and perform alignment. After the mapping parameter training is completed, the initially obtained multi-modal feature vectors are input into the trained fully connected layer again. After mapping through the trained fully connected layer, multi-modal alignment feature vectors are obtained, which have better consistency and comparability in the shared hidden space. The weighted image alignment feature vectors, text alignment feature vectors and time sequence alignment feature vectors are spliced in a certain order (for example, image alignment feature vectors-text alignment feature vectors-time sequence alignment feature vectors) to obtain joint feature vectors. For example, the image alignment feature vectors are assigned a weight value of 0.4, the text alignment feature vectors are assigned a weight value of 0.3, and the time sequence alignment feature vectors are assigned a weight value of 0.3. Through weighting, a higher weight can be allocated to image features to capture scene details. The joint feature vectors realize the deep fusion of images, texts and time sequence data in a unified semantic space, so that the information of different modalities can be enhanced and supplemented with each other, thereby improving the comprehensive reasoning ability of the multi-model system in complex tasks.

[0030] According to one embodiment of the present application, the alignment loss function is determined according to the multi-modal mapping feature vectors, comprising: determining the alignment loss function according to formula (1) ,

[0031] (1),

[0032] wherein U is a set of pairs of multi-modal mapping feature vectors of the same sample, and is a pair of vectors in the set U, , is a Pearson correlation coefficient between and , is an L1 norm of a vector, is a first preset weight.

[0033] According to one embodiment of the present application, in formula (1), is a Pearson correlation coefficient between and , is a Pearson correlation coefficient between The degree of synchronization of changes in two vectors across various dimensions focuses on the shape and trend of change of the vectors, rather than their absolute numerical magnitude. Even if the numerical ranges of the two vectors are different, as long as the change patterns are consistent (e.g., one increases and the other also increases), the Pearson correlation coefficient will be high. The squared loss term is calculated by subtracting the Pearson correlation coefficient from 1 and then squaring it. If the Pearson correlation coefficient between two vectors is 1 (perfectly positive correlation), then the squared loss term is 0. For all multimodal mapping eigenvector pairs in set U The squared loss term is averaged, where set U includes three pairs of multimodal mapping feature vectors: (image mapping feature vector, text mapping feature vector), (image mapping feature vector, time-series mapping feature vector), and (text mapping feature vector, time-series mapping feature vector). These pairs originate from the same sample, such as medical images (image data), medical records (text data), and changes in vital signs over time (time-series data) of the same patient. This allows for a focus on the covariant patterns of feature distribution, rather than spatial perspective, to better capture deep, statistically significant correlations across modalities. for and The sum of the absolute values ​​of all elements in the difference vector of two vectors represents the magnitude of the difference vector. The L1 norm is relatively insensitive to outliers and can better handle noise and anomalies in the data. To average and weight the difference vectors of all multimodal mapping eigenvector pairs in set U, A first preset weight (e.g., 0.3) is used to make the multimodal mapping feature vector pairs numerically similar. and The sum of these values ​​yields the alignment loss function. During the training of the mapping parameters, minimizing this alignment loss function promotes better alignment of the multimodal mapping feature vectors in the shared space.

[0034] In this way, by combining the Pearson correlation coefficient and the L1 norm loss of vector element-level operations, the alignment between multimodal mapping feature vectors can be measured from the perspectives of linear correlation and numerical proximity. During the training of mapping parameters, minimizing the alignment loss function can promote better alignment of multimodal mapping feature vectors in the shared space, thereby improving the processing capability and performance of multi-model combination models for different modal data.

[0035] According to an embodiment of the present application, in step S4, the image generation model can be a deep learning neural network model such as a generative adversarial network model, which can output generated image data based on the input joint feature vector. The text reconstruction model can be a deep learning neural network model such as a recurrent neural network, which can output reconstructed text data based on the input joint feature vector. The time series prediction model can be a deep learning neural network model such as a recurrent neural network, which can output predicted time series data based on the input joint feature vector. The generated image data output by the image generation model, the reconstructed text data output by the text reconstruction model, and the predicted time series data output by the time series prediction model are integrated according to certain data organization and storage formats to form a set containing multiple modal training data, i.e., a multi-modal training data set. The multi-modal training data set can differ from the multi-modal sample data set, and therefore, the multi-model combination model can be trained to reduce the above-mentioned difference.

[0036] According to an embodiment of the present application, in step S5, a multi-model combination loss function is determined based on the multi-modal sample data set and the multi-modal training data set.

[0037] Figure 3 An exemplary flowchart of calculating a multi-model combination loss function according to an embodiment of the present application is shown.

[0038] According to an embodiment of the present application, step S5 includes: step S51, obtaining a generated gradient amplitude of each pixel point of the generated image data; step S52, obtaining a real gradient amplitude of each pixel point of the image data; step S53, determining an image consistency loss function based on the generated image data, the image data, the generated gradient amplitude, and the real gradient amplitude; step S54, performing word segmentation on the reconstructed text data and the text data, respectively; step S55, performing feature extraction on each word to obtain a semantic vector; step S56, determining a text reconstruction loss function based on the semantic vector; step S57, determining a time series pattern loss function based on the predicted time series data and the time series data; and step S58, determining a multi-model combination loss function based on the image consistency loss function, the text reconstruction loss function, and the time series pattern loss function.

[0039] According to one embodiment of the present application, after the image generation model generates image data, for each pixel point in the generated image data, a specific gradient calculation method (such as a gradient calculation method based on an edge detection operator such as a Sobel operator, a Prewitt operator, etc.) is used to calculate the generated gradient amplitude. The generated gradient amplitude reflects the change intensity of the pixel points in the generated image data in space, and embodies the edge and texture features of the generated image. For the image data (i.e. the real image data), the same gradient calculation method as that used to obtain the generated gradient amplitude is used to calculate each pixel point, thereby obtaining the real gradient amplitude, which represents the actual change of the edges and textures of the real image. The image consistency loss function is a measure of the consistency of the generated image data and the image data in terms of gradient features. For the reconstructed text data and the text data, a word segmentation tool (for example, a rule-based word segmentation method, a statistical-based word segmentation method, or a deep learning-based word segmentation method, etc.) is used to segment the text into individual words, and the number of segmented words of the reconstructed text data and the text data is consistent. Word segmentation is a basic step in text processing, which can convert continuous text sequences into discrete word units, facilitating subsequent feature extraction and processing. A word embedding technique (for example, Word2Vec, GloVe, etc.) is used to extract features from each segmented word and map it to a low-dimensional vector space to obtain a semantic vector for each word. The semantic vector can capture the semantic information of the word, so that words with similar semantics are closer in the vector space, thereby providing a basis for subsequent text similarity calculation. The text reconstruction loss function is used to measure the similarity of the reconstructed text data and the text data at the semantic level. According to the predicted time series data and the time series data, a time series pattern loss function is constructed, which is a measure of the similarity of the predicted time series data and the time series data in terms of pattern. The multi-model combination loss function is a comprehensive index that integrates the image consistency loss function, the text reconstruction loss function, and the time series pattern loss function to comprehensively evaluate the performance of the multi-model combination model in each modality, thereby providing a unified guidance standard for the optimization and training of the multi-model combination model and improving the performance of the multi-model combination model in multiple modalities.

[0040] According to one embodiment of the present application, the image consistency loss function is determined according to the generated image data, the image data, the generated gradient amplitude and the real gradient amplitude, comprising: determining the image consistency loss function according to formula (2) ,

[0041] (2),

[0042] wherein, is the pixel value of the generated image data of the i th sample at coordinates (x, y), is a pixel value of the image data of the i th sample at coordinates (x, y), is a gradient amplitude of the generated image data of the i th sample in the x and y directions, is a gradient amplitude of the image data of the i th sample in the x and y directions, is a second preset weight, N is a number of samples in a training batch, H is a length of the generated image data, W is a width of the generated image data, i≤N, x≤H, y≤W, and i, x, y, N, H, and W are all positive integers.

[0043] According to one embodiment of the present application, in formula (2), is a comparison of the brightness values of each pixel point of the two images of the generated image data and the image data. If the pixel values of the two images are exactly the same, this part is 0. is a comparison of the differences in edges and textures of the two images of the generated image data and the image data. Places with large gradients correspond to the edges (object outlines, texture changes, etc.) of the image. The smaller this loss is, the more similar the generated image data and the image data have in edge structure, so that the generated image is clearer and the texture is more realistic, wherein, is a second preset weight (for example, 0.5). For all samples, and are added and averaged to obtain the image consistency loss function. In the training process, the image consistency loss function is minimized to improve the similarity between the generated image data and the image data, so that the image generation model can effectively generate images that are more visually sharp and have clearer textures when generating images.

[0044] In this way, the image consistency loss function can be determined by the pixel value difference and the gradient amplitude difference between the generated image data and the image data, thereby improving the similarity between the generated image data and the image data, improving the quality and performance of the image generation model, and effectively generating images that are more visually sharp and have clearer textures.

[0045] According to one embodiment of the present application, the text reconstruction loss function is determined according to the semantic vector, including: determining the text reconstruction loss function according to formula (3) ,

[0046] (3),

[0047] wherein, is a semantic vector of the j th word in the reconstructed text data of the i th sample, is a semantic vector of the j th word in the text data of the i th sample, L2 norm of a vector, N is the number of samples in a training batch, M is the number of words in the text, i≤N, j≤M, and i, j, N and M are positive integers.

[0048] According to one embodiment of the present application, in formula (3), L2 norm of the i-th sample between the semantic vector of the j-th word in the reconstructed text data and the semantic vector of the j-th word in the i-th sample text data, i.e. Euclidean distance, the smaller the Euclidean distance, the closer the two semantic vectors, i.e. the semantic representation of the reconstructed text data on the word is more consistent with the sample text data. The Euclidean distance of the plurality of samples between the sample text data and the plurality of words of the reconstructed text data is averaged, and the text reconstruction loss function is obtained, and in the text reconstruction model training process, by minimizing the text reconstruction loss function, the reconstructed text data is closer to the text data in semantics.

[0049] In this way, the text reconstruction loss function can be determined by calculating the L2 norm difference of each word semantic vector in the reconstructed text data and the text data, so that the reconstructed text data is close to the text data in semantics, thereby improving the performance and reconstruction quality of the text reconstruction model in the text processing task.

[0050] According to one embodiment of the present application, the time series pattern loss function is determined according to the predicted time series data and the time series data, comprising: determining the time series pattern loss function according to formula (4) ,

[0051] (4),

[0052] wherein, is the value of the predicted time series data of the i-th sample at time point t, is the value of the time series data of the i-th sample at time point t, is the first order derivative with respect to time, is the second order derivative with respect to time, is a weight factor of the first order derivative error, is a weight factor of the second order derivative error, N is the number of samples in a training batch, K is the length of the predicted time series data, i≤N, t≤K, and i, t, N and K are positive integers.

[0053] According to one embodiment of the present application, in formula (4), is the squared error between the value of the predicted time series data of the i-th sample at time point t and the value of the time series data of the i-th sample at time point t, indicating the squared error between the predicted value and the true value, To predict the square error between the first derivative of the predicted time series data at time point t and the first derivative of the time series data at time point t, represents the square error between the predicted value and the real value in the change rate, measures whether the change trend of the predicted value is consistent with the real value, wherein, is a weight factor of the first derivative error (for example, 1.2). To predict the square error between the second derivative of the predicted time series data at time point t and the second derivative of the time series data at time point t, represents the square error between the predicted value and the real value in the curvature, measures whether the change of the change trend of the predicted value is consistent with the real value, and can make the time series prediction model capture more fine dynamics, such as turning points of trends, smoothness of changes, etc., wherein, is a weight factor of the second derivative error (for example, 0.8). For all samples, the three terms of , and are added and averaged to obtain a time series pattern loss function, which is minimized in the training process, thereby improving the similarity between the predicted time series data and the time series data in the numerical value, change trend and change of change trend.

[0054] In this way, the time series pattern loss function can be determined based on the square error of the numerical value, the first derivative and the second derivative, so that the time series prediction model learns and reproduces the dynamic change pattern of the time series data, thereby improving the similarity between the predicted time series data and the time series data in the numerical value, change trend and change of change trend, and improving the performance of the time series prediction model.

[0055] According to one embodiment of the present application, the multi-model combination loss function is determined according to the image consistency loss function, the text reconstruction loss function and the time series pattern loss function, comprising: determining the multi-model combination loss function according to formula (5) ,

[0056] (5),

[0057] wherein, is the image consistency loss function, is the text reconstruction loss function, is the time series pattern loss function, is a preset basic weight of , is a preset basic weight of , is a preset basic weight of , is a preset constant.

[0058] According to one embodiment of the present application, in formula (5), wherein, the preset basic weight of is, for example, 1.3. When the image consistency loss function is larger, the adaptive weight is smaller. Similarly, is the preset basic weight of is, for example, 1.3. When the image consistency loss function is larger, the adaptive weight is smaller. Similarly, is the adaptive weight of the text reconstruction loss function and the adaptive weight of the time series pattern loss function, which will not be described herein again, wherein, is the preset basic weight of is, for example, 1, is the preset basic weight of is, for example, 0.7. When the value of a certain loss function is larger, the adaptive weight of the loss function is relatively reduced, so as to avoid the loss function from dominating in the total loss, and make each loss function play a more balanced role in the training process. When the value of a certain loss function is smaller, the adaptive weight of the loss function is relatively increased, which helps the multi-model combination model pay more attention to the task represented by the loss function, and improve the performance of the multi-model combination model on the task. The image consistency loss function, the text reconstruction loss function and the time series pattern loss function are weighted and summed to obtain a multi-model combination loss function, so as to minimize the multi-model combination loss function, and thus improve the performance of the multi-model combination model on multiple tasks such as image processing, text processing and time series prediction.

[0059] In this way, the adaptive weight of each loss function and the loss function are multiplied to determine the multi-model combination loss function, so as to realize stable and balanced multi-task joint training, and improve the performance of the multi-model combination model on multiple tasks such as image processing, text processing and time series prediction.

[0060] According to an embodiment of the present application, in step S6, the multi-model combination model is trained according to the multi-model combination loss function to obtain a trained multi-model combination model.

[0061] According to one embodiment of the present application, the parameters of the image generation model and the text reconstruction model are fixed, the time series prediction model is trained using the time series shape loss function, the parameters of the time series prediction model and the text reconstruction model are fixed, the image generation model is trained using the image consistency loss function, the parameters of the image generation model and the time series prediction model are fixed, the text reconstruction model is trained using the text reconstruction loss function, and the above steps are alternately performed until the multi-model combination loss function converges. The convergence mode is similar to that of the alignment loss function, which will not be described here. The trained multi-model combination model has the ability to process and generate multi-modal data, and has various application scenarios and methods in actual use, for example, improving the understanding and response ability of an intelligent interaction system (for example, an intelligent customer service, a virtual assistant), so that it can handle complex queries containing images, text and time series information, and give more accurate responses. Users can interact with the system through text, uploaded images or provide time series related data (such as operation records within a period of time). The system uses the multi-model combination model, the text reconstruction model understands the text intent, the image generation model assists the visual response (such as generating related pictures according to user description), and the time series prediction model analyzes the time series data and provides suggestions based on time trends. For example, a user asks for a travel itinerary plan and uploads a destination picture, the system combines text planning and picture information to generate a detailed itinerary, and can also predict the best travel time based on historical tourism data.

[0062] The multi-modal data multi-model combined training method according to the embodiment of the present application, through the image, text and time sequence three different types of data in the multi-modal sample data set, first respectively feature extraction, multi-modal alignment to obtain joint feature vector, then input multi-model combined model, and according to the comprehensive and detailed processing mode of multi-model combined loss function, obtain the trained multi-model combined model which can process multiple different types of modal data and has better training effect. When determining the alignment loss function, the alignment degree between the multi-modal mapping feature vectors can be measured from the linear correlation and the numerical proximity by combining the Pearson correlation coefficient and the L1 norm loss of the vector element level operation. In the mapping parameter training process, minimizing the alignment loss function can promote the multi-modal mapping feature vectors to better align in the shared space, thereby improving the processing ability and performance of the multi-model combined model for different modal data. When determining the image consistency loss function, the image consistency loss function can be determined by generating the pixel value difference and gradient amplitude difference between the image data and the image data, thereby improving the similarity between the generated image data and the image data, improving the quality and performance of the image generation model, and effectively generating images that are more visually sharp and have clearer textures. When determining the text reconstruction loss function, the text reconstruction loss function can be determined by calculating the L2 norm difference of the reconstructed text data and the semantic vector of each word in the text data, so that the reconstructed text data is semantically close to the text data, thereby improving the performance and reconstruction quality of the text reconstruction model in the text processing task. When determining the time sequence form loss function, the time sequence form loss function can be determined based on the square error of the value, first derivative and second derivative, so that the time sequence prediction model learns and reproduces the dynamic change pattern of the time sequence data, thereby improving the similarity between the predicted time sequence data and the time sequence data in terms of value, change trend and change trend change, and improving the performance of the time sequence prediction model. When determining the multi-model combined loss function, the multi-model combined loss function can be determined based on the product of the adaptive weight of each loss function and the loss function, thereby realizing the stability and balance of multi-task joint training, and improving the performance of the multi-model combined model in image processing, text processing and time sequence prediction.

[0063] Figure 4An example of a block diagram of a multi-modal data multi-model combined training system according to an embodiment of the present application is shown, the system comprising: a multi-modal sample data set module for obtaining a plurality of sample multi-modal sample data sets, wherein the multi-modal sample data set comprises image data, text data and time series data; a multi-modal feature vector module for respectively performing feature extraction on the multi-modal sample data set to obtain a multi-modal feature vector, wherein the multi-modal feature vector comprises an image feature vector, a text feature vector and a time series feature vector; a joint feature vector module for multi-modal alignment of the multi-modal feature vector to obtain a joint feature vector; a multi-modal training data set module for inputting the joint feature vector into a multi-model combined model to obtain a multi-modal training data set, wherein the multi-model combined model comprises an image generation model, a text reconstruction model and a time series prediction model, and the multi-modal training data set comprises generated image data, reconstructed text data and predicted time series data; a multi-model combined loss function module for determining a multi-model combined loss function according to the multi-modal sample data set and the multi-modal training data set; a trained multi-model combined model module for performing combined training on the multi-model combined model according to the multi-model combined loss function to obtain a trained multi-model combined model.

[0064] The present application can be a method, apparatus, system, and / or computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for performing various aspects of the present application.

[0065] Those skilled in the art will understand that the above description and the embodiments of the present application shown in the drawings are only examples and do not limit the present application. The purpose of the present application has been fully and effectively achieved. The functional and structural principles of the present application have been demonstrated and described in the embodiments, and the embodiments of the present application can be modified or changed in any way without departing from the principles.

Claims

1. A multi-modal data multi-model combination training method, characterized in that, The method comprises the steps of: A plurality of samples are obtained, and a plurality of modal sample data sets are obtained, wherein the plurality of modal sample data sets include image data, text data and time series data; feature extraction is performed on the plurality of modal sample data sets respectively to obtain a plurality of modal feature vectors, wherein the plurality of modal feature vectors include image feature vectors, text feature vectors and time series feature vectors; the plurality of modal feature vectors are aligned to obtain a joint feature vector; the joint feature vector is input into a multi-model combination model to obtain a plurality of modal training data sets, wherein the multi-model combination model includes an image generation model, a text reconstruction model and a time series prediction model, and the plurality of modal training data sets include generated image data, reconstructed text data and predicted time series data; a multi-model combination loss function is determined according to the plurality of modal sample data sets and the plurality of modal training data sets; the multi-model combination model is trained according to the multi-model combination loss function to obtain a trained multi-model combination model; wherein the multi-model combination loss function is determined according to the plurality of modal sample data sets and the plurality of modal training data sets, comprising: obtaining a generated gradient amplitude of each pixel point of the generated image data; obtaining a real gradient amplitude of each pixel point of the image data; determining an image consistency loss function according to the generated image data, the image data, the generated gradient amplitude and the real gradient amplitude; performing word segmentation on the reconstructed text data and the text data respectively; performing feature extraction on each word to obtain a semantic vector; determining a text reconstruction loss function according to the semantic vector; determining a time series pattern loss function according to the predicted time series data and the time series data; determining the multi-model combination loss function according to the image consistency loss function, the text reconstruction loss function and the time series pattern loss function; determining the time series pattern loss function according to the predicted time series data and the time series data, comprising: determining the time series pattern loss function according to the formula: determining the time series pattern loss function wherein, is a value of the predicted time series data of the i th sample at a time point t, is a value of the time series data of the i th sample at the time point t, is a first-order derivative with respect to time, is a second-order derivative with respect to time, is a weight factor of the first-order derivative error, is a weight factor of the second-order derivative error, N is a number of samples in a training batch, K is a length of the predicted time series data, i≤N, t≤K, and i, t, N and K are all positive integers. 2.The multi-model combination training method of multi-modal data according to claim 1, characterized in that, The multi-modal feature vectors are aligned to obtain a joint feature vector, comprising: mapping the multi-modal feature vectors to a shared D-dimensional hidden space through a full connection layer to obtain multi-modal mapping feature vectors, wherein the multi-modal mapping feature vectors comprise image mapping feature vectors, text mapping feature vectors and time sequence mapping feature vectors, D is a positive integer; determining an alignment loss function according to the multi-modal mapping feature vectors; training the mapping parameters of the full connection layer using the alignment loss function until the alignment loss function converges; after the training of the mapping parameters is completed, the multi-modal feature vectors are mapped through the trained full connection layer to obtain multi-modal alignment feature vectors, wherein the multi-modal alignment feature vectors comprise image alignment feature vectors, text alignment feature vectors and time sequence alignment feature vectors; the multi-modal alignment feature vectors are weighted and spliced to obtain a joint feature vector. 3.The multi-model combination training method of multi-modal data according to claim 2, characterized in that, According to the multi-modal mapping feature vector, determine an alignment loss function, comprising: according to formula: Determine an alignment loss function Wherein U is a set of pairs of multi-modal mapping feature vectors of the same sample, And Is a pair of vectors in the set U, , Is the Pearson correlation coefficient between And Is the L1 norm of the vector, Is the first preset weight.​ 4.The multi-model combination training method of multi-modal data according to claim 1, characterized in that, According to the generated image data, the image data, the generated gradient amplitude and the real gradient amplitude, determining an image consistency loss function, comprising: determining the image consistency loss function according to a formula: determining an image consistency loss function wherein, is a pixel value of the generated image data of the i th sample at a coordinate (x, y), is a pixel value of the image data of the i th sample at the coordinate (x, y), is a gradient amplitude of the generated image data of the i th sample in x and y directions, is a gradient amplitude of the image data of the i th sample in x and y directions, is a second preset weight, N is a sample quantity in a training batch, H is a length of the generated image data, W is a width of the generated image data, i≤N, x≤H, y≤W, and i, x, y, N, H and W are all positive integers. 5.The multi-model combination training method of multi-modal data according to claim 1, characterized in that, According to the semantic vector, a text reconstruction loss function is determined, including: according to the formula: determining a text reconstruction loss function wherein, is a semantic vector of the jth word in the ith sample reconstructed text data, is a semantic vector of the jth word in the ith sample text data, is an L2 norm of a vector, N is a number of samples in a training batch, M is a number of words in the text, i≤N, j≤M, and i, j, N and M are all positive integers. 6.The multi-model combination training method of multi-modal data according to claim 1, characterized in that, According to the image consistency loss function, the text reconstruction loss function and the time sequence form loss function, a multi-model combination loss function is determined, including: according to the formula: determining a multi-model combination loss function , wherein, is an image consistency loss function, is a text reconstruction loss function, is a time sequence form loss function, is a preset basic weight of the image consistency loss function, is a preset basic weight of the text reconstruction loss function, is a preset basic weight of the time sequence form loss function.

7. A multi-modal data multi-model combination training system, configured to perform the multi-modal data multi-model combination training method according to any one of claims 1-6. The method comprises the steps of: A multi-modal sample data set module is configured to obtain a multi-modal sample data set of a plurality of samples, wherein the multi-modal sample data set comprises image data, text data and time sequence data; a multi-modal feature vector module is configured to perform feature extraction on the multi-modal sample data set to obtain multi-modal feature vectors, wherein the multi-modal feature vectors comprise image feature vectors, text feature vectors and time sequence feature vectors; a joint feature vector module is configured to align the multi-modal feature vectors to obtain a joint feature vector; a multi-modal training data set module is configured to input the joint feature vector into a multi-model combination model to obtain a multi-modal training data set, wherein the multi-model combination model comprises an image generation model, a text reconstruction model and a time sequence prediction model, and the multi-modal training data set comprises generated image data, reconstructed text data and predicted time sequence data; a multi-model combination loss function module is configured to determine a multi-model combination loss function according to the multi-modal sample data set and the multi-modal training data set; a trained multi-model combination model module is configured to perform combination training on the multi-model combination model according to the multi-model combination loss function to obtain a trained multi-model combination model.

Images