Multi-modal data enhancement method and system based on AI training platform

Abstract

The invention discloses a multi-modal data enhancement method and system based on an AI training platform, and belongs to the technical field of multi-modal data processing.The method comprises the steps that firstly, a multi-modal data fusion pipeline is developed on the AI training platform, and data of different modals are collected through a data collection module for preprocessing; in the preprocessing stage, a multi-modal data fusion pipeline is used for carrying out feature extraction on data from different modals, and joint modeling is carried out at a specific level; then, automatically generating high-quality synthetic data by adopting a data enhancement pipeline; and finally, developing a multi-modal deep learning pipeline to fully learn and understand deep features of different modal data. According to the method, the problem of model learning bias caused by single-modal data in the prior art is solved, the generalization ability and the data processing efficiency of the model are improved, meanwhile, the processing difficulty of a multi-modal data task is simplified for an AI training platform user, and the method has important application value.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and specifically to a multimodal data enhancement method and system based on an AI training platform. Background Art

[0002] With the rapid development of artificial intelligence technology, AI training platforms are increasingly being used in various industries. They have gradually evolved from traditional single-modality processing (such as image processing, text processing, etc.) to more complex multi-modal data fusion and processing. The main function of these platforms is to provide a complete set of tool chains for training, testing, and deploying deep learning models, and integrate functions such as data processing, algorithm development, model training, and reasoning.

[0003] However, as the scale of AI models grows and the application scenarios become more complex, traditional AI R&D platforms also face some challenges: in multimodal processing scenarios, AI models require appropriate data processing strategies to enable AI models to deeply mine the features of multimodal data and learn patterns in multimodal scenarios. How to achieve multimodal data fusion by combining data from different modalities (such as images, text, sensor data, etc.) to make up for the limitations of single modality data and improve the generalization ability of the model and data processing efficiency is a problem that needs to be solved at present. In addition, when facing multimodal data processing tasks, existing AI training platforms generally have the problem of frequent manual modifications by users and low degree of platform training automation. Summary of the invention

[0004] The technical task of the present invention is to address the above shortcomings and provide a multimodal data enhancement method and system based on an AI training platform to solve the problem of model learning bias caused by single modality data in the prior art, improve the generalization ability of the model and data processing efficiency, and enhance the degree of automation of the AI ​​training platform.

[0005] The technical solution adopted by the present invention to solve its technical problem is: A multimodal data enhancement method based on an AI training platform, the implementation of the method includes: First, a multimodal data fusion pipeline is developed on the AI ​​training platform. Data of different modalities (such as images, text, sensor data, etc.) are collected through the data acquisition module and preprocessed. In the preprocessing stage, the multimodal data fusion pipeline is used to extract features from data from different modalities and perform joint modeling at a specific level. Then, a data augmentation pipeline is used to automatically generate high-quality synthetic data; Finally, a multimodal deep learning pipeline is developed to fully learn and understand the deep features of data from different modalities.

[0006] Furthermore, the multimodal data fusion pipeline is used to process data sources of different modalities and synchronize, align and fuse the data according to actual application scenarios; The multimodal data fusion pipeline consists of the following steps: (1) Data standardization: standardize multimodal data (images, text, sensor data, and speech) collected from different data sources; (2) Time series synchronization and alignment: Synchronize and align time series data (sensor data) and non-time series data (images, text) to ensure that multimodal data are effectively integrated within the same time period; (3) Data feature extraction: Multimodal feature extraction models include convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformer.

[0007] Furthermore, the multimodal data collected from different data sources are standardized, including: For image data: scale and normalize, supporting two normalization methods: the first is to adjust the image pixel value from the range of 0-255 to 0-1; the second is to normalize according to a specific mean and standard deviation, using the standardized values ​​of the ImageNet dataset: mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225]; For text data: perform word segmentation and encoding, use natural language processing tool libraries (SpaCy, NLTK) for word segmentation, and use models such as BERT and Word2Vec to convert the segmented text into word embedding vectors; For time series data: interpolation and alignment are performed to ensure that all data can be analyzed on the same time axis or in the same dimension; The timing synchronization and alignment are specifically implemented as follows: Slice the entire time series into multiple time periods, each of which contains data from multiple modes, and fill in missing time points through interpolation; The data feature extraction specifically includes: Image data feature extraction: Use a pre-trained convolutional neural network (CNN) ResNet model to extract high-level features from images; the pre-trained model has been trained on a large image dataset and can be directly applied to feature extraction; after the last convolution layer, global average pooling is used to convert the feature map into a fixed-length feature vector, which is further used for multimodal fusion; Text data feature extraction: Use the pre-trained language model Transformer to encode text data and extract the semantic feature vector of the text; through pre-training and fine-tuning, the model can understand the semantics and structure of the text; then, each word or subword after segmentation is converted into a vector of fixed length, and the output feature vector can represent the contextual relationship and semantic information in the text; Sensor data feature extraction: Use recurrent neural network (RNN) to extract time series features from sensor data; for time series data, recurrent neural network (RNN) can capture time dependencies and is suitable for feature extraction of time series data.

[0008] Furthermore, the data augmentation pipeline generates new samples using existing multimodal data by generating models through generative adversarial networks (GANs): To train the GAN model, existing multimodal data is used as input. The generator generates new samples, while the discriminator evaluates the authenticity of the generated samples. Through continuous iterative training, the generator can learn the distribution of real data and generate new samples similar to real samples. For example, new images, texts, or time series can be generated to expand the data set. The generated samples will be input into the model as additional training data to improve the generalization ability of the model.

[0009] Furthermore, in order to optimize the generation effect of GANs, the parameters of the generation model are updated regularly, and retraining is performed using newly collected multimodal data to improve the diversity and quality of generated samples; through a continuous feedback mechanism, the effects of feature extraction and data enhancement are continuously optimized.

[0010] Furthermore, the multimodal deep learning pipeline includes a feature-level fusion strategy and a decision-level fusion strategy; The feature-level fusion strategy is to first extract features from the data of each modality, align the features of different modalities in a certain space or dimension, and then input them into a joint model; the advantage of feature-level fusion is that it can retain the detailed information of each modality, and in deep learning, the correlation between modalities can be extracted by sharing the network structure; therefore, this method integrates feature-level fusion; The decision-level fusion strategy is to process the data of each modality separately and then make a comprehensive decision on the respective output results; therefore, this method integrates the decision-level fusion strategy.

[0011] Furthermore, the feature-level fusion strategy includes: Feature alignment: Create a shared latent space based on a fully connected neural network so that features of different modalities can be aligned in the same space; optimize the alignment effect through a loss function (minimizing the reconstruction error) to ensure that the correlation between modalities is maximized; and, during the alignment process, introduce regularization technology (L2 regularization) to prevent overfitting; Deep fusion model: Design a multimodal joint model, multimodal Transformer, and embed the deep fusion model into the AI ​​training platform as a built-in model, while supporting user-defined models; use the aligned features as input for further processing; the specific network architecture of the multimodal joint model is as follows: The multimodal Transformer input layer receives the aligned feature vectors, performs nonlinear transformations through multiple fully connected layers, and finally outputs the prediction results. In order to retain the original feature information, jump connections are introduced between different layers to improve the learning ability of the model. The self-attention layer enables the model to focus on the important parts of the features in different modalities, so that the model can adaptively adjust the weights of the features of each modality. The decision-level fusion strategy includes: Independent model training: Design an independent model for each modality: use CNN for images, BERT for text, and LSTM for other data; embed the trained model into the AI ​​training platform as a built-in base model, and users can choose to customize the base model; each model is independently trained on its own modality data to capture modality-specific features; after training, each independent model predicts the input data and generates corresponding output results (such as classification probability); Weighted voting: Set the weight of each modality model and combine the outputs of the models; the weight is adjusted based on the performance of the model on the validation set, for example, models with higher accuracy have a larger proportion in the final decision.

[0012] The multimodal data processing method of feature-level fusion strategy and decision-level fusion strategy not only helps to improve the performance and generalization ability of the model in theory, but also can reduce costs, improve efficiency and enhance the degree of automation of multimodal task training for users on AI platforms in practical applications.

[0013] The present invention also claims protection for a multimodal data enhancement system based on an AI training platform, comprising: Multimodal data fusion pipeline, which is used to process data sources of different modalities and synchronize, align and fuse the data according to the actual application scenario; The data augmentation pipeline is used to generate models through generative adversarial networks (GANs), which automatically generate new samples for users' learning tasks using existing multimodal data to improve the generalization and robustness of model training: Multimodal deep learning pipeline, including feature-level fusion strategy and decision-level fusion strategy, is used to fully learn and understand the deep features of different modal data, and assist users to conveniently train multimodal learning models; The system specifically implements multimodal data enhancement through the above method.

[0014] The present invention also claims protection for a multimodal data enhancement device based on an AI training platform, characterized in that it comprises: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the above method.

[0015] The present invention also claims protection for a computer-readable medium, characterized in that computer instructions are stored on the computer-readable medium, and when the computer instructions are executed by a processor, the above method is implemented.

[0016] Compared with the prior art, the multimodal data enhancement method and system based on the AI ​​training platform of the present invention has the following beneficial effects: 1. Improve the automation level of AI training platform: The data fusion channel can automatically integrate the features of different modalities, automatically and comprehensively capture the information in the data, reduce the bias caused by a single modality, improve the ability to adapt to new data, and further improve the automation level of multimodal training tasks.

[0017] 2. Improve the multimodal data training optimization of the AI ​​training platform: In the decision fusion strategy, independent models are trained for specific modalities, which can give full play to their respective advantages. After comprehensive decision-making, the accuracy of the final prediction can be improved, and users can be assisted in better model training and tuning.

[0018] 3. Reduce data labeling costs: Through intelligent feature extraction and fusion, the need for manual labeling and data screening is reduced, and the time cost in the data preparation stage is reduced.

[0019] 4. Improve data processing efficiency: The integrated strategy can automatically complete multiple tasks during the data processing process, saving time in multimodal data processing in the case of large-scale data sets and improving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a diagram of a multimodal data fusion pipeline module provided by an embodiment of the present invention; Figure 2 It is a diagram showing the structure of a multimodal deep learning pipeline provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with specific embodiments.

[0022] An embodiment of the present invention provides a multimodal data enhancement method based on an AI training platform, and the implementation of the method includes: First, a multimodal data fusion pipeline is developed on the AI ​​training platform. Data of different modalities (such as images, text, sensor data, etc.) are collected through the data acquisition module and preprocessed. In the preprocessing stage, the multimodal data fusion pipeline is used to extract features from data from different modalities and perform joint modeling at a specific level. Then, a data augmentation pipeline is used to automatically generate high-quality synthetic data; Finally, a multimodal deep learning pipeline is developed to fully learn and understand the deep features of data from different modalities.

[0023] This method comprehensively improves the multimodal data processing capabilities of the AI ​​training platform in many aspects. In terms of multimodal data training optimization of the AI ​​training platform, this method effectively compensates for the limitations of single modal data through intelligent processing technology, and improves the generalization ability of the model and data processing efficiency. In terms of reducing data standard costs, this method reduces manual input while improving data processing efficiency by realizing efficient feature extraction and data enhancement. In terms of improving data processing efficiency, this method deeply mines multimodal data feature information through a hybrid strategy, providing strong support for the application of AI training platforms in multimodal environments. Combined with the above characteristics, this method can automatically and comprehensively capture information in the data, improve data processing capabilities, and greatly improve the degree of automation of multimodal training tasks. For users, multimodal task training on the AI ​​platform will reduce interaction costs, and the efficiency of training and optimizing multimodal learning task models will be greatly improved.

[0024] The specific implementation of this method is as follows: 1. Develop a multimodal data fusion pipeline. This pipeline is used to process data sources of different modalities and synchronize, align and fuse the data according to the actual application scenario. It includes the following three steps: (1) Data standardization: Standardize multimodal data (images, text, sensor data, and voice) collected from different data sources.

[0025] For image data: scale and normalize, supporting two normalization methods: the first is to adjust the image pixel value from the range of 0-255 to 0-1; the second is to normalize according to a specific mean and standard deviation, using the standardized values ​​of the ImageNet dataset: mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225].

[0026] For text data: perform word segmentation and encoding, use natural language processing tool libraries (SpaCy, NLTK) for word segmentation, and use models such as BERT and Word2Vec to convert the segmented text into word embedding vectors.

[0027] For time series data: perform interpolation and alignment to ensure that all data can be analyzed on the same time axis or in the same dimension.

[0028] (2) Time series synchronization and alignment: Synchronize and align time series data (sensor data) and non-time series data (images, text) to ensure that multimodal data is effectively integrated within the same time period. The specific approach is to slice the entire time series into multiple time periods, each of which contains data from multiple modalities, and interpolate to fill in missing time points.

[0029] The data acquisition module collects data of different modalities from multiple data sources to ensure that each time period of the slice can contain multi-modal data, which will be explained in detail below: Time series data: For time series data with clear timestamps (such as sensor data and audio data), the data will be timestamp aligned to ensure that the data can be organized in chronological order.

[0030] Text data: First of all, the sources of text data can be basically divided into the following categories. First, text data comes from real-time collection (such as conversation records), so a timestamp can be assigned to each group of conversations during collection. Second, text data is some log records, so the log will contain the time when the text was generated; third, text data comes from crawling online news or comments using crawler software, so the timestamp will also be recorded when crawling. Therefore, the text data will record the timestamp when it is collected to ensure that it can be organized according to time.

[0031] Image data: Image data is usually collected by a camera or other imaging device and has a clear timestamp. The data can be organized in chronological order based on the timestamp.

[0032] Therefore, the timestamp information of multimodal data can be obtained during the data acquisition process. As long as appropriate experimental data are selected, it can be ensured that each time period of the slice contains multimodal data.

[0033] For missing time points, in the time series, some timestamp data points are missing, and the timestamp interval check method is used to find them. During data preprocessing, the interval between each timestamp and the previous timestamp is checked. If the time interval between a timestamp and the previous timestamp does not meet the predetermined sampling interval, it means that the data at that time point is missing.

[0034] (3) Data feature extraction: Multimodal feature extraction models include convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformer.

[0035] Image data feature extraction: Use a pre-trained convolutional neural network (CNN) ResNet model to extract high-level features from images. The pre-trained model has been trained on a large image dataset and can be directly applied to feature extraction. After the last convolution layer, global average pooling is used to convert the feature map into a fixed-length feature vector, which is further used for multimodal fusion.

[0036] In image processing tasks, high-level features refer to more abstract features with a higher semantic level extracted from the original image data. These features can represent complex information of the image, such as objects, scenes, textures, and other semantic content, and can help machine learning models better understand the image content. The following will explain in detail the extraction of high-level features by the ResNet network used in this method: In deep learning, image data is processed into a multidimensional array with dimensions (batch size, height, width, number of channels). First, the image data is standardized, and the data dimension remains unchanged during this process. Then, the ResNet network is used to obtain a high-dimensional vector through a series of operations such as convolution and pooling. This necklace is the high-level feature of the image, and its dimension is (batch_size, 2048), which has 2048 dimensions.

[0037] Text data feature extraction: Use the pre-trained language model Transformer to encode text data and extract the semantic feature vector of the text. Through pre-training and fine-tuning, the model can understand the semantics and structure of the text. Then, each word or subword after segmentation is converted into a vector of fixed length, and the output feature vector can represent the contextual relationship and semantic information in the text.

[0038] Sensor data feature extraction: Use recurrent neural network (RNN) to extract time series features from sensor data. For time series data, recurrent neural network (RNN) can capture time dependencies and is suitable for feature extraction of time series data.

[0039] 2. Develop a multimodal data augmentation pipeline that generates new samples using existing multimodal data by using a generative adversarial network (GANs) model.

[0040] The GAN model is trained using existing multimodal data as input. The generator generates new samples, while the discriminator evaluates the authenticity of the generated samples. Through continuous iterative training, the generator is able to learn the distribution of real data and generate new samples that are similar to real samples. For example, new images, texts, or time series can be generated to expand the dataset. The generated samples will be input into the model as additional training data to improve the generalization ability of the model.

[0041] Generative adversarial networks are mainly composed of two opposing parts: the generator and the discriminator. They compete with each other through adversarial training. The following is a detailed introduction: The task of the generator is to generate very realistic data. The generator takes some random noise (usually a low-dimensional vector, such as random values ​​drawn from a normal distribution) as input, processes it through a neural network, and outputs fake data.

[0042] The task of the discriminator is to determine whether the input data is "real" or "generated". It is a binary classifier that outputs a value to indicate the authenticity of the input data. Usually, it outputs a probability, which represents the probability that the data is real data. The discriminator and the generator are trained alternately. The equilibrium point of the GAN network is reached, at which point the data generated by the generator is realistic enough.

[0043] The discriminator evaluates the authenticity of the generated samples as follows: During the training process of GAN, the discriminator uses the binary cross entropy loss function to measure the gap between the predicted result and the true label. The goal of the loss function is to maximize the probability that the real data is the "true" category and maximize the probability that the generated data is the "false" category. The goal of optimizing the discriminator is to minimize the above loss function. Through the back-propagation algorithm, the parameters (weights and biases) of the discriminator are optimized so that the discriminator can accurately distinguish between real data and generated data.

[0044] In order to optimize the generation effect of GANs, the parameters of the generation model are updated regularly, and the newly collected multimodal data is used for retraining to improve the diversity and quality of the generated samples. Through a continuous feedback mechanism, the effect of feature extraction and data enhancement is continuously optimized.

[0045] 3. Develop a multimodal deep learning pipeline. The multimodal deep learning pipeline includes a feature-level fusion strategy and a decision-level fusion strategy.

[0046] The feature-level fusion strategy is to first extract features from the data of each modality, align the features of different modalities in a certain space or dimension, and then input them into a joint model. The advantage of feature-level fusion is that it can retain detailed information of each modality, and in deep learning, the correlation between modalities can be extracted by sharing the network structure. Therefore, this method integrates feature-level fusion. Feature-level fusion strategies include: Feature alignment: A shared latent space is created based on a fully connected neural network, so that features of different modalities can be aligned in the same space. The alignment effect is optimized through a loss function (minimizing the reconstruction error) to ensure that the correlation between modalities is maximized. In addition, during the alignment process, regularization technology (L2 regularization) is introduced to prevent overfitting.

[0047] Deep fusion model: Design a multimodal joint model, multimodal Transformer, and embed the deep fusion model into the AI ​​training platform as a built-in model, while supporting user-defined models. Use the aligned features as input for further processing. The specific network architecture of the multimodal joint model is as follows: The multimodal Transformer input layer receives the aligned feature vectors, performs nonlinear transformations through multiple fully connected layers, and finally outputs the prediction results. In order to retain the original feature information, jump connections are introduced between different layers to improve the learning ability of the model. The self-attention layer can help the model focus on the important parts of the features in different modalities, so that the model can adaptively adjust the weights of the features of each modality.

[0048] The decision-level fusion strategy is to process the data of each modality separately and then make a comprehensive decision on the output results of each modality. Therefore, this method integrates the decision-level fusion strategy. The decision-level fusion strategy includes: Independent model training: Design an independent model for each modality. Use CNN for images, BERT for text, and LSTM for other data. The trained model is embedded in the AI ​​training platform as a built-in base model, and users can choose to customize the base model. Each model is trained independently on its own modality data to capture modality-specific features. After training, each independent model predicts the input data and generates corresponding output results (such as classification probability).

[0049] Weighted voting: Set the weight of each modality model and combine the outputs of the models. The weight is adjusted based on the performance of the model on the validation set, for example, the model with higher accuracy will account for a larger proportion in the final decision.

[0050] The multimodal data processing method of feature-level fusion strategy and decision-level fusion strategy not only helps to improve the performance and generalization ability of the model in theory, but also can reduce costs, improve efficiency and enhance user experience in practical applications. In general, this method has broad application prospects and potential, and is of great significance to promoting the development and application of multimodal data processing technology.

[0051] The joint modeling at a specific level mentioned in this method refers to the fusion of data of different modalities (images, texts, time series data) at a specific feature level during multimodal data processing. The feature level mentioned in this method refers to the features processed by multimodal feature extraction. These features are no longer direct pixels or word vectors, but data reflecting their semantic information.

[0052] The joint model refers to a model that combines data from multiple modalities (such as images, text, and time series) for training. The joint model mentioned in this method refers to the multimodal Transformer model. The multimodal model is built on the Transformer infrastructure, and its network structure is as follows: First, there is the multimodal feature input layer, which unifies the features of different modalities to the same dimension through a fully connected layer. For image features, the 2048-dimensional feature vector is converted to 768 dimensions; for text features, the text is processed by BERT to obtain 768-dimensional features; for time series features, the 256-dimensional features obtained by RNN processing are converted to 768 dimensions (the common and effective dimension setting size of the Bert model). Then there is the 12-layer ENCODER structure of the Transformer model. Each ENCODER layer includes a self-attention mechanism, a feedforward neural network, a residual connection, and layer normalization. Next, a fusion layer is set to combine the features of different modal data encoded by the Encoder with the self-attention mechanism to obtain a joint multimodal representation, so as to make full use of the complementary information between different modalities and highlight more relevant and important information. Next, based on the fused multimodal representation, the 12-layer Decoder layer generates the output required for the task. The structure of the Decoder layer is similar to that of the Encoder, and is also based on self-attention and feedforward neural networks. In addition, this method introduces jump connections (such as residual connections) between different layers of the Encoder, so that the gradient can be directly transferred to the bottom layer, alleviating the optimization problem of the deep network. At the same time, the jump connection can also fuse features at different levels to obtain a richer representation. Finally, there are 3 fully connected output layers with 128 nodes, 512 nodes, 256 nodes, and 3 nodes (taking the classification task as an example). The activation function of the network layer uses softmax.

[0053] The embodiment of the present invention further provides a multimodal data enhancement system based on an AI training platform, comprising: Multimodal data fusion pipeline, which is used to process data sources of different modalities and synchronize, align and fuse the data according to the actual application scenario; Data augmentation pipeline for generating models using generative adversarial networks (GANs) to generate new samples from existing multimodal data: Multimodal deep learning pipeline, including feature-level fusion strategy and decision-level fusion strategy, to fully learn and understand the deep features of different modal data; The system specifically implements multimodal data enhancement through the multimodal data enhancement method based on the AI ​​training platform described in the above embodiments.

[0054] First, a multimodal data fusion pipeline is developed on the AI ​​training platform. Data of different modalities (such as images, text, sensor data, etc.) are collected through the data acquisition module and preprocessed. In the preprocessing stage, the multimodal data fusion pipeline is used to extract features from data from different modalities and perform joint modeling at a specific level. Then, a data augmentation pipeline is used to automatically generate high-quality synthetic data; Finally, a multimodal deep learning pipeline is developed to fully learn and understand the deep features of data from different modalities.

[0055] The multimodal data fusion pipeline includes the following three steps: (1) Data standardization: Standardize multimodal data (images, text, sensor data, and voice) collected from different data sources.

[0056] For image data: scale and normalize, supporting two normalization methods: the first is to adjust the image pixel value from the range of 0-255 to 0-1; the second is to normalize according to a specific mean and standard deviation, using the standardized values ​​of the ImageNet dataset: mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225].

[0057] For text data: perform word segmentation and encoding, use natural language processing tool libraries (SpaCy, NLTK) for word segmentation, and use models such as BERT and Word2Vec to convert the segmented text into word embedding vectors.

[0058] For time series data: perform interpolation and alignment to ensure that all data can be analyzed on the same time axis or in the same dimension.

[0059] (2) Time series synchronization and alignment: Synchronize and align time series data (sensor data) and non-time series data (images, text) to ensure that multimodal data is effectively integrated within the same time period. The specific approach is to slice the entire time series into multiple time periods, each of which contains data from multiple modalities, and interpolate to fill in missing time points.

[0060] (3) Data feature extraction: Multimodal feature extraction models include convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformer.

[0061] Image data feature extraction: Use a pre-trained convolutional neural network (CNN) ResNet model to extract high-level features from images. The pre-trained model has been trained on a large image dataset and can be directly applied to feature extraction. After the last convolution layer, global average pooling is used to convert the feature map into a fixed-length feature vector, which is further used for multimodal fusion.

[0062] Text data feature extraction: Use the pre-trained language model Transformer to encode text data and extract the semantic feature vector of the text. Through pre-training and fine-tuning, the model can understand the semantics and structure of the text. Then, each word or subword after segmentation is converted into a vector of fixed length, and the output feature vector can represent the contextual relationship and semantic information in the text.

[0063] Sensor data feature extraction: Use recurrent neural network (RNN) to extract time series features from sensor data. For time series data, recurrent neural network (RNN) can capture time dependencies and is suitable for feature extraction of time series data.

[0064] The multimodal data augmentation pipeline trains the GAN model and uses existing multimodal data as input. The generator generates new samples, while the discriminator evaluates the authenticity of the generated samples. Through continuous iterative training, the generator can learn the distribution of real data and generate new samples similar to real samples. For example, new images, texts, or time series can be generated to expand the dataset. The generated samples will be input into the model as additional training data to improve the generalization ability of the model.

[0065] In order to optimize the generation effect of GANs, the parameters of the generation model are updated regularly, and the newly collected multimodal data is used for retraining to improve the diversity and quality of the generated samples. Through a continuous feedback mechanism, the effect of feature extraction and data enhancement is continuously optimized.

[0066] The multimodal deep learning pipeline includes a feature-level fusion strategy and a decision-level fusion strategy: The feature-level fusion strategy is to first extract features from the data of each modality, align the features of different modalities in a certain space or dimension, and then input them into a joint model. The advantage of feature-level fusion is that it can retain detailed information of each modality, and in deep learning, the correlation between modalities can be extracted by sharing the network structure. Therefore, this method integrates feature-level fusion. Feature-level fusion strategies include: Feature alignment: A shared latent space is created based on a fully connected neural network, so that features of different modalities can be aligned in the same space. The alignment effect is optimized through a loss function (minimizing the reconstruction error) to ensure that the correlation between modalities is maximized. In addition, during the alignment process, regularization technology (L2 regularization) is introduced to prevent overfitting.

[0067] Deep fusion model: Design a multimodal joint model, multimodal Transformer, and embed the deep fusion model into the AI ​​training platform as a built-in model, while supporting user-defined models. Use the aligned features as input for further processing. The specific network architecture of the multimodal joint model is as follows: The multimodal Transformer input layer receives the aligned feature vectors, performs nonlinear transformations through multiple fully connected layers, and finally outputs the prediction results. In order to retain the original feature information, jump connections are introduced between different layers to improve the learning ability of the model. The self-attention layer can help the model focus on the important parts of the features in different modalities, so that the model can adaptively adjust the weights of the features of each modality.

[0068] The decision-level fusion strategy is to process the data of each modality separately and then make a comprehensive decision on the output results of each modality. Therefore, this method integrates the decision-level fusion strategy. The decision-level fusion strategy includes: Independent model training: Design an independent model for each modality. Use CNN for images, BERT for text, and LSTM for other data. The trained model is embedded in the AI ​​training platform as a built-in base model, and users can choose to customize the base model. Each model is trained independently on its own modality data to capture modality-specific features. After training, each independent model predicts the input data and generates corresponding output results (such as classification probability).

[0069] Weighted voting: Set the weight of each modality model and combine the outputs of the models. The weight is adjusted based on the performance of the model on the validation set, for example, the model with higher accuracy will account for a larger proportion in the final decision.

[0070] An embodiment of the present invention further provides a multimodal data enhancement device based on an AI training platform, characterized in that it includes: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the multimodal data enhancement method based on the AI ​​training platform described in the above embodiment.

[0071] The embodiment of the present invention also provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the processor executes the multimodal data enhancement method based on the AI ​​training platform described in the above embodiment. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0072] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0073] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0074] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0075] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0076] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A multimodal data enhancement method based on an AI training platform, characterized in that: The implementation of this method includes: First, a multimodal data fusion pipeline is developed on the AI ​​training platform. Data from different modalities are collected and preprocessed through the data acquisition module. In the preprocessing stage, the multimodal data fusion pipeline is used to extract features from data from different modalities and perform joint modeling at a specific level. Then, a data augmentation pipeline is used to automatically generate high-quality synthetic data; Finally, a multimodal deep learning pipeline is developed to fully learn and understand the deep features of data from different modalities.

2. According to claim 1, a multimodal data enhancement method based on an AI training platform is characterized in that: The multimodal data fusion pipeline is used to process data sources of different modalities and synchronize, align and fuse the data according to actual application scenarios; The multimodal data fusion pipeline consists of the following steps: (1) Data standardization: standardize multimodal data collected from different data sources; (2) Time series synchronization and alignment: Synchronize and align time series data with non-time series data to ensure that multimodal data are effectively integrated within the same time period; (3) Data feature extraction: Multimodal feature extraction models include convolutional neural networks, recurrent neural networks, and Transformer.

3. According to claim 2, a multimodal data enhancement method based on an AI training platform is characterized in that: Standardize multimodal data collected from different data sources, including: For image data: scale and normalize, supporting two normalization methods: the first is to adjust the image pixel value from the range of 0-255 to 0-1; the second is to normalize according to a specific mean and standard deviation, using the standardized values ​​of the ImageNet dataset: mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225]; For text data: perform word segmentation and encoding, use the natural language processing tool library to perform word segmentation, and use the BERT and Word2Vec models to convert the segmented text into word embedding vectors; For time series data: interpolation and alignment are performed to ensure that all data can be analyzed on the same time axis or in the same dimension; The timing synchronization and alignment are specifically implemented as follows: Slice the entire time series into multiple time periods, each of which contains data from multiple modes, and can fill in missing time points through interpolation; The data feature extraction specifically includes: Image data feature extraction: Use the pre-trained convolutional neural network ResNet model to extract high-level features from the image; after the convolution layer, use global average pooling to convert the feature map into a fixed-length feature vector, which is further used for multimodal fusion; Text data feature extraction: Use the pre-trained language model Transformer to encode the text data and extract the semantic feature vector of the text; then, convert each word or subword after segmentation into a vector of fixed length. The output feature vector can represent the contextual relationship and semantic information in the text; Sensor Data Feature Extraction: Extract time series features from sensor data using recurrent neural networks.

4. According to claim 1, a multimodal data enhancement method based on an AI training platform is characterized in that: The data augmentation pipeline generates new samples using existing multimodal data by generating a model through a generative adversarial network: Training the GAN model uses existing multimodal data as input; the generator generates new samples, while the discriminator evaluates the authenticity of the generated samples; through continuous iterative training, the generator can learn the distribution of real data and generate new samples similar to real samples; The generated samples will be fed into the model as additional training data.

5. According to claim 4, a multimodal data enhancement method based on an AI training platform is characterized in that: Regularly update the parameters of the generative model and use newly collected multimodal data for retraining to improve the diversity and quality of generated samples; through a continuous feedback mechanism, continuously optimize the effects of feature extraction and data enhancement.

6. The multimodal data enhancement method based on an AI training platform according to claim 1, characterized in that: The multimodal deep learning pipeline includes a feature-level fusion strategy and a decision-level fusion strategy; The feature-level fusion strategy is to first extract features from the data of each modality, align the features of different modalities in a specified space or dimension, and then input them into a joint model; The decision-level fusion strategy is to process the data of each modality separately and then make a comprehensive decision based on the respective output results.

7. The multimodal data enhancement method based on an AI training platform according to claim 6, characterized in that: The feature-level fusion strategy includes: Feature alignment: Create a shared latent space based on a fully connected neural network so that features of different modalities can be aligned in the same space; optimize the alignment effect through a loss function to ensure that the correlation between modalities is maximized; and introduce regularization technology during the alignment process to prevent overfitting; Deep fusion model: Design a multimodal joint model, multimodal Transformer, and embed the deep fusion model into the AI ​​training platform as a built-in model, while supporting user-defined models; use the aligned features as input for further processing; the specific network architecture of the multimodal joint model is as follows: The multimodal Transformer input layer receives the aligned feature vectors, performs nonlinear transformations through multiple fully connected layers, and finally outputs the prediction results. Skip connections are introduced between different layers. The self-attention layer enables the model to focus on the important parts of the features in different modalities, so that the model can adaptively adjust the weights of the features of each modality. The decision-level fusion strategy includes: Independent model training: Design an independent model for each modality: use CNN for images, BERT for text, and LSTM for other data; embed the trained model into the AI ​​training platform as a built-in base model, and users can choose to customize the base model; each model is independently trained on its own modality data to capture modality-specific features; after training, each independent model predicts the input data and generates the corresponding output results; Weighted voting: Set the weights of each modality model and combine the outputs of the models; the weights are adjusted based on the performance of the models on the validation set.

8. A multimodal data enhancement system based on an AI training platform, characterized in that: include: Multimodal data fusion pipeline, which is used to process data sources of different modalities and synchronize, align and fuse the data according to the actual application scenario; Data augmentation pipeline for generating new samples using existing multimodal data through generative adversarial network generation models: Multimodal deep learning pipeline, including feature-level fusion strategy and decision-level fusion strategy, to fully learn and understand the deep features of different modal data; The system specifically implements multimodal data enhancement through the method described in any one of claims 1 to 7.

9. A multimodal data enhancement device based on an AI training platform, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images