A model pre-training method based on dynamic particle data enhancement

Abstract

The invention relates to the field of data processing, and discloses a model pre-training method based on dynamic particle data enhancement, which is used to solve the problem that when processing multimodal data, it needs to be divided into multiple pre-training tasks. The method comprises the following steps: collecting multimodal data, dividing the collected multimodal data into modules, rasterizing the divided image data, establishing a particle set and a particle filtering operation, dividing the particles after the particle filtering operation into a training set and a validation set, designing a multimodal model and a loss function based on a joint probability distribution, integrating the designed loss function into the training process of the model, using a dynamic particle data enhancement method to generate diverse training samples, using the loss function of the joint probability distribution to train the model, using a cross-validation method to evaluate the model, analyzing the performance of the model on different modal data, and effectively improving the ability of the model to adapt to different sizes and various deformations of objects.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and more specifically to a model pre-training method based on dynamic particle data enhancement. Background Art

[0002] The model based on dynamic particle data enhancement is a model used to train multimodal data. It introduces the concept of dynamic particles and uses technologies such as particle filtering to enhance training data and improve the generalization ability and robustness of the model.

[0003] Existing dynamic particle data augmentation models usually use multiple pre-training tasks, but this training method will result in an inability to quickly process the data when the training data is large, reducing the efficiency of model training, and cannot overcome the problems of inconsistent object sizes and scale deformation.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a model pre-training method based on dynamic particle data enhancement, which solves the problems raised in the above-mentioned background technology.

[0006] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0007] A model pre-training method based on dynamic particle data enhancement includes the following steps:

[0008] Collect multimodal data, and divide the collected multimodal data into modules, namely, video modal data, voice modal data, and text modal data;

[0009] The divided image data is rasterized, and the raster size can be determined according to the size of the input data;

[0010] The rasterized image is used to establish a particle set in a relative position manner, and a particle filter operation is performed, and the particles after the particle filter operation are divided into a training set and a validation set;

[0011] Design a multimodal model that can process data from various modalities simultaneously, determine the input and output layers of the model, and the connection between the data from each modality;

[0012] Design a loss function based on joint probability distribution to assist the model in learning the correlation mapping relationship between multimodal features, so that it has the ability of self-reasoning and can recognize unlearned knowledge;

[0013] Integrate the designed loss function into the model training process, use the dynamic particle data augmentation method to generate diverse training samples to improve the generalization ability of the model, and use the joint probability distribution loss function to train the model to learn the association mapping relationship between multimodal features;

[0014] After training is completed, the model is evaluated using the cross-validation method to analyze the model's performance on data of different modalities, as well as the model's learning effect on the correlation mapping relationship between multimodal features, and the model is adjusted and optimized as needed.

[0015] Preferably, the step of establishing a particle set by using a relative position method for the rasterized image is:

[0016] Each pixel in the image is regarded as a particle, and the position of the particle is represented by two-dimensional coordinates. The coordinates of each pixel are used as the position information of the particle, and the pixel value is used as the state information of the particle.

[0017] Select a certain pixel in the image as a reference point and express the positions of other pixels as relative to the reference point;

[0018] All pixels are converted into particles, and their relative positions and pixel values ​​are used as the state information of the particle set, thus obtaining a particle set representing the spatial structure of the image.

[0019] Preferably, the particle filtering operation steps are:

[0020] Perform CNN to extract image edges, select the extracted edge points as particle sets, and set corresponding initial values ​​for the target representation model and the particle motion model;

[0021] The iterative process of estimating particles is done using the maximum likelihood estimation method;

[0022] Determine the weight of each particle according to the Bhattacharyya coefficient;

[0023] The parameters of the particle motion model are adaptively adjusted according to the observed values ​​of the target characterization state to obtain an estimated value of the target feature characterization.

[0024] Preferably, the steps of performing CNN on the image to extract the image edge are:

[0025] Select the CNN model of the LeNet image edge detection task and build a neural network structure for edge detection;

[0026] Use the training set to train the built CNN model and adjust the model parameters through the back propagation algorithm and optimizer;

[0027] Evaluate the trained model using the test set to check the performance of the model on unseen data;

[0028] Use the trained model to extract the edge of the image to be processed, and perform filtering on the extracted edge image.

[0029] Preferably, the step of determining the weight of each particle according to the Bhattacharyya coefficient is:

[0030] For each particle, calculate its corresponding probability distribution according to the characteristics or states it represents;

[0031] Calculate the Bhattacharyya coefficient between the probability distribution represented by each particle and the target distribution. The calculation formula is: Among them, BC i is the Bhattacharyya coefficient of particle i, P(x) is the target distribution, Q i (x) represents the probability distribution of particle i;

[0032] According to the calculated Bhattacharyya coefficient, the weight of each particle is obtained by calculating the inverse of the Bhattacharyya coefficient. The calculation formula is: where ω i Expressed as the weight of particle i, BC i It is expressed as the Bhattacharyya coefficient of particle i;

[0033] The weights are normalized by dividing each weight by the sum of all weights, and the normalized weight is used as the weight of the particle, which is given by

[0034] Preferably, the step of designing a loss function based on joint probability distribution is:

[0035] Establish the relationship between the video latitude and the prediction result, the relationship is: Where L v Expressed as video loss;

[0036] Establish the relationship between the voice latitude and the prediction result, the relationship is: Where L a It is expressed as speech loss;

[0037] Establish the relationship between text latitude and prediction results, the relationship is Where L t Expressed as text loss;

[0038] The final loss is obtained by averaging the video loss, speech loss, and text loss, and its calculation formula is: L r Expressed as final loss.

[0039] Preferably, the step of evaluating the model using a validation set or cross-validation method is:

[0040] Use the trained model to predict the samples in the validation set and obtain the prediction results of the model on the validation set;

[0041] The evaluation indicators of the model are obtained through the prediction results, and the evaluation indicators include the accuracy, precision, recall rate and F1 value of the model;

[0042] According to the obtained model evaluation indicators, the value of the selected evaluation indicator is calculated, and its calculation expression is ZQ=f(Z1,Z2,Z3,Z4), where ZQ represents the accuracy index of the model, Z1 represents the accuracy rate of the model, Z2 represents the precision rate of the model, Z3 represents the recall rate of the model, and Z4 represents the F1 value of the model.

[0043] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all the advantages described below at the same time:

[0044] Collect multimodal data, divide the collected multimodal data into modules, rasterize the divided image data, establish particle sets and particle filtering operations, divide the particles after the particle filtering operation into training sets and validation sets, design a multimodal model and a loss function based on joint probability distribution, integrate the designed loss function into the training process of the model, use the dynamic particle data enhancement method to generate diverse training samples, and use the loss function of joint probability distribution to train the model, use the cross-validation method to evaluate the model, and analyze the performance of the model on different modal data, which effectively improves the model's ability to adapt to objects of different sizes and various deformations.

[0045] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described below are only some embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0047] In the figure:

[0048] Figure 1 It is the overall flow chart of the present invention.

[0049] Figure 2 This is a flow chart of dynamic particle visual data enhancement of the present invention.

[0050] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0051] The present invention will now be described in further detail with reference to the accompanying drawings.

[0052] See also Figure 1 As shown, in this embodiment, a model pre-training method based on dynamic particle data enhancement is provided, comprising the following steps:

[0053] Collect multimodal data, divide the collected multimodal data into modules, and divide them into video modal data, voice modal data and text modal data. The multimodal data refers to a collection of data of different types. These data may come from different sensors, sources or representation methods, and have different characteristics and structures. Common multimodal data include text data, image data, video data and voice data;

[0054] The divided multimodal data is rasterized, and the grid size can be determined according to the size of the input data. The rasterization process is the process of converting continuous image data into discrete grid or grid representation. This process divides the image into small squares, each of which is called a grid or pixel, and the size of the grid can be determined according to the needs. Rasterization converts the image into a two-dimensional matrix, in which each element represents the pixel value of the corresponding position in the image. This representation method makes it easier to process and analyze image data;

[0055] Take images as an example, Figure 2 As shown, the rasterized image is used to establish a particle set in a 2-D relative position manner, and a particle filter operation is performed as a training sample, and the 2-D representation is a two-dimensional coordinate;

[0056] Design a multimodal model that can process data of various modalities at the same time, determine the input layer and output layer of the model, and the connection method between the data of each modality. The multimodal model is a deep learning model that can process multiple types of data. It can obtain information from different data sources and use this information to complete various tasks. Multimodal models are usually composed of multiple sub-models, each of which specializes in processing one type of data, and then integrates the features of different types of data for task execution;

[0057] Design a loss function based on joint probability distribution to assist the model in learning the association mapping relationship between multimodal features, so that it has the ability of self-reasoning and can recognize unlearned knowledge. The association mapping relationship between multimodal features refers to the mutual relationship or correlation between features of different modal data. In a multimodal dataset, different data modalities may describe different aspects of the same entity or the same event, so there is often a certain association between them;

[0058] Integrate the designed loss function into the model training process, use the dynamic particle data augmentation method to generate diverse training samples to improve the generalization ability of the model, and use the loss function of the joint probability distribution to train the model to learn the correlation mapping relationship between multimodal features;

[0059] After training is completed, the model is evaluated using validation sets or cross-validation methods to analyze the model's performance on data of different modalities, as well as the model's learning effect on the correlation mapping relationship between multimodal features. The model's training and evaluation results are analyzed, and the model is adjusted and optimized as needed.

[0060] In this embodiment, it should be specifically explained that the step of establishing a particle set by using a relative position method for the rasterized image is as follows:

[0061] Treat each pixel in the image as a particle, and use their position information and pixel values ​​as the state of the particle. You can use two-dimensional coordinates to represent the position of the particle, use the coordinates of each pixel as the position information of the particle, and use the pixel value as the state information of the particle;

[0062] The position information of the particle is expressed as a relative position, that is, a position relative to a reference point. A pixel in the image can be selected as a reference point, and the positions of other pixels are expressed as positions relative to the reference point;

[0063] All pixels are converted into particles, and their relative positions and pixel values ​​are used as the state information of the particle set. In this way, a particle set representing the spatial structure of the image is obtained, where each particle represents a position in the image and contains the pixel value and relative position information of the position.

[0064] In this embodiment, it should be specifically explained that the steps of performing the particle filtering operation are:

[0065] Perform CNN to extract image edges, then select the extracted edge points as particle sets, and set corresponding initial values ​​for the target representation model and the particle motion model. CNN refers to a convolutional neural network. It is a deep learning model that is particularly suitable for processing data with a grid structure, such as images. The core idea of ​​CNN is to extract features from images through convolution and pooling operations, and learn high-level representations of features through a network consisting of multiple convolutional layers and fully connected layers.

[0066] The maximum likelihood estimation method (Estimate Maximum, EM) is used to estimate the iterative process of particles. Particles will propagate according to the initial motion model and perform the iterative process of E step and M step. The maximum likelihood estimation (Maximum Likelihood Estimation, MLE) is a commonly used parameter estimation method, which is used to estimate the parameters of probability distribution through observed data. Its basic idea is to find the parameter value that makes the observed data have the highest probability of appearing under a given model;

[0067] The weight of each particle is determined according to the Bhattacharyya coefficient, which is a statistic used to measure the similarity between two probability distributions. When determining the weight of a particle, the Bhattacharyya coefficient can be used to measure the similarity between the probability distribution represented by each particle and the target distribution, thereby assigning a corresponding weight to each particle;

[0068] The parameters of the particle motion model are adaptively adjusted according to the observed values ​​of the target characterization state to obtain an estimated value of the target feature characterization.

[0069] In this embodiment, it should be specifically explained that the steps of performing CNN on an image to extract image edges are as follows:

[0070] Select the CNN model of the LeNet image edge detection task and build a neural network structure for edge detection;

[0071] Use the training set to train the built CNN model, and adjust the model parameters through the back propagation algorithm and optimizer (such as SGD, Adam, etc.) so that the model can learn edge features from the training data;

[0072] Evaluate the trained model using the test set to check the performance of the model on unseen data;

[0073] Use the trained model to extract the edge of the image to be processed, and post-process the extracted edge image, such as applying thresholds, filtering and other operations, to enhance the effect of edge detection.

[0074] In this embodiment, it should be specifically explained that the iterative process steps of estimating particles using the maximum likelihood estimation method are:

[0075] Step 1: You need to define a probability model that describes the probability distribution of the parameters to be estimated and use the probability density function as the likelihood function;

[0076] Step 2: Construct a complete likelihood function based on the selected probability model and likelihood function;

[0077] Step 3: Solve analytically by taking the logarithm of the likelihood function, then taking the derivative and setting the derivative to zero to find the parameter value that maximizes the likelihood function.

[0078] Step 4: According to the selected optimization algorithm, iteratively update the parameter value so that the likelihood function gradually converges to the maximum value. In each iteration, the parameter value is updated according to the observed data, and the new likelihood function value is calculated;

[0079] Step 5: If the likelihood function has converged to the maximum value, the final parameter estimation result is output; otherwise, return to step 4 and continue iterative updating until the convergence condition is met.

[0080] In this embodiment, it should be specifically explained that the step of determining the weight of each particle according to the Bhattacharyya coefficient is:

[0081] For each particle, we first need to calculate its corresponding probability distribution based on the characteristics or states it represents;

[0082] Calculate the Bhattacharyya coefficient between the probability distribution represented by each particle and the target distribution. Assume that the target distribution is P(x) and the probability distribution represented by particle i is Q i (x), the Bhattacharyya coefficient is calculated as Among them, BC i It is expressed as the Bhattacharyya coefficient of particle i;

[0083] According to the calculated Bhattacharyya coefficient, the weight of each particle is obtained by calculating the inverse of the Bhattacharyya coefficient. The calculation formula is: where ω i Expressed as the weight of particle i, BC i It is expressed as the Bhattacharyya coefficient of particle i;

[0084] The weights are normalized by dividing each weight by the sum of all weights, and the normalized weight is used as the weight of the particle, which is given by

[0085] In this embodiment, it should be specifically explained that the steps of designing a loss function based on joint probability distribution are:

[0086] Project the text, image and video to the same dimension in the order of 3D-2D-1D, where v is the video feature, a is the voice feature, and t is the text feature. The 3D representation is the video, the 2D representation is the voice or image, and the 1D representation is the text;

[0087] Establish the relationship between the video latitude and the prediction result, the relationship is: Where L v Expressed as video loss;

[0088] Establish the relationship between the voice latitude and the prediction result, the relationship is: Where L a It is expressed as speech loss;

[0089] Establish the relationship between text latitude and prediction results, the relationship is Where L t Expressed as text loss;

[0090] The final loss is obtained by averaging the video loss, speech loss, and text loss, and its calculation formula is: L r Expressed as final loss.

[0091] In this embodiment, it should be specifically explained that the steps of generating diverse training samples using the method of dynamic particle data enhancement are as follows:

[0092] Randomly select some samples from the original training data as the initial particle set;

[0093] Iterate and update the particles. According to the current state information of the particles, a random walk strategy is used to update the state. According to the updated state information, the corresponding sample data is generated.

[0094] According to the updated state information of each particle, the corresponding sample data is generated using the data generation model. The sample data can be a deformation, expansion or synthesis of the original training data.

[0095] For the generated sample data, the corresponding label information is generated according to the label of the original training data; according to the similarity between the generated sample data and the original training data, the weight of each particle is calculated using the Bhattacharyya coefficient evaluation index, and the weight of each particle in the particle set is updated according to the calculated weight;

[0096] The particle set is resampled according to the particle weights, particles with higher weights are retained, and particles with lower weights are removed to ensure the diversity of sample data.

[0097] In this embodiment, it should be specifically explained that the steps of training the model using the loss function of the joint probability distribution are:

[0098] Define the joint probability distribution of the model output, and select the cross entropy loss function based on the defined joint probability distribution to measure the difference between the model output and the target value. The cross entropy loss function is a commonly used loss function for measuring the difference between two probability distributions, and is usually used in classification tasks. It measures the difference between the actual observed data and the model predicted data. The smaller the value, the more accurate the model prediction.

[0099] According to the selected loss function, the loss function of the joint probability distribution of the model is constructed, and its expression is loss = -∑ i P data (x i )logP model (x i ), where P data (x i ) is represented as the true joint probability distribution of the i-th sample in the data, P model (x i ) represents the joint probability distribution predicted by the model;

[0100] Select the stochastic gradient descent optimization algorithm to adjust the model parameters by minimizing the loss function. The stochastic gradient descent optimization algorithm (SGD) is a commonly used optimization algorithm for training machine learning models. Its basic idea is to continuously update the model parameters in an iterative manner so that the loss function gradually decreases, thereby achieving the purpose of training the model. Unlike the traditional gradient descent algorithm, SGD only uses one random sample to calculate the gradient in each iteration and updates the model parameters based on this gradient;

[0101] The model is trained using the training data set, and the model parameters are updated through an iterative optimization algorithm so that the loss function is gradually reduced and the model gradually converges to the optimal solution.

[0102] In this embodiment, it should be specifically explained that the steps of evaluating the model using a validation set or a cross-validation method are as follows:

[0103] Use the trained model to predict the samples in the validation set and obtain the prediction results of the model on the validation set;

[0104] Obtaining evaluation indicators of the model through the prediction results, wherein the evaluation indicators include the accuracy, precision, recall and F1 value of the model, and the acquisition of the accuracy, precision, recall and F1 value of the model is a prior art, and the specific steps thereof are not specifically described in this embodiment;

[0105] According to the obtained model evaluation indicators, the value of the selected evaluation indicator is calculated, and its calculation expression is ZQ=f(Z1,Z2,Z3,Z4), where ZQ represents the accuracy index of the model, Z1 represents the accuracy rate of the model, Z2 represents the precision rate of the model, Z3 represents the recall rate of the model, and Z4 represents the F1 value of the model.

[0106] The present invention is not limited to the above-mentioned embodiments. Anyone should be aware that any structural changes made under the enlightenment of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the protection scope of the present invention. The technology, shape, and structural parts not described in detail in the present invention are all well-known technologies.

Claims

1. A model pre-training method based on dynamic particle data enhancement, characterized in that: Includes the following steps: Collect multimodal data, and divide the collected multimodal data into modules, namely, video modal data, voice modal data, and text modal data; The divided image data is rasterized, and the raster size can be determined according to the size of the input data; The rasterized image is used to establish a particle set in a relative position manner, and a particle filter operation is performed, and the particles after the particle filter operation are divided into a training set and a validation set; Design a multimodal model that can process data from various modalities simultaneously, determine the input and output layers of the model, and the connection between the data from each modality; Design a loss function based on joint probability distribution to assist the model in learning the correlation mapping relationship between multimodal features, so that it has the ability of self-reasoning and can recognize unlearned knowledge; Integrate the designed loss function into the model training process, use the dynamic particle data augmentation method to generate diverse training samples to improve the generalization ability of the model, and use the loss function of the joint probability distribution to train the model to learn the correlation mapping relationship between multimodal features; After training is completed, the model is evaluated using the cross-validation method to analyze the model's performance on data of different modalities, as well as the model's learning effect on the associated mapping relationship between multimodal features, and the model is adjusted and optimized as needed.

2. The model pre-training method based on dynamic particle data enhancement according to claim 1, characterized in that: The step of establishing a particle set by using a relative position method for the rasterized image is as follows: Each pixel in the image is regarded as a particle, and the position of the particle is represented by two-dimensional coordinates. The coordinates of each pixel are used as the position information of the particle, and the pixel value is used as the state information of the particle. Select a certain pixel in the image as a reference point and express the positions of other pixels as relative to the reference point; All pixels are converted into particles, and their relative positions and pixel values ​​are used as the state information of the particle set, thus obtaining a particle set representing the spatial structure of the image.

3. The model pre-training method based on dynamic particle data enhancement according to claim 1, characterized in that: The particle filtering operation steps are as follows: Perform CNN to extract image edges, select the extracted edge points as particle sets, and set corresponding initial values ​​for the target representation model and the particle motion model; The iterative process of estimating particles is done using the maximum likelihood estimation method; Determine the weight of each particle according to the Bhattacharyya coefficient; The parameters of the particle motion model are adaptively adjusted according to the observed values ​​of the target characterization state to obtain an estimated value of the target feature characterization.

4. The model pre-training method based on dynamic particle data enhancement according to claim 3, characterized in that: The steps of performing CNN on an image to extract image edges are as follows: Select the CNN model of the LeNet image edge detection task and build a neural network structure for edge detection; Use the training set to train the built CNN model and adjust the model parameters through the back propagation algorithm and optimizer; Evaluate the trained model using the test set to check the performance of the model on unseen data; Use the trained model to extract the edge of the image to be processed, and perform filtering on the extracted edge image.

5. The model pre-training method based on dynamic particle data enhancement according to claim 3, characterized in that: The steps of determining the weight of each particle according to the Bhattacharyya coefficient are as follows: For each particle, calculate its corresponding probability distribution according to the characteristics or states it represents; Calculate the Bhattacharyya coefficient between the probability distribution represented by each particle and the target distribution. The calculation formula is: Among them BC i is the Bhattacharyya coefficient of particle i, P(x) is the target distribution, Q i (x) represents the probability distribution of particle i; According to the calculated Bhattacharyya coefficient, the weight of each particle is obtained by calculating the inverse of the Bhattacharyya coefficient. The calculation formula is: where ω i Expressed as the weight of particle i, BC i It is expressed as the Bhattacharyya coefficient of particle i; The weights are normalized by dividing each weight by the sum of all weights, and the normalized weight is used as the weight of the particle, which is given by 6. The model pre-training method based on dynamic particle data enhancement according to claim 1, characterized in that: The steps of designing a loss function based on joint probability distribution are: Establish the relationship between the video latitude and the prediction result, the relationship is: Where L v Expressed as video loss; Establish the relationship between the voice latitude and the prediction result, the relationship is: Where L a It is expressed as speech loss; Establish the relationship between text latitude and prediction results, the relationship is Where L t Expressed as text loss; The final loss is obtained by averaging the video loss, speech loss, and text loss, and its calculation formula is: L r Expressed as final loss.

7. The model pre-training method based on dynamic particle data enhancement according to claim 1, characterized in that: The steps of evaluating the model using the cross-validation method are: Use the trained model to predict the samples in the validation set and obtain the prediction results of the model on the validation set; The evaluation indicators of the model are obtained through the prediction results, and the evaluation indicators include the accuracy, precision, recall rate and F1 value of the model; According to the obtained model evaluation indicators, the value of the selected evaluation indicator is calculated, and its calculation expression is ZQ=f(Z1,Z2,Z3,Z4), where ZQ represents the accuracy index of the model, Z1 represents the accuracy rate of the model, Z2 represents the precision rate of the model, Z3 represents the recall rate of the model, and Z4 represents the F1 value of the model.

Images