Multi-modal large model training method

Abstract

The invention discloses a multi-modal large model training method, and relates to the technical field of artificial intelligence, and the method comprises the steps: data collection, preprocessing, feature extraction, multi-modal fusion, and model training. During preprocessing, for temperature, humidity and pressure parameters, abnormal value analysis is carried out by calculating an average value and a standard deviation, and a model is obtained; calculating an intermediate value by using front and back normal values to replace an abnormal value; for coordinate position information, calculating a direction angle according to adjacent coordinates, comparing the direction angle with a threshold value to judge abnormity, and replacing abnormal coordinates with middle coordinates of front and back normal coordinates; a feature extraction link: calculating a temperature change rate mean value of adjacent nodes as a feature coefficient according to the temperature; calculating the mean value of the humidity and the absolute value of the mean value difference; the pressure adopts a maximum and minimum parameter difference value; calculating a speed mean value of adjacent nodes according to coordinates; calculating a fusion feature vector according to a preset weight during multi-modal fusion; according to the method, multi-modal data can be effectively processed, and the model performance and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal large model training method. Background Art

[0002] In today's science and technology field, the diversity and complexity of data are increasing. The processing and application of multimodal data has become a key research direction, especially in model training. With the rapid development of sensor technology, a variety of rich information can be collected from the physical environment, which covers different modes, such as temperature, humidity, pressure, and coordinate position. In many practical application scenarios, it is crucial to accurately obtain and process multimodal data. For example, in the field of industrial automation, the data of temperature, humidity, and pressure sensors can reflect the operating status of the equipment, while the coordinate position information may involve the movement trajectory of the equipment or product in space. However, there are often many problems with the original collected data, and the data may produce abnormal values ​​due to factors such as sensor failure and environmental interference. In terms of temperature, humidity, and pressure data, the appearance of abnormal values ​​may lead to deviations in subsequent analysis and fail to accurately reflect the real physical environment conditions. For coordinate position information, abnormal coordinate data may cause location-based analysis and application to produce erroneous results. For example, in scenarios such as navigation positioning and logistics tracking, if the position coordinates are abnormal, it will affect the accuracy of functions such as path planning and cargo monitoring.

[0003] By scientifically and effectively processing and training models for multimodal data, we can provide more accurate and efficient solutions for many fields such as smart monitoring, environmental monitoring, industrial manufacturing, etc., to meet the growing demand for intelligence. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a multimodal large model training method.

[0005] A multimodal large model training method comprises the following steps:

[0006] Step 1: Data Collection

[0007] Collect specified multimodal data from various physical sensors, where the multimodal data includes but is not limited to temperature, humidity, pressure, and parameter information corresponding to coordinate positions;

[0008] Step 2: Preprocessing

[0009] Preprocess the multimodal data to obtain data features after multiple attribute standardization;

[0010] Step 3: Feature extraction

[0011] Performing feature extraction processing on the preprocessed multimodal data, and obtaining a plurality of feature coefficients corresponding to the multimodal data based on the feature extraction processing;

[0012] Step 4: Multimodal Fusion

[0013] Perform feature fusion calculation on multiple feature coefficients corresponding to the multimodal data, and obtain a multimodal fusion feature vector;

[0014] Step 5: Model training

[0015] Use deep learning algorithms to train models, which use a multi-layer neural network structure;

[0016] The model training method is as follows: the fused multimodal fusion feature vector is input into the multi-layer neural network. During the training process, the output value of the model is calculated through forward propagation, the loss value is calculated according to the loss function, and then the gradient of each network parameter is calculated through the back propagation algorithm, and the parameters are updated according to the learning rate; this process is repeated until the loss value of the model on the training set reaches stability or meets the preset accuracy requirements.

[0017] As a further solution of the present invention: the feature extraction processing method is as follows:

[0018] StepA1: For the parameter information corresponding to temperature, humidity and pressure, the preprocessing method is as follows:

[0019] StepA1.1. Select a parameter information and define it as a feature vector C;

[0020] Get the feature vector collected at multiple time nodes and record it as C i , where i = 1, 2, ... n, n represents the number of time nodes;

[0021] StepA1.2, then calculate all C i The mean and standard deviation of , and they are marked as CP and CB respectively;

[0022] StepA1.3, then each C i Combine CP and CB to perform outlier analysis:

[0023] If |C i -CP|>k×CB, which means C corresponding to the corresponding time node i is an outlier;

[0024] If |C i -CP|≤k×CB, which means C corresponding to the corresponding time node i is the normal value;

[0025] Wherein, k is the preset weight coefficient;

[0026] StepA1.4, then extract the C with abnormal value i ;

[0027] Select a C that is an outlier i , and then mark the time node corresponding to the outlier value Ci as a parameter patching node;

[0028] StepA1.5, according to the time trend, obtain the feature vector with the smallest time difference between the corresponding time node and the parameter patching node and the feature vector at its time node being a normal value, and mark it as C i1 ;

[0029] At the same time, according to the time trend, the feature vector with the smallest time difference between the corresponding time node and the parameter patching node and the feature vector at its time node being a normal value is obtained backward, and it is marked as C i2 ;

[0030] StepA1.6, then calculate the two eigenvectors C i1 and C i2 The middle value of the outlier C i 's replacement value;

[0031] StepA2: For the parameter information corresponding to the coordinate position, the preprocessing method is as follows:

[0032] StepA2.1. Extract the position coordinates collected at multiple time nodes and mark the position coordinates of each time node as (x i ,y i );

[0033] StepA2.2, then pass Calculate the direction angle θi from the previous position coordinate to the next position coordinate at each adjacent time node, where the value of i is not n;

[0034] in,

[0035] If x i+1 -x i =0 and y i+1 -y i > 0, the direction angle from the previous position coordinate to the next position coordinate at the relevant adjacent time node is

[0036] If x i+1 -x i =0 and y i+1 -y i <0, then the direction angle from the previous position coordinate to the next position coordinate at the relevant adjacent time node is

[0037] StepA2.3, then compare the direction angle θi from the previous position coordinate to the next position coordinate at each adjacent time node with the preset direction angle threshold θy:

[0038] If θi≤θy, it means that the coordinates of the next position at the adjacent time node are normal;

[0039] If θi>θy, it means that the coordinates of the next position at the adjacent time node are abnormal;

[0040] StepA2.4, then extract the position coordinates of the abnormal result, select a position coordinate of the abnormal result, and obtain the time node corresponding to the coordinate position of the abnormality, and record it as the coordinate repair node;

[0041] StepA2.5, then, according to the time trend, get the position coordinates of the time node with the smallest time difference between the corresponding time node and the coordinate repair node and the normal position coordinates on the time node, and mark it as (x i1 ,y i1 );

[0042] At the same time, according to the time trend, the time difference between the corresponding time node and the coordinate repair node is the smallest and the normal position coordinate on the time node is obtained, and it is marked as (x i2 ,y i2 );

[0043] StepA2.6, then calculate the position coordinates (x i1 ,y i1 ) and position coordinates (x i2 ,y i2 ), and then use its intermediate coordinates as the replacement coordinates of the abnormal position coordinates;

[0044] The middle coordinates are

[0045] As a further solution of the present invention: the feature extraction processing method is as follows:

[0046] Step B1: Select temperature for feature extraction

[0047] StepB1.1, mark the temperature parameters collected at multiple time nodes as Wi;

[0048] At the same time, the time intervals corresponding to adjacent time nodes are extracted and recorded as T i，i+1 ;

[0049] Among them, T i，i+1 Represents the time interval between the i-th time node and the i+1-th time node;

[0050] StepB1.2, subtract Wi from Wi+1 to get the temperature difference WC between the i-th time node and the i+1-th time node i，i+1 , and then divide the temperature difference by the time interval T i，i+1 , the temperature change rate between the i-th time node and the i+1-th time node is obtained;

[0051] Step B1.3, calculate the mean of the corresponding temperature change rates between all adjacent time nodes, and record it as the temperature characteristic coefficient;

[0052] Step B2: Select humidity for feature extraction

[0053] StepB2.1, mark the humidity parameters collected at multiple time nodes as Si;

[0054] Step B2.2, then calculate the average value Sp of all Si, then calculate the absolute value of the difference between each Si and Sp, and then find the average value of all the absolute values ​​of the difference, and record it as the humidity characteristic coefficient;

[0055] Step B3: Select pressure for feature extraction

[0056] Step B3.1, among the pressure parameters collected at multiple time nodes, respectively obtain the pressure parameters with the largest value and the smallest value;

[0057] Step B3.2, then subtract the pressure parameter corresponding to the maximum value from the pressure parameter corresponding to the minimum value, and record the difference as the pressure characteristic coefficient;

[0058] Step B4: Select the position coordinates for feature extraction

[0059] Step B4.1, mark the position coordinates collected at multiple time nodes as (x i ,y i );

[0060] StepB4.2, then pass Calculate the distance D between the i-th time node and the i+1-th time node i，i+1 ;

[0061] Step B4.3, extract the time interval corresponding to adjacent time nodes and record it as T i，i+1 ;

[0062] Step B4.4, D i，i+1 Divide by T i，i+1, the speed value between the i-th time node and the i+1-th time node is obtained, and then the mean of the corresponding speed values ​​between all adjacent time nodes is obtained and recorded as the speed characteristic coefficient.

[0063] As a further solution of the present invention: wherein, the method for extracting the temperature characteristic coefficient can also be obtained according to the method for extracting the humidity characteristic coefficient or the pressure characteristic coefficient;

[0064] The method of extracting the humidity characteristic coefficient can also be obtained in the same way as the method of extracting the temperature characteristic coefficient or the pressure characteristic coefficient;

[0065] The method for extracting the pressure characteristic coefficient may also be obtained according to the method for extracting the temperature characteristic coefficient or the humidity characteristic coefficient.

[0066] As a further solution of the present invention: the feature fusion calculation method is as follows:

[0067] Multiple characteristic coefficients are marked as G j , j = 1, 2, ... m, m represents the number of characteristic coefficients;

[0068] Then through Calculate the multimodal fusion feature vector GR;

[0069] In the formula, β j It is expressed as a preset weight value corresponding to each characteristic coefficient.

[0070] As a further solution of the present invention: the forward propagation calculation method is:

[0071] In a multi-layer neural network, for each node in the input layer, its input value is the corresponding element in the feature vector;

[0072] Then, through the calculation of the hidden layer, the input value of each node in the hidden layer is where w i0 is the weight from the input layer to the hidden layer, x i0 is the value of the input layer node, b is the bias of the hidden layer node, i0=1, 2, ... n0;

[0073] After being processed by an activation function (such as a ReLU function), the output value a=Fz of the hidden layer node is obtained;

[0074] Next, the input value of the output layer node is calculated in a similar way, which is where w` j0 is the weight from the hidden layer to the output layer, a j0 is the output value of the hidden layer node, b` is the bias of the output layer node, and finally the output value y0 of the model is obtained, j0=1, 2,...m0.

[0075] As a further solution of the present invention: the loss value is calculated as follows:

[0076] The difference between the model prediction value y1` and the true target value y1 is calculated according to the loss function;

[0077] Among them, the loss function uses mean square error MSE;

[0078] For a sample data point, the calculation formula of MSE is: Get the loss value L;

[0079] Where k0 is the number of nodes in the output layer, y1` g0 and the true target value y1 g0 are the g0th elements of the true value and the predicted value, respectively, g0=1, 2, ..., k0;

[0080] For the entire training dataset, calculate the average loss value.

[0081] As a further solution of the present invention: the back propagation method is: starting from the output layer, the gradient is calculated according to the gradient calculation formula of the loss function to the weight;

[0082] For the output layer weight W 2 , the gradient calculation formula is in It can be calculated based on the loss function and the chain rule. According to the calculation formula of z`, we can derive it;

[0083] For each element, in is the input value of the pth node in the output layer, is the weight from the i-th node in the hidden layer to the j-th node in the output layer; then the hidden layer weight W is calculated by a similar method 1 The gradient of

[0084] According to the above method, the gradient of the deviation parameter is calculated;

[0085] For the output layer bias b`, For the hidden layer bias b, Where z is the input value of the hidden layer node.

[0086] As a further solution of the present invention: the parameter updating method is:

[0087] Based on the calculated gradient, an optimization algorithm is used to update the parameters of the neural network.

[0088] Among them, the optimization algorithm adopts the gradient descent algorithm;

[0089] For the weight parameter W, the update formula is Where α is the learning rate, which determines the step size of parameter update;

[0090] The update formula of the deviation parameter b is

[0091] Compared with the prior art, the present invention has the following beneficial effects:

[0092] The present invention can effectively remove abnormal data and improve the accuracy and reliability of data for parameter information corresponding to temperature, humidity and pressure through abnormal value analysis and repair mechanism. Taking temperature parameters as an example, when an abnormal value appears, the intermediate value is calculated by replacing it with the normal value feature vector at the previous and next time nodes, which ensures the continuity and stability of the data and provides a high-quality data basis for subsequent feature extraction and model training; for parameter information corresponding to the coordinate position, the direction angle calculation and threshold comparison are used to quickly and accurately determine whether the position coordinate is abnormal. Once the abnormal coordinates are found, they are replaced by the intermediate coordinates of the normal position coordinates at the previous and next time nodes, which ensures the rationality and effectiveness of the position data and provides accurate position information for multimodal data fusion.

[0093] The present invention adopts different feature extraction methods for different modal data, and fully considers the characteristics of various data. For example, for temperature parameters, the temperature characteristic coefficient is obtained by calculating the average value of the temperature change rate between adjacent time nodes, which can reflect the temperature change trend over time; for humidity parameters, the average value of the absolute value of the difference between all humidity values ​​and the average value is calculated as the humidity characteristic coefficient, which reflects the discrete degree and stability of humidity data; for pressure parameters, the pressure characteristic coefficient is obtained by the difference between the maximum value and the minimum value, which simply and intuitively reflects the range of pressure changes; for position coordinate parameters, the average value of the speed value between adjacent time nodes is calculated as the speed characteristic coefficient, which can describe the motion state of the object in space. The extraction of these characteristic coefficients provides rich feature information for multimodal fusion.

[0094] The present invention obtains a multimodal fusion feature vector by performing feature fusion calculation on multiple feature coefficients, which can comprehensively utilize the advantages of different modal data and improve the generalization ability and accuracy of the model. During the fusion process, the calculation is performed based on the preset weight values ​​of each feature coefficient, and the contribution of different features to the model is fully considered, so that the fused feature vector is more representative and discriminative. The fused feature vector is input into a multi-layer neural network for training, and the deep learning algorithm can automatically learn the complex patterns and relationships in the data, thereby improving the prediction ability and adaptability of the model.

[0095] The present invention has a clear forward propagation calculation method, and the calculation process from the input layer to the hidden layer and then to the output layer is clear. The activation function processing can increase the nonlinear expression ability of the model and improve the model's ability to fit complex data. The loss value calculation adopts the mean square error MSE, which can accurately measure the difference between the model prediction value and the true target value, and provides a reliable basis for back propagation and parameter updating. The back propagation algorithm can calculate the gradient of each network parameter according to the loss function, so that the model can continuously adjust the parameters to reduce the loss value. The gradient calculation method for weight and bias parameters is detailed, ensuring the accuracy and effectiveness of parameter updating. The parameter update adopts the gradient descent algorithm, and the step size of the parameter update is controlled by the learning rate, which can avoid falling into the local optimal solution while ensuring the convergence speed.

[0096] In summary, this multimodal large model training method can effectively process multimodal data and improve the performance and accuracy of the model through steps such as data acquisition, preprocessing, feature extraction, multimodal fusion and model training, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 It is a schematic diagram of the process of the present invention;

[0098] Figure 2 Schematic diagram of the feature extraction process in the present invention. DETAILED DESCRIPTION

[0099] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0100] Embodiment 1

[0101] See also Figure 1 and Figure 2 As shown, the present invention is a multimodal large model training method, comprising the following steps:

[0102] Step 1: Data Collection

[0103] Collect specified multimodal data from various physical sensors, where the multimodal data includes but is not limited to temperature, humidity, pressure, and parameter information corresponding to coordinate positions;

[0104] In this embodiment, data collection covers multimodal data, including parameter information corresponding to temperature, humidity, pressure, and coordinate position, which provides a rich data source for the model and can more comprehensively reflect the actual situation.

[0105] Step 2: Feature Extraction

[0106] Performing feature extraction processing on the multimodal data, and obtaining a plurality of feature coefficients corresponding to the multimodal data based on the feature extraction processing;

[0107] The feature extraction process is as follows:

[0108] Step B1. Select the parameter information corresponding to the temperature

[0109] StepB1.1, mark the temperature parameters collected at multiple time nodes as Wi;

[0110] At the same time, the time intervals corresponding to adjacent time nodes are extracted and recorded as T i，i+1 ;

[0111] Among them, T i，i+1 Represents the time interval between the i-th time node and the i+1-th time node;

[0112] StepB1.2, subtract Wi from Wi+1 to get the temperature difference WC between the i-th time node and the i+1-th time node i，i+1 , and then divide the temperature difference by the time interval T i，i+1 , the temperature change rate between the i-th time node and the i+1-th time node is obtained;

[0113] Step B1.3, calculate the mean of the corresponding temperature change rates between all adjacent time nodes, and record it as the temperature characteristic coefficient;

[0114] Step B2: Select the parameter information corresponding to humidity

[0115] StepB2.1, mark the humidity parameters collected at multiple time nodes as Si;

[0116] Step B2.2, then calculate the average value Sp of all Si, then calculate the absolute value of the difference between each Si and Sp, and then find the average value of all the absolute values ​​of the difference, and record it as the humidity characteristic coefficient;

[0117] Step B3: Select the parameter information corresponding to the pressure

[0118] Step B3.1, among the pressure parameters collected at multiple time nodes, respectively obtain the pressure parameters with the largest value and the smallest value;

[0119] Step B3.2, then subtract the pressure parameter corresponding to the maximum value from the pressure parameter corresponding to the minimum value, and record the difference as the pressure characteristic coefficient;

[0120] Step B4: Select the parameter information corresponding to the position coordinates

[0121] Step B4.1, mark the position coordinates collected at multiple time nodes as (x i ,y i );

[0122] StepB4.2, then pass Calculate the distance D between the i-th time node and the i+1-th time node i，i+1 ;

[0123] Step B4.3, extract the time interval corresponding to adjacent time nodes and record it as T i，i+1 ;

[0124] Step B4.4, D i，i+1 Divide by T i，i+1 , get the speed value between the i-th time node and the i+1-th time node, then find the mean of the corresponding speed values ​​between all adjacent time nodes and record it as the speed characteristic coefficient;

[0125] In this embodiment, through feature extraction processing, characteristic coefficients are calculated for data of different modes, such as the mean temperature change rate as the temperature characteristic coefficient, the average value of the absolute value of the humidity difference as the humidity characteristic coefficient, etc., so as to accurately extract the key features of each modal data.

[0126] Step 3: Multimodal Fusion

[0127] Perform feature fusion calculation on multiple feature coefficients corresponding to the multimodal data, and obtain a multimodal fusion feature vector;

[0128] The specific method is as follows:

[0129] Multiple characteristic coefficients are marked as G j , j = 1, 2, ... m, m represents the number of characteristic coefficients;

[0130] Then through Calculate the multimodal fusion feature vector GR;

[0131] In the formula, β j It is expressed as a preset weight value corresponding to each characteristic coefficient;

[0132] In this embodiment, the multimodal fusion step fuses multiple feature coefficients to obtain a multimodal fusion feature vector, making full use of the features of different modal data and improving the comprehensive analysis capability of the model.

[0133] Step 4: Model training

[0134] Use deep learning algorithms to train models, which use a multi-layer neural network structure;

[0135] The model training method is as follows: the fused multimodal fusion feature vector is input into the multi-layer neural network. During the training process, the output value of the model is calculated by forward propagation, the loss value is calculated according to the loss function, and then the gradient of each network parameter is calculated by the back propagation algorithm, and the parameters are updated according to the learning rate. This process is repeated until the loss value of the model on the training set reaches stability or meets the preset accuracy requirements.

[0136] Among them, the forward propagation calculation method is:

[0137] In a multi-layer neural network, for each node in the input layer, its input value is the corresponding element in the feature vector;

[0138] Then, through the calculation of the hidden layer, the input value of each node in the hidden layer is where w i0 is the weight from the input layer to the hidden layer, x i0 is the value of the input layer node, b is the bias of the hidden layer node, i0=1, 2, ... n0;

[0139] After being processed by an activation function (such as a ReLU function), the output value a=Fz of the hidden layer node is obtained;

[0140] Next, the input value of the output layer node is calculated in a similar way, which is where w` j0 is the weight from the hidden layer to the output layer, a j0 is the output value of the hidden layer node, b` is the bias of the output layer node, and finally the output value y0 of the model is obtained, j0=1, 2, ... m0;

[0141] The loss value is calculated as:

[0142] The difference between the model prediction value y1` and the true target value y1 is calculated according to the loss function;

[0143] In this embodiment, the loss function adopts mean square error MSE;

[0144] For a sample data point, the calculation formula of MSE is: Get the loss value L;

[0145] Where k0 is the number of nodes in the output layer, y1` g0 and the true target value y1 g0 are the g0th elements of the true value and the predicted value, respectively, g0=1, 2, ..., k0;

[0146] For the entire training dataset, calculate the average loss value;

[0147] The back propagation method is: starting from the output layer, the gradient is calculated according to the gradient calculation formula of the loss function to the weight;

[0148] For the output layer weight W 2 , the gradient calculation formula is in It can be calculated based on the loss function and the chain rule. According to the calculation formula of z`, we can derive it;

[0149] For each element, in is the input value of the pth node in the output layer, is the weight from the i-th node in the hidden layer to the j-th node in the output layer). Then the hidden layer weight W is calculated by a similar method. 1 gradient.

[0150] According to the above method, the gradient of the deviation parameter is calculated;

[0151] For the output layer bias b`, For the hidden layer bias b, Where z is the input value of the hidden layer node;

[0152] The parameter update method is:

[0153] Based on the calculated gradient, an optimization algorithm is used to update the parameters of the neural network.

[0154] Among them, the optimization algorithm adopts the gradient descent algorithm;

[0155] For the weight parameter W, the update formula is Where α is the learning rate, which determines the step size of parameter update;

[0156] The update formula of the deviation parameter b is

[0157] In this embodiment, a deep learning algorithm is used for model training. A multi-layer neural network structure is used to continuously optimize the model parameters through steps such as forward propagation, loss value calculation, back propagation, and parameter update, so that the loss value of the model on the training set becomes stable or meets the preset accuracy requirements, thereby improving the accuracy and reliability of the model.

[0158] Embodiment 2

[0159] As the second embodiment of the present invention, when the present application is implemented, compared with the first embodiment, the technical solution of the present embodiment is different from that of the first embodiment only in that:

[0160] In this embodiment, the method for extracting the temperature characteristic coefficient may also be obtained according to the method for extracting the humidity characteristic coefficient or the pressure characteristic coefficient;

[0161] The method of extracting the humidity characteristic coefficient can also be obtained in the same way as the method of extracting the temperature characteristic coefficient or the pressure characteristic coefficient;

[0162] The method for extracting the pressure characteristic coefficient may also be obtained according to the method for extracting the temperature characteristic coefficient or the humidity characteristic coefficient.

[0163] This embodiment increases the flexibility of the characteristic coefficient extraction method. The temperature, humidity and pressure characteristic coefficients can be obtained according to the other two extraction methods, providing more options for different data characteristics and improving the adaptability and versatility of the method.

[0164] Embodiment 3

[0165] As the third embodiment of the present invention, when the present application is implemented, compared with the first and second embodiments, the technical solution of this embodiment is to combine the solutions of the first and second embodiments, and the difference between the technical solution of this embodiment and the first and second embodiments is that:

[0166] This embodiment also includes the following pre-processing steps:

[0167] Preprocess the multimodal data to obtain data features after multiple attribute standardization;

[0168] StepA1: For the parameter information corresponding to temperature, humidity and pressure, the preprocessing method is as follows:

[0169] StepA1.1. Select a parameter information and define it as a feature vector C;

[0170] Get the feature vector collected at multiple time nodes and record it as C i , where i = 1, 2, ... n, n represents the number of time nodes;

[0171] StepA1.2, then calculate all C i The mean and standard deviation of , and they are marked as CP and CB respectively;

[0172] StepA1.3, then each C i Combine CP and CB to perform outlier analysis:

[0173] If |C i -CP|>k×CB, which means C corresponding to the corresponding time node i is an outlier;

[0174] If |C i-CP|≤k×CB, which means C corresponding to the corresponding time node i is the normal value;

[0175] Wherein, k is a preset weight coefficient, and in this embodiment, the value of k is 2.5;

[0176] StepA1.4, then extract the C with abnormal value i ;

[0177] Select a C that is an outlier i , and then mark the time node corresponding to the outlier value Ci as a parameter patching node;

[0178] StepA1.5, according to the time trend, obtain the feature vector with the smallest time difference between the corresponding time node and the parameter patching node and the feature vector at its time node being a normal value, and mark it as C i1 ;

[0179] At the same time, according to the time trend, the feature vector with the smallest time difference between the corresponding time node and the parameter patching node and the feature vector at its time node being a normal value is obtained backward, and it is marked as C i2 ;

[0180] StepA1.6, then calculate the two eigenvectors C i1 and C i2 The middle value of the outlier C i 's replacement value;

[0181] For example:

[0182] When the feature vectors at the two time nodes adjacent to the time node i corresponding to the abnormal value are both normal values, the middle value of the two feature vectors is calculated, and then the middle value is used as the replacement value of the abnormal value;

[0183] Among them, the two adjacent time nodes are i-1 and i+1 respectively;

[0184] If at least one of the feature vectors at two adjacent time nodes is an outlier, it is assumed that: when the feature vector at the i-1th time node is an outlier, and the feature vector at the i+1th time node is a normal value;

[0185] Then, the feature vector at the i-2th time node is obtained, and then it is determined whether the feature vector at this time node is an outlier:

[0186] If it is not an outlier, then find the middle value between the feature vector at the i-2th time node and the feature vector at the i+1th time node, and use the middle value as the outlier C.i 's replacement value;

[0187] If it is an outlier, then obtain the feature vector at the i-2th time node, and then determine whether the feature vector at this time node is an outlier, and so on, until the feature vector at the corresponding time node is not an outlier;

[0188] StepA2: For the parameter information corresponding to the coordinate position, the preprocessing method is as follows:

[0189] StepA2.1. Extract the position coordinates collected at multiple time nodes and mark the position coordinates of each time node as (x i ,y i );

[0190] StepA2.2, then pass Calculate the direction angle θi from the previous position coordinate to the next position coordinate at each adjacent time node, where the value of i is not n;

[0191] in,

[0192] If x i+1 -x i =0 and y i+1 -y i > 0, the direction angle from the previous position coordinate to the next position coordinate at the relevant adjacent time node is

[0193] If x i+1 -x i =0 and y i+1 -y i <0, then the direction angle from the previous position coordinate to the next position coordinate at the relevant adjacent time node is

[0194] StepA2.3, then compare the direction angle θi from the previous position coordinate to the next position coordinate at each adjacent time node with the preset direction angle threshold θy:

[0195] If θi≤θy, it means that the coordinates of the next position at the adjacent time node are normal;

[0196] If θi>θy, it means that the coordinates of the next position at the adjacent time node are abnormal;

[0197] StepA2.4, then extract the position coordinates of the abnormal result, select a position coordinate of the abnormal result, and obtain the time node corresponding to the coordinate position of the abnormality, and record it as the coordinate repair node;

[0198] StepA2.5, then, according to the time trend, get the position coordinates of the time node with the smallest time difference between the corresponding time node and the coordinate repair node and the normal position coordinates on the time node, and mark it as (x i1 ,y i1 );

[0199] At the same time, according to the time trend, the time difference between the corresponding time node and the coordinate repair node is the smallest and the normal position coordinate on the time node is obtained, and it is marked as (x i2 ,y i2 );

[0200] StepA2.6, then calculate the position coordinates (x i1 ,y i1 ) and position coordinates (x i2 ,y i2 ), and then use its intermediate coordinates as the replacement coordinates of the abnormal position coordinates;

[0201] The middle coordinates are

[0202] This embodiment adds a preprocessing step to standardize the multimodal data and repair outliers to improve data quality.

[0203] For temperature, humidity, and pressure parameter information, outlier analysis is performed by calculating the mean and standard deviation, and the median value is used to replace the outlier to ensure the accuracy and reliability of the data.

[0204] For the parameter information corresponding to the coordinate position, the position coordinates are judged whether they are abnormal by comparing the calculated direction angle with the preset threshold, and the abnormal coordinates are replaced by the intermediate coordinates to improve the accuracy of the position data.

[0205] Embodiment 4

[0206] As the fourth embodiment of the present invention, when the present application is specifically implemented, compared with the first, second and third embodiments, the technical solution of this embodiment is to combine and implement the solutions of the above-mentioned first, second and third embodiments.

[0207] This embodiment combines the advantages of embodiments one, two, and three, and implements a combination of multiple solutions to further improve the performance and applicability of the multimodal large model, so as to more accurately process multimodal data and improve the training effect and prediction ability of the model.

[0208] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0209] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A multimodal large model training method, characterized in that: The following steps are involved: Step 1: Collect specified multimodal data from various physical sensors, where the multimodal data includes but is not limited to temperature, humidity, pressure, and parameter information corresponding to coordinate positions; Step 2: Preprocess the multimodal data to obtain data features after multiple attribute standardization; Step 3: performing feature extraction processing on the preprocessed multimodal data, and obtaining a plurality of feature coefficients corresponding to the multimodal data based on the feature extraction processing; Step 4: Perform feature fusion calculation on multiple feature coefficients corresponding to the multimodal data, and obtain a multimodal fusion feature vector; Step 5: Use a deep learning algorithm to train the multimodal fusion feature vector model, which adopts a multi-layer neural network structure.

2. A multimodal large model training method according to claim 1, characterized in that: The preprocessing method for the parameter information corresponding to temperature, humidity and pressure is as follows: StepA1.

1. Select a parameter information and define it as a feature vector C; Get the feature vector collected at multiple time nodes and record it as C i , where i = 1, 2, ... n, n represents the number of time nodes; StepA1.2, then calculate all C i The mean and standard deviation of , and they are marked as CP and CB respectively; StepA1.3, then each C i Combine CP and CB to perform outlier analysis: If |C i -CP|>k×CB, which means C corresponding to the corresponding time node i is an outlier; If |C i -CP|≤k×CB, which means C corresponding to the corresponding time node i is the normal value; Wherein, k is the preset weight coefficient; StepA1.4, then extract the C with abnormal value i ; Select a C that is an outlier i , and then mark the time node corresponding to the abnormal value Ci as a parameter repair node; StepA1.5, according to the time trend, obtain the feature vector with the smallest time difference between the corresponding time node and the parameter patching node and the feature vector at its time node being a normal value, and mark it as C i1 ; At the same time, according to the time trend, the feature vector with the smallest time difference between the corresponding time node and the parameter patching node and the feature vector at its time node being a normal value is obtained backward, and it is marked as C i2 ; StepA1.6, then calculate the two eigenvectors C i1 and C i2 The middle value of the outlier C i The replacement value of .

3. A multimodal large model training method according to claim 2, characterized in that: The preprocessing method for the coordinate position corresponding parameter information is as follows: StepA2.

1. Extract the position coordinates collected at multiple time nodes and mark the position coordinates of each time node as (x i ,y i ); StepA2.2, according to the time trend, calculate the vector of the position coordinates corresponding to the adjacent time nodes, and then combine the vector with the inverse tangent function to calculate the direction angle from the previous position coordinate to the next position coordinate at each adjacent time node, and mark it as θi; StepA2.3, compare the direction angle θi from the previous position coordinate to the next position coordinate at each adjacent time node with the preset direction angle threshold θy: If θi≤θy, it means that the coordinates of the next position at the adjacent time node are normal, otherwise, it means that the coordinates of the next position at the adjacent time node are abnormal; StepA2.4, then extract the position coordinates of the abnormal result, select a position coordinate of the abnormal result, and obtain the time node corresponding to the coordinate position of the abnormality, and record it as the coordinate repair node; StepA2.5, then, according to the time trend, get the position coordinates of the time node with the smallest time difference between the corresponding time node and the coordinate repair node and the normal position coordinates on the time node, and mark it as (x i1 ,y i1 ); At the same time, according to the time trend, the time difference between the corresponding time node and the coordinate repair node is the smallest and the normal position coordinate on the time node is obtained, and it is marked as (x i2 ,y i2 ); StepA2.6, then calculate the position coordinates (x i1 ,y i1 ) and position coordinates (x i2 ,y i2 ), and then use its intermediate coordinates as the replacement coordinates of the abnormal position coordinates.

4. A multimodal large model training method according to claim 3, characterized in that: In StepA2.6, the intermediate coordinates are 5. A multimodal large model training method according to claim 3, characterized in that: The feature extraction and processing method of temperature corresponding parameter information is as follows: StepB1.1, mark the temperature parameters collected at multiple time nodes as Wi; At the same time, the time intervals corresponding to adjacent time nodes are extracted and recorded as T i，i+1 ; Among them, T i，i+1 Represents the time interval between the i-th time node and the i+1-th time node; StepB1.2, subtract Wi from Wi+1 to get the temperature difference WC between the i-th time node and the i+1-th time node i，i+1 , and then divide the temperature difference by the time interval T i，i+1 , the temperature change rate between the i-th time node and the i+1-th time node is obtained; Step B1.3, calculate the mean of the corresponding temperature change rates between all adjacent time nodes and record it as the temperature characteristic coefficient.

6. A multimodal large model training method according to claim 5, characterized in that: The feature extraction and processing method of humidity corresponding parameter information is as follows: StepB2.1, mark the humidity parameters collected at multiple time nodes as Si; Step B2.2, then calculate the average value Sp of all Si, then calculate the absolute value of the difference between each Si and Sp, and then find the average value of all the absolute values ​​of the difference and record it as the humidity characteristic coefficient.

7. A multimodal large model training method according to claim 6, characterized in that: The feature extraction and processing method of pressure corresponding parameter information is as follows: Step B3.1, among the pressure parameters collected at multiple time nodes, respectively obtain the pressure parameters with the largest value and the smallest value; Step B3.2, then subtract the pressure parameter corresponding to the maximum value from the pressure parameter corresponding to the minimum value, and record the difference as the pressure characteristic coefficient.

8. A multimodal large model training method according to claim 7, characterized in that: The feature extraction and processing method of the parameter information corresponding to the position coordinates is as follows: Step B4.1, mark the position coordinates collected at multiple time nodes as (x i ,y i ); Step B4.2, then calculate the distance D between the i-th time node and the i+1-th time node using the Euclidean distance formula i，i+1 ; Step B4.3, extract the time interval corresponding to adjacent time nodes and record it as T i，i+1 ; Step B4.4, D i，i+1 Divide by T i，i+1 , the speed value between the i-th time node and the i+1-th time node is obtained, and then the mean of the corresponding speed values ​​between all adjacent time nodes is obtained and recorded as the speed characteristic coefficient.

9. A multimodal large model training method according to claim 8, characterized in that feature fusion The calculation is as follows: Multiple characteristic coefficients are marked as G j , j = 1, 2, ... m, m represents the number of characteristic coefficients; Then through Calculate the multimodal fusion feature vector GR; In the formula, β j It is expressed as a preset weight value corresponding to each characteristic coefficient.

10. A multimodal large model training method according to claim 1, characterized in that: The model training method in step five is: input the fused multimodal fusion feature vector into the multi-layer neural network. During the training process, the output value of the model is calculated through forward propagation, the loss value is calculated according to the loss function, and then the gradient of each network parameter is calculated through the back propagation algorithm, and the parameters are updated according to the learning rate; repeat the above process until the loss value of the model on the training set reaches stability or meets the preset accuracy requirements.

Images