A multi-modal test data fusion evaluation processing method and device

Abstract

The application discloses a kind of multi-modal test data fusion evaluation processing method and device, the method includes: the multi-modal test data set is obtained by acquisition;The multi-modal test data set includes the test numerical information sequence set, test text information sequence set, test picture information sequence set obtained by acquisition in equipment system evaluation process;The multi-modal test data set is preprocessed, and the preprocessed data set is obtained;Using the fusion evaluation model that is trained, the preprocessed data set is processed, and the fusion evaluation result value of equipment system evaluation is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of industrial data processing and the field of equipment system effectiveness evaluation technology, and in particular to a multi-modal test data fusion evaluation processing method and device. BACKGROUND

[0002] In modern equipment system evaluation, multi-modal test data fusion processing is a key link to achieve accurate evaluation. Multi-modal test data usually includes test numerical information, test text information and test picture information and other forms. These data are widely sourced, various in form and interrelated, and can provide comprehensive information support for performance evaluation of equipment systems. However, the existing multi-modal data processing methods have many shortcomings.

[0003] Firstly, noise and redundant information are easily introduced in the process of collecting multi-modal data, resulting in uneven data quality. For example, there may be outliers in test numerical information, irrelevant content in test text information, and blurred or incomplete conditions in test picture information. These noise and redundant information can seriously affect the effectiveness and usability of data, and thus reduce the accuracy of evaluation results.

[0004] Secondly, multi-modal data fusion processing faces the problem of inconsistent data formats and dimensional differences. Different modal data has different structures and characteristics, and direct fusion will lead to information loss or poor fusion effect. For example, the processing methods of numerical data and text data are completely different, and the high-dimensional characteristics of picture data also increase the complexity of fusion.

[0005] In addition, the existing fusion evaluation models mostly rely on feature extraction of single modal data, lacking deep fusion and comprehensive analysis of multi-modal data. This single modal processing method cannot fully utilize the complementary information between multi-modal data, and it is difficult to achieve comprehensive and accurate evaluation of equipment system performance.

[0006] How to efficiently and accurately evaluate the effectiveness of equipment systems based on multi-modal test data is a problem that needs to be solved at present. SUMMARY

[0007] The present application mainly solves the problem of efficiently and accurately evaluating the effectiveness of equipment systems based on multi-modal test data, and discloses a multi-modal test data fusion evaluation processing method and device.

[0008] In a first aspect, the present application discloses a multi-modal test data fusion evaluation processing method, comprising:

[0009] S1, a multi-modal test data set is collected; the multi-modal test data set includes a test numerical information sequence set, a test text information sequence set and a test picture information sequence set collected in the equipment system evaluation process; the test numerical information sequence set includes a test numerical information sequence; the test text information sequence set includes a test text information sequence; and the test picture information sequence set includes a test picture information sequence;

[0010] S2, the multi-modal test data set is preprocessed to obtain a preprocessed data set;

[0011] S3, the preprocessed data set is processed by using the trained fusion evaluation model to obtain a fusion evaluation result value of the equipment system evaluation.

[0012] The multi-modal test data set is preprocessed to obtain a preprocessed data set, including:

[0013] S21, the multi-modal test data set is subjected to data cleaning processing to obtain a first data set;

[0014] S22, the first data set is subjected to format detection processing to obtain a second data set;

[0015] S23, the second data set is subjected to data reduction processing to obtain a preprocessed data set.

[0016] The second data set is subjected to data reduction processing to obtain a preprocessed data set, including:

[0017] S231, for each type of data attribute of the second data set, the data is subjected to autoregressive-moving average modeling with the data acquisition information of the data as the independent variable and the data value of the data as the dependent variable, to obtain a first approximation model of the type of data attribute;

[0018] S232, all information sequences of the information sequence set of each type of data attribute are used as row vectors to construct an information matrix of the type of data attribute;

[0019] S233, the information matrix is subjected to singular value calculation processing to obtain a singular value sequence;

[0020] S234, the element value of the singular value sequence is used as a known dependent variable, and the element sequence number of the singular value sequence is used as a known independent variable, and the known independent variable and the known dependent variable are used to construct a to-be-approximated curve; the to-be-approximated curve is subjected to polynomial fitting to obtain a second approximation model of the type of data attribute;

[0021] S235, multiplying the second approximation model and the first approximation model to obtain a fusion test model of the data attribute;

[0022] S236, using the fusion test model of each data attribute to calculate and process the data acquisition information of the data of the data attribute to obtain an approximate dependent variable;

[0023] S237, determining whether the absolute value of the difference between the approximate dependent variable and the corresponding data is greater than a first regression threshold value; if greater than the first regression threshold value, deleting the data from the second data set; if less than or equal to the first regression threshold value, not processing the data;

[0024] S238, performing fusion processing on all data after S236 to S237 of the second data set to obtain a preprocessed data set.

[0025] The fusion evaluation model comprises a first feature extraction network, a second feature extraction network and a feature fusion network;

[0026] The first feature extraction network comprises a first input module, a first convolution module, a depth separable convolution module, a first dimension increasing convolution module, a second dimension increasing convolution module, a third dimension increasing convolution module, a fourth dimension increasing convolution module, a second convolution module, a first pooling module, a third convolution module and a first full connection module;

[0027] An input end of a first input module of the first feature extraction network is configured to receive a numerical information sequence; an output end of the first input module of the first feature extraction network is connected to an input end of a first convolution module of the first feature extraction network; an output end of the first convolution module of the first feature extraction network is connected to an input end of a depth separable convolution module of the first feature extraction network; an output end of the depth separable convolution module of the first feature extraction network is connected to an input end of a first dimension increasing convolution module of the first feature extraction network; an output end of the first dimension increasing convolution module of the first feature extraction network is connected to an input end of a second dimension increasing convolution module of the first feature extraction network; an output end of the second dimension increasing convolution module of the first feature extraction network is connected to an input end of a third dimension increasing convolution module of the first feature extraction network; an output end of the third dimension increasing convolution module of the first feature extraction network is connected to an input end of a fourth dimension increasing convolution module of the first feature extraction network; an output end of the fourth dimension increasing convolution module of the first feature extraction network is connected to an input end of a second convolution module of the first feature extraction network; an output end of the second convolution module of the first feature extraction network is connected to an input end of a first pooling module of the first feature extraction network; an output end of the first pooling module of the first feature extraction network is connected to an input end of a third convolution module of the first feature extraction network; an output end of the third convolution module of the first feature extraction network is connected to an input end of a first full connection module of the first feature extraction network; and an output end of the first full connection module of the first feature extraction network is configured to output feature information of the numerical information sequence.

[0028] The second feature extraction network comprises a second input module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a second pooling module, a seventh convolution module, a second full connection module and a third full connection module.

[0029] The first input end of the second input module of the second feature extraction network is configured to receive the text information; the second input end of the second input module of the second feature extraction network is connected with the output end of the second dimension-increasing convolution module of the first feature extraction network; the output end of the second input module of the second feature extraction network is connected with the input end of the fourth convolution module of the second feature extraction network; the output end of the fourth convolution module of the second feature extraction network is connected with the input end of the fifth convolution module of the second feature extraction network; the output end of the fifth convolution module of the second feature extraction network is connected with the input end of the sixth convolution module of the second feature extraction network; the output end of the sixth convolution module of the second feature extraction network is connected with the input end of the second pooling module of the second feature extraction network; the output end of the second pooling module of the second feature extraction network is connected with the input end of the seventh convolution module of the second feature extraction network; the output end of the seventh convolution module of the second feature extraction network is connected with the input end of the second full connection module of the second feature extraction network; the output end of the second full connection module of the second feature extraction network is connected with the input end of the third full connection module of the second feature extraction network; and the output end of the second full connection module of the second feature extraction network is configured to output the feature information of the text information sequence.

[0030] The feature fusion network comprises a difference module, a cross-scale fusion attention module, an eighth convolution module, an up-sampling module and a ninth convolution module, the output end of the cross-scale fusion attention module is connected with the input end of the eighth convolution module, the output end of the up-sampling module is connected with the input end of the ninth convolution module, the output end of the eighth convolution module is connected with the first input end of the difference module, and the output end of the ninth convolution module is connected with the second input end of the difference module.

[0031] The first input end and the second input end of the feature fusion network are respectively the input end of the cross-scale fusion attention module and the input end of the up-sampling module; the input end of the cross-scale fusion attention module is connected with the output end of the first full connection module of the first feature extraction network and the output end of the second full connection module of the second feature extraction network respectively.

[0032] The input end of the up-sampling module is configured to receive the picture information; and the output end of the feature fusion network is the output end of the difference module.

[0033] The output end of the difference module is configured to output a fusion evaluation result value of the equipment system evaluation.

[0034] The training process of the fusion evaluation model comprises:

[0035] obtaining a training data set; the training data set comprising training data and corresponding label information; the training data comprising numerical information, text information and picture information collected in equipment system evaluation, and the label information being an evaluation result value of the equipment system corresponding to the training data;

[0036] initializing a training iteration number value;

[0037] inputting the training data in the training data set as input data into the fusion evaluation model;

[0038] processing the input data by using the fusion evaluation model to obtain a prediction value; the prediction value being a fusion evaluation result value of the equipment system evaluation;

[0039] performing difference calculation processing on the prediction value obtained and the label information corresponding to the input data to obtain a difference value;

[0040] judging whether the difference value meets a convergence condition to obtain a first judgment result;

[0041] when the first judgment result is no, judging whether the training iteration number value is equal to a training number threshold to obtain a second judgment result;

[0042] when the second judgment result is no, determining that a model training state is not meeting a training termination condition;

[0043] when the second judgment result is yes, determining that the model training state is meeting the training termination condition;

[0044] when the first judgment result is yes, determining that the model training state is meeting the training termination condition;

[0045] when the model training state is not meeting the training termination condition, performing parameter updating on the fusion evaluation model by using a parameter updating model, increasing the training iteration number value by 1, and triggering execution of inputting the training data in the training data set as input data into the fusion evaluation model;

[0046] when the model training state is meeting the training termination condition, completing the training processing process of the fusion evaluation model to obtain a trained fusion evaluation model.

[0047] the parameter updating model has a calculation expression as follows:

[0048]

[0049] wherein, θ represents a parameter of the fusion evaluation model, l1 represents an updating step length of the fusion evaluation model, 0 < l1 < 1, represents gradient of the parameter in L1(0), L1(0) represents a loss function of the fusion evaluation model, represents a parameter update value of the fusion evaluation model, represents using updating the parameter 0 of the fusion evaluation model.

[0050] The second aspect of the present application discloses a multi-modal test data fusion evaluation processing device, the device comprises:

[0051] a memory storing executable program codes;

[0052] a processor coupled with the memory;

[0053] the processor calls the executable program codes stored in the memory to execute the multi-modal test data fusion evaluation processing method.

[0054] The third aspect of the present application discloses a computer storage medium, the computer storage medium stores computer instructions, when the computer instructions are called by a computer, the computer instructions are used to execute the multi-modal test data fusion evaluation processing method.

[0055] The fourth aspect of the present application discloses an information data processing terminal, the information data processing terminal is used to realize the multi-modal test data fusion evaluation processing method.

[0056] The beneficial effects of the present application are:

[0057] In the data preprocessing stage, the present application adopts data cleaning, format detection and data reduction processing, effectively removes noise and redundant information, and improves the quality and availability of data. Through autoregressive-moving average modeling and singular value calculation processing, combined with polynomial fitting and fusion test model, the data is deeply screened and optimized, and the purity and consistency of the data are further improved. This series of preprocessing operations provides a high-quality data basis for subsequent fusion evaluation, avoiding the interference of noise data on the evaluation results.

[0058] The fusion evaluation model of the present application adopts the architecture of multi-feature extraction network and feature fusion network, which can deeply extract and comprehensively analyze multi-modal data. The first feature extraction network extracts features of numerical data, text data and picture data through multi-layer convolution modules and dimensionality increasing convolution modules, fully utilizes the efficiency of the depth separable convolution module and the classification ability of the first full connection module. This deep fusion method of multi-modal data can fully utilize the complementary information between different modal data, and realize comprehensive and accurate evaluation of the performance of the equipment system. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 An implementation flowchart of the method of the present application. DETAILED DESCRIPTION

[0060] For a better understanding of the present application, an embodiment is given here.

[0061] Figure 1 An implementation flowchart of the method of the present application.

[0062] In a first aspect of the embodiments of the present application, a multi-modal test data fusion evaluation processing method is disclosed, comprising:

[0063] S1, a multi-modal test data set is collected; the multi-modal test data set includes a test numerical information sequence set, a test text information sequence set and a test picture information sequence set collected in the equipment system evaluation process; the test numerical information sequence set includes a test numerical information sequence; the test text information sequence set includes a test text information sequence; and the test picture information sequence set includes a test picture information sequence;

[0064] S2, the multi-modal test data set is preprocessed to obtain a preprocessed data set;

[0065] S3, the preprocessed data set is processed by using a trained fusion evaluation model to obtain a fusion evaluation result value of the equipment system evaluation;

[0066] The preprocessing of the multi-modal test data set to obtain the preprocessed data set comprises:

[0067] S21, the multi-modal test data set is subjected to data cleaning processing to obtain a first data set;

[0068] S22, the first data set is subjected to format detection processing to obtain a second data set;

[0069] S23, the second data set is subjected to data reduction processing to obtain a preprocessed data set.

[0070] The data reduction processing of the second data set to obtain the preprocessed data set comprises:

[0071] S231, for each type of data attribute of the second data set, autoregressive-moving average modeling is performed on the data of the data attribute with the data acquisition information of the data as the independent variable and the data value of the data as the dependent variable to obtain a first approximation model of the type of data attribute;

[0072] S232, using all information sequences of the information sequence set of each type of data attribute as a row vector, an information matrix of the type of data attribute is constructed;

[0073] S233, singular value calculation processing is performed on the information matrix to obtain a singular value sequence; the singular value sequence is a sequence composed of all singular values;

[0074] S234, taking the element value of the singular value sequence as a known dependent variable and the element sequence number of the singular value sequence as a known independent variable, a to-be-approximated curve is constructed using the known independent variable and the known dependent variable; polynomial fitting is performed on the to-be-approximated curve to obtain a second approximation model of the type of data attribute;

[0075] S235, the second approximation model and the first approximation model are multiplied to obtain a fusion test model of the type of data attribute;

[0076] S236, using the fusion test model of each type of data attribute, the data acquisition information of the data of the type of data attribute is calculated and processed to obtain an approximate dependent variable;

[0077] S237, it is judged whether the absolute value of the difference between the approximate dependent variable and the corresponding data is greater than a set first regression discrimination threshold; if greater than the first regression discrimination threshold, the data is deleted from the second data set, and if less than or equal to the first regression discrimination threshold, the data is not processed;

[0078] S238, the data after S236 to S237 is performed on all data of the second data set is fused to obtain a preprocessed data set.

[0079] The fusion evaluation model comprises a first feature extraction network, a second feature extraction network and a feature fusion network;

[0080] The first feature extraction network comprises a first input module, a first convolution module, a depth separable convolution module, a first dimension increasing convolution module, a second dimension increasing convolution module, a third dimension increasing convolution module, a fourth dimension increasing convolution module, a second convolution module, a first pooling module, a third convolution module and a first full connection module;

[0081] An input end of a first input module of the first feature extraction network is configured to receive a numerical information sequence; an output end of the first input module of the first feature extraction network is connected to an input end of a first convolution module of the first feature extraction network; an output end of the first convolution module of the first feature extraction network is connected to an input end of a deep separable convolution module of the first feature extraction network; an output end of the deep separable convolution module of the first feature extraction network is connected to an input end of a first dimension-increasing convolution module of the first feature extraction network; an output end of the first dimension-increasing convolution module of the first feature extraction network is connected to an input end of a second dimension-increasing convolution module of the first feature extraction network; an output end of the second dimension-increasing convolution module of the first feature extraction network is connected to an input end of a third dimension-increasing convolution module of the first feature extraction network; an output end of the third dimension-increasing convolution module of the first feature extraction network is connected to an input end of a fourth dimension-increasing convolution module of the first feature extraction network; an output end of the fourth dimension-increasing convolution module of the first feature extraction network is connected to an input end of a second convolution module of the first feature extraction network; an output end of the second convolution module of the first feature extraction network is connected to an input end of a first pooling module of the first feature extraction network; an output end of the first pooling module of the first feature extraction network is connected to an input end of a third convolution module of the first feature extraction network; an output end of the third convolution module of the first feature extraction network is connected to an input end of a first full connection module of the first feature extraction network; and an output end of the first full connection module of the first feature extraction network is configured to output feature information of the numerical information sequence.

[0082] The second feature extraction network comprises a second input module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a second pooling module, a seventh convolution module, a second full connection module and a third full connection module.

[0083] The first input end of the second input module of the second feature extraction network is configured to receive text information; the second input end of the second input module of the second feature extraction network is connected with the output end of the second dimension-increasing convolution module of the first feature extraction network; the output end of the second input module of the second feature extraction network is connected with the input end of the fourth convolution module of the second feature extraction network; the output end of the fourth convolution module of the second feature extraction network is connected with the input end of the fifth convolution module of the second feature extraction network; the output end of the fifth convolution module of the second feature extraction network is connected with the input end of the sixth convolution module of the second feature extraction network; the output end of the sixth convolution module of the second feature extraction network is connected with the input end of the second pooling module of the second feature extraction network; the output end of the second pooling module of the second feature extraction network is connected with the input end of the seventh convolution module of the second feature extraction network; the output end of the seventh convolution module of the second feature extraction network is connected with the input end of the second full connection module of the second feature extraction network; the output end of the second full connection module of the second feature extraction network is connected with the input end of the third full connection module of the second feature extraction network; and the output end of the second full connection module of the second feature extraction network is configured to output feature information of the text information sequence.

[0084] The feature fusion network comprises a difference module, a cross-scale fusion attention module, an eighth convolution module, an up-sampling module and a ninth convolution module, the output end of the cross-scale fusion attention module is connected with the input end of the eighth convolution module, the output end of the up-sampling module is connected with the input end of the ninth convolution module, the output end of the eighth convolution module is connected with the first input end of the difference module, and the output end of the ninth convolution module is connected with the second input end of the difference module.

[0085] The input end of the cross-scale fusion attention module is connected with the output end of the first full connection module of the first feature extraction network and the output end of the second full connection module of the second feature extraction network respectively.

[0086] The input end of the up-sampling module is configured to receive picture information; the first input end and the second input end of the feature fusion network are respectively the input end of the cross-scale fusion attention module and the input end of the up-sampling module; and the output end of the feature fusion network is the output end of the difference module.

[0087] The output end of the difference module is configured to output a fusion evaluation result value of the equipment system evaluation.

[0088] The cross-scale fusion attention module is implemented by using a multi-head attention model in a Transfromer model.

[0089] The residual module can be implemented by using a Resnet network.

[0090] The difference module is configured to implement a differential evolution algorithm. The differential evolution algorithm realizes optimization through three core operations of differential mutation, crossover and selection. First, a new parameter vector is generated through differential mutation operation. Then, the trial individual is generated by combining the mutation vector and the current individual through the crossover operation. Finally, the fitness of the trial individual and the current individual is compared through the selection operation, and the individual with better fitness is selected as the member of the next generation population. This process is repeated until the termination condition is met, and the optimized neural network parameters are finally obtained.

[0091] The training process of the fusion evaluation model comprises:

[0092] Obtain a training data set; the training data set comprises training data and corresponding label information; the training data comprises numerical information, text information and picture information collected in equipment system evaluation, and the label information is the evaluation result value of the equipment system corresponding to the training data;

[0093] Initialize the training iteration number value;

[0094] Input the training data in the training data set as input data into the fusion evaluation model;

[0095] Process the input data using the fusion evaluation model to obtain a predicted value; the predicted value is the fusion evaluation result value of the equipment system evaluation;

[0096] Differential calculation is performed on the predicted value and the label information corresponding to the input data to obtain a difference value;

[0097] Determine whether the difference value meets the convergence condition to obtain a first determination result;

[0098] When the first determination result is no, determine whether the training iteration number value is equal to the training number threshold to obtain a second determination result;

[0099] When the second determination result is no, it is determined that the model training state does not meet the termination training condition;

[0100] When the second determination result is yes, it is determined that the model training state meets the termination training condition;

[0101] When the first determination result is yes, it is determined that the model training state meets the termination training condition;

[0102] When the model training state is not satisfied with the termination training condition, the parameter updating model is used to update the parameters of the fusion evaluation model, the training iteration number value is increased by 1, and the training data in the training data set is input into the fusion evaluation model as input data.

[0103] When the model training state is satisfied with the termination training condition, the training process of the fusion evaluation model is completed, and a trained fusion evaluation model is obtained.

[0104] The training data set can be obtained according to historical data of equipment system evaluation; the historical data includes test information and corresponding evaluation result information.

[0105] The difference value satisfies the convergence condition, that is, the difference value is less than a preset convergence threshold; the difference value does not satisfy the convergence condition, that is, the difference value is not less than the preset convergence threshold.

[0106] The difference calculation process can be realized by using a loss function.

[0107] The loss function can be a cross-entropy loss function.

[0108] The parameter updating model has a calculation expression as follows:

[0109]

[0110] Wherein, θ represents the parameters of the fusion evaluation model, l1 represents the update step of the fusion evaluation model, 0 < l1 < 1, represents the gradient of θ parameters in L1(θ), L1(θ) represents the loss function of the fusion evaluation model, represents the parameter update value of the fusion evaluation model, represents using to update the parameters θ of the fusion evaluation model;

[0111] The test text information and test picture information are digital information obtained by numerical conversion of collected text or picture.

[0112] The data attribute includes text value, numerical value, and picture value.

[0113] The format detection process includes: judging whether the format of each data sequence in the first data set is consistent with the preset format of the data sequence, and deleting inconsistent data from the data sequence.

[0114] The data cleaning process includes filling in missing values, smoothing noise data, smoothing or deleting outliers; the smoothing noise data is first identified as noise data, and then smoothed according to the data before and after the noise data; the noise data is a value less than the detection sensitivity of the sensor of the observation data or greater than the measurement upper limit of the sensor of the observation data. The identification of the outlier point can use the Kalman filter method. For the determination of the filling value of the missing value, the measurement values in a certain sampling interval before and after the missing value are averaged.

[0115] The loss function L1(φ) has a calculation expression as follows:

[0116]

[0117] Wherein, represents the output of the fusion evaluation model after the training data s in the training data set is input into the fusion evaluation model, V(s) represents the label value corresponding to the training data s in the training data set, represents the second order norm value of all s corresponding to the sampling data set The second order norm value of is taken as the τ quantile, τ is a preset quantile value, 0<τ<1. Specifically,

[0118] The fusion evaluation result value of the equipment system evaluation is obtained by using the trained fusion evaluation model to process the preprocessed data set, including:

[0119] The fusion evaluation result value of the time is obtained by using the trained fusion evaluation model to process the preprocessed data set of each time.

[0120] The fusion evaluation result values of all times are fused and calculated to obtain the fusion evaluation result value of the equipment system evaluation.

[0121] The expression of the fusion calculation processing is as follows:

[0122]

[0123] Wherein, rhy is the fusion evaluation result value of the equipment system evaluation, T2() is the second order Legendre polynomial, y i is the i-th time, y0 is the average value of the fusion evaluation result values of all times, and N1 is the total number of the fusion evaluation result values.

[0124] The expression is transformed by means of a second-order Legendre polynomial T2() to the relative difference between the fusion evaluation result value and the average value at each time. The Legendre polynomial can effectively capture the high-order relationship between data, compared with simple linear calculation, it can dig out more complex and subtle features in the data, so as to more carefully reflect the internal relationship and change rule of the equipment system performance at different times, avoid the loss of data characteristics caused by linear calculation, and make the evaluation result more reflect the real situation of the equipment system operation.

[0125] The second aspect of the present application discloses a multi-modal test data fusion evaluation processing device, which comprises:

[0126] a memory storing executable program codes;

[0127] a processor coupled with the memory;

[0128] The processor calls the executable program codes stored in the memory to execute the multi-modal test data fusion evaluation processing method.

[0129] The third aspect of the present application discloses a computer storage medium, which stores computer instructions, and the computer instructions are called by a computer to execute the multi-modal test data fusion evaluation processing method.

[0130] The fourth aspect of the present application discloses an information data processing terminal, which is used to realize the multi-modal test data fusion evaluation processing method.

[0131] The above is only an embodiment of the present application and is not used to limit the present application. The present application can have various changes and variations for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for fusing and evaluating multimodal experimental data, characterized in that, include: S1, Collect a multimodal test data set; the multimodal test data set includes a set of test numerical information sequences, a set of test text information sequences, and a set of test image information sequences collected during the equipment system evaluation process; the set of test numerical information sequences includes test numerical information sequences; the set of test text information sequences includes test text information sequences. The set of test image information sequences includes test image information sequences; S2, preprocess the multimodal test data set to obtain a preprocessed data set, including: S21, perform data cleaning processing on the multimodal test data set to obtain a first data set; S22, perform format detection processing on the first data set to obtain the second data set; S23, perform data reduction processing on the second dataset to obtain a preprocessed dataset, including: S231, For each type of data attribute in the second data set, using the data collection information of the data as the independent variable and the data value of the data as the dependent variable, perform autoregressive-moving average modeling to obtain the first approximation model of the data attribute. S232, using all the information sequences of the information sequence set of each type of data attribute as row vectors, construct the information matrix of the data attribute of the type; S233, Perform singular value calculation on the information matrix to obtain a singular value sequence; S234, using the element values ​​of the singular value sequence as known dependent variables and the element indices of the singular value sequence as known independent variables, a curve to be approximated is constructed using the known independent variables and the known dependent variables; polynomial fitting is performed on the curve to be approximated to obtain a second approximation model of the class data attribute; S235, Multiply the second approximation model and the first approximation model to obtain the fusion verification model of the class data attributes; S236, Using the fusion test model for each type of data attribute, the data collection information of the data of the data attribute is calculated and processed to obtain an approximate dependent variable; S237, determine whether the absolute value of the difference between the approximate dependent variable and the corresponding data is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, delete the data from the second data set; if it is less than or equal to the first regression discrimination threshold, do not process the data. S238, perform fusion processing on all data in the second data set after executing S236 to S237 to obtain a preprocessed dataset; S3. Using the trained fusion evaluation model, the preprocessed data set is processed to obtain the fusion evaluation result value of the equipment system evaluation.

2. The method for fusing and evaluating multimodal test data as described in claim 1, characterized in that, The fusion evaluation model includes: a first feature extraction network, a second feature extraction network, and a feature fusion network; The first feature extraction network includes a first input module, a first convolution module, a depthwise separable convolution module, a first up-dimensional convolution module, a second up-dimensional convolution module, a third up-dimensional convolution module, a fourth up-dimensional convolution module, a second convolution module, a first pooling module, a third convolution module, and a first fully connected module; The input terminal of the first input module of the first feature extraction network is used to receive the numerical information sequence; the output terminal of the first input module of the first feature extraction network is connected to the input terminal of the first convolution module of the first feature extraction network; the output terminal of the first convolution module of the first feature extraction network is connected to the input terminal of the depthwise separable convolution module of the first feature extraction network; the output terminal of the depthwise separable convolution module of the first feature extraction network is connected to the input terminal of the first up-dimensional convolution module of the first feature extraction network; the output terminal of the first up-dimensional convolution module of the first feature extraction network is connected to the input terminal of the second up-dimensional convolution module of the first feature extraction network; the output terminal of the second up-dimensional convolution module of the first feature extraction network is connected to the input terminal of the third up-dimensional convolution module of the first feature extraction network. The connections are as follows: the output of the third up-dimensional convolutional module of the first feature extraction network is connected to the input of the fourth up-dimensional convolutional module of the first feature extraction network; the output of the fourth up-dimensional convolutional module of the first feature extraction network is connected to the input of the second convolutional module of the first feature extraction network; the output of the second convolutional module of the first feature extraction network is connected to the input of the first pooling module of the first feature extraction network; the output of the first pooling module of the first feature extraction network is connected to the input of the third convolutional module of the first feature extraction network; the output of the third convolutional module of the first feature extraction network is connected to the input of the first fully connected module of the first feature extraction network; the output of the first fully connected module of the first feature extraction network is used to output the feature information of the numerical information sequence.

3. The method for fusing and evaluating multimodal test data as described in claim 2, characterized in that, The second feature extraction network includes a second input module, a fourth convolutional module, a fifth convolutional module, a sixth convolutional module, a second pooling module, a seventh convolutional module, a second fully connected module, and a third fully connected module; The first input terminal of the second input module of the second feature extraction network is used to receive obtained text information; the second input terminal of the second input module of the second feature extraction network is connected to the output terminal of the second dimensionality-upgrading convolution module of the first feature extraction network; the output terminal of the second input module of the second feature extraction network is connected to the input terminal of the fourth convolution module of the second feature extraction network; the output terminal of the fourth convolution module of the second feature extraction network is connected to the input terminal of the fifth convolution module of the second feature extraction network; the output terminal of the fifth convolution module of the second feature extraction network is connected to the input terminal of the sixth convolution module of the second feature extraction network; The output of the sixth convolutional module of the second feature extraction network is connected to the input of the second pooling module of the second feature extraction network; the output of the second pooling module of the second feature extraction network is connected to the input of the seventh convolutional module of the second feature extraction network; the output of the seventh convolutional module of the second feature extraction network is connected to the input of the second fully connected module of the second feature extraction network; the output of the second fully connected module of the second feature extraction network is connected to the input of the third fully connected module of the second feature extraction network; the output of the second fully connected module of the second feature extraction network is used to output the feature information of the text information sequence. The feature fusion network consists of a difference module, a cross-scale fusion attention module, an eighth convolution module, an upsampling module, and a ninth convolution module. The output of the cross-scale fusion attention module is connected to the input of the eighth convolution module, the output of the upsampling module is connected to the input of the ninth convolution module, the output of the eighth convolution module is connected to the first input of the difference module, and the output of the ninth convolution module is connected to the second input of the difference module. The first and second input terminals of the feature fusion network are respectively the input terminals of the attention module and the upsampling module of the cross-scale fusion; the input terminal of the attention module of the cross-scale fusion is connected to the output terminal of the first fully connected module of the first feature extraction network and the output terminal of the second fully connected module of the second feature extraction network. The input terminal of the upsampling module is used to receive image information; the output terminal of the feature fusion network is the output terminal of the difference module. The output of the differential module is used to output the fusion evaluation result value of the equipment system evaluation.

4. The method for fusing and evaluating multimodal test data as described in claim 3, characterized in that, The training process of the fusion evaluation model includes: Obtain a training dataset; the training dataset includes training data and corresponding label information; the training data includes numerical information, text information and image information collected in the equipment system evaluation, and the label information is the evaluation result value of the equipment system corresponding to the training data; Initialize the number of training iterations; The training data in the training dataset is used as input data and input into the fusion evaluation model; The input data is processed using the fusion evaluation model to obtain a predicted value; the predicted value is the fusion evaluation result value of the equipment system evaluation. The difference between the predicted value and the label information corresponding to the input data is calculated to obtain the difference value. Determine whether the difference value satisfies the convergence condition to obtain the first determination result; When the first judgment result is negative, it is determined whether the training iteration count value is equal to the training count threshold to obtain the second judgment result; When the second judgment result is negative, the model training state is determined to be that the termination training condition is not met. When the second judgment result is yes, it is determined that the model training state meets the termination training condition; When the first judgment result is yes, it is determined that the model training state meets the termination training condition; When the training state of the model does not meet the termination training condition, the parameters of the fusion evaluation model are updated using the parameter update model, the number of training iterations is increased by 1, and the training data in the training dataset is used as input data to the fusion evaluation model. When the model training state meets the termination training condition, the training process of the fusion evaluation model is completed, and the trained fusion evaluation model is obtained.

5. The method for fusing and evaluating multimodal test data as described in claim 4, characterized in that, The parameter update model is calculated using the following expression: Where θ represents the parameters of the fusion evaluation model, l1 represents the update step size of the fusion evaluation model, and 0 <l1<1, This indicates that the gradient of the θ parameter in L1(θ) is calculated, where L1(θ) represents the loss function of the fusion evaluation model. This represents the updated parameter values ​​of the fusion evaluation model. Indicates the use of The parameters θ of the fusion evaluation model are updated.

6. A fusion evaluation and processing device for multimodal test data, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the fusion evaluation processing method for multimodal test data as described in any one of claims 1 to 5.

7. A computer-storable medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked by the computer, are used to execute the multimodal test data fusion evaluation processing method as described in any one of claims 1 to 5.

8. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the fusion evaluation processing method for multimodal test data as described in any one of claims 1 to 5.

Images