Multi-modal data quality evaluation and optimization method and system of Internet of Vehicles

Abstract

The invention discloses a multi-modal data quality evaluation and optimization method and system for the Internet of Vehicles, and relates to the technical field of data quality evaluation and optimization, and the method comprises the steps: carrying out the calculation of the preprocessed multi-modal data through five dimension factors, and obtaining a corresponding quality score and a comprehensive score; obtaining multi-modal data needing to be optimized according to the comprehensive score; obtaining all quality scores and corresponding weights of the multi-modal data needing to be optimized, inputting the quality scores and the corresponding weights into a factor perception optimization network, outputting an optimization strategy suggestion value, and calculating corresponding quality scores of the optimized multi-modal data and the non-optimized multi-modal data through five dimension factors after optimization; and judging whether a convergence condition is met or not according to the comprehensive score variation and the marginal income. According to the method, efficient evaluation and closed-loop optimization of the Internet of Vehicles data set are realized through quality factor calculation, dynamic weighted scoring, and data completion and convergence control strategy of strategy perception optimization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data quality evaluation and optimization, in particular to a multi-modal data quality evaluation and optimization method and system for Internet of Vehicles. BACKGROUND

[0002] With the rapid development of Internet of Vehicles technology, massive perception data has become the core resource for training perception and decision-making models, and its quality directly determines the upper limit of model performance. Especially in complex environments such as low light, occlusion, and extreme weather, low-quality data can seriously interfere with model training and deployment performance. Therefore, it is particularly important to evaluate and optimize data for Internet of Vehicles to obtain high-quality data.

[0003] In existing general data quality evaluation and optimization methods, most of them are for single modal or static rules. Such data quality evaluation and optimization methods lack unified modeling of multi-modal data quality evaluation, and are difficult to apply to collaborative quality judgment of multi-modal data such as images, point clouds, and labels in Internet of Vehicles scenarios, and cannot comprehensively evaluate the overall quality of multi-modal data, resulting in inaccurate data evaluation results. In the existing scoring method, most of the weights are fixed values or subjectively set, but research has found that different data stages and collection tasks have different quality factor concerns. The traditional scoring method cannot dynamically adjust the weights and cannot adapt to changes.

[0004] And in the existing optimization scheme, most of them rely on empirical threshold judgment, the optimization strategy is fixed, it is difficult to combine, and it cannot be personalized according to the real defects of the sample. In addition, the existing optimization process lacks a feedback judgment mechanism, which is prone to problems such as repeated optimization, invalid enhancement, and resource waste. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a multi-modal data quality evaluation and optimization method and system for Internet of Vehicles. The present application realizes efficient evaluation and closed-loop optimization of Internet of Vehicles data sets through quality factor calculation, dynamic weighted scoring, and data completion and convergence control strategy based on strategy perception optimization.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: In a first aspect, the present application provides a multi-modal data quality evaluation and optimization method for Internet of Vehicles, comprising: acquiring multi-modal data and pre-processing the multi-modal data; calculating the corresponding quality scores of the pre-processed multi-modal data through five dimension factors respectively, and weighting and fusing the quality scores with the corresponding weights to obtain a comprehensive score, and obtaining multi-modal data that needs to be optimized according to the comprehensive score; All quality scores of the multi-modal data to be optimized and corresponding weights are acquired and spliced into a joint input vector, which is input into the trained factor-aware optimization network to output an optimization strategy suggestion value, the multi-modal data to be optimized is optimized according to the optimization strategy suggestion value, and the quality scores of the multi-modal data before and after optimization are calculated again through the five dimension factors respectively. The comprehensive scores before and after optimization are acquired to obtain a comprehensive score change and a marginal benefit, and it is judged whether a convergence condition is met, if the convergence condition is met, the optimization is completed, otherwise the optimization is continued.

[0007] As a further technical solution, the multi-modal data includes image data, laser radar point cloud, motion state data and inertial navigation data; the pre-processing method is: first, time synchronization and format conversion are performed on the multi-modal data, and unified into a standard data structure, then abnormal frames, empty frames and invalid labeled multi-modal data are removed, and finally meta information labels are extracted and attached.

[0008] As a further technical solution, the five dimension factors include integrity factor, diversity factor, robustness factor, consistency factor and complexity factor.

[0009] As a further technical solution, the weight calculation method is: based on the score fluctuation of the historical quality score, the dimension weight is calculated, and the specific method is: ; wherein, represents the weight corresponding to the th dimension, represents the score standard deviation of the th dimension.

[0010] As a further technical solution, the comprehensive score calculation method is: ; wherein, represents the quality score of the th dimension, represents the weight corresponding to the th dimension.

[0011] As a further technical solution, the quality score of the multi-modal data to be optimized calculated through the five dimension factors is acquired to generate a quality score vector, which is represented as: The weight corresponding to the quality score is acquired to generate a weight vector, which is represented as: The quality score vector and the weight vector are spliced into a joint input vector, which is input into the trained factor-aware optimization network to output an optimization strategy suggestion value, the optimization strategy suggestion value includes supplementing missing samples, data enhancement, noise enhancement, time-space alignment verification and class balance sampling.

[0012] As a further technical solution, the convergence condition is represented as: , ; wherein, represents the change in the comprehensive score before and after optimization, represents the change threshold, represents the marginal benefit, represents the marginal benefit threshold; wherein , ; represents the current comprehensive score, represents the previous round comprehensive score.

[0013] In a second aspect, the application provides a multi-modal data quality evaluation and optimization system for Internet of Vehicles, comprising the following modules: A data acquisition module configured to acquire multi-modal data and pre-process the multi-modal data; A quality factor calculation module configured to calculate corresponding quality scores of the pre-processed multi-modal data through five dimension factors respectively, and to obtain a comprehensive score by weighted fusion of the quality scores and corresponding weights, and to obtain multi-modal data to be optimized according to the comprehensive score; A multi-round optimization module configured to obtain all quality scores of the multi-modal data to be optimized and corresponding weights, and to splice them into a joint input vector, and to input the joint input vector into a trained factor-aware optimization network to output an optimization strategy suggestion value, and to optimize the multi-modal data to be optimized according to the optimization strategy suggestion value, and to calculate corresponding quality scores through the five dimension factors again according to the optimized multi-modal data and the unoptimized multi-modal data; A convergence control and feedback module configured to obtain the comprehensive score before and after optimization, to obtain a comprehensive score change and a marginal benefit, and to judge whether the convergence condition is met, and if the convergence condition is met, to complete the optimization, otherwise to continue the cyclic optimization.

[0014] The one or more technical solutions of the application have the following beneficial effects: (1) To solve the problem of lack of unified modeling in multi-modal data quality evaluation, the application calculates corresponding quality scores of the multi-modal data through five dimension factors (integrity factor, diversity factor, robustness factor, consistency factor and complexity factor) respectively. The multi-modal data is converted into unified quantitative indicators, realizing collaborative evaluation and unified modeling of multi-modal data quality, and thus improving the accuracy of data evaluation results.

[0015] (2) The application obtains a comprehensive score by weighting and fusing the quality score and the corresponding weight, so that the response to the key quality dimension in different stages is more sensitive, the dynamic adjustment of the weight is realized, the defects caused by the fixed weight or subjective setting of the weight are avoided, and the self-adaptive ability is improved.

[0016] (3) The application inputs the quality score and the corresponding weight into the trained factor perception optimization network, outputs an optimization strategy suggestion value, avoids relying on experience threshold judgment and fixed optimization strategy, and formulates a personalized optimization strategy according to the real defect of the sample.

[0017] (4) The application calculates the comprehensive score change and marginal benefit, formulates a convergence feedback judgment mechanism, and avoids problems such as repeated optimization, invalid enhancement and resource waste. BRIEF DESCRIPTION OF DRAWINGS

[0018] The drawings accompanying the specification of the application form part of the application and serve to provide further understanding of the application, and the illustrative embodiments of the application and the description thereof serve to explain the application, and do not constitute an improper limitation of the application.

[0019] Figure 1 A flowchart of the multi-modal data quality evaluation and optimization method of the application; Figure 2 A framework diagram of the multi-modal data quality evaluation and optimization system of the application. DETAILED DESCRIPTION

[0020] Embodiment one In this embodiment, a multi-modal data quality evaluation and optimization method for Internet of Vehicles is provided, as shown in Figure 1 The specific method steps are as follows: S1: Obtain multi-modal data and pre-process the multi-modal data.

[0021] In step S1, the multi-modal data obtained through the vehicle-mounted camera, laser radar, millimeter wave radar, GPS and IMU includes image data, laser radar point cloud, motion state data and inertial navigation data. In order to standardize the multi-modal raw data obtained by the vehicle of the Internet of Vehicles, ensure the format uniformity and structure specification of the system input data, in this embodiment, the pre-processing method is as follows: first, time synchronization and format conversion are performed on the multi-modal data, and the data of different sources are aligned to a unified data structure; then, the multi-modal data is cleaned, including: removing abnormal frames, empty frames and invalid labeled multi-modal data; finally, meta information labels are extracted and attached, the meta information labels include weather type, time period, road environment and the like, and provide context information support for subsequent quality evaluation.

[0022] S2: The pre-processed multi-modal data is calculated by five dimension factors respectively to obtain corresponding quality scores, and the quality scores are weighted and fused with corresponding weights to obtain a comprehensive score, and the multi-modal data to be optimized is obtained according to the comprehensive score.

[0023] In step S2, for each data, the corresponding quality score is calculated by five dimension factors respectively, wherein the five dimension factors include integrity factor, diversity factor, robustness factor, consistency factor and complexity factor.

[0024] In this embodiment, the integrity factor is used to evaluate the missing situation of each modality data and label, and reflects the completeness of the data. The calculation method is as follows: ; wherein, represents the number of missing modalities or labels, represents the total sample. The closer the integrity factor is to 1, the more complete the data is.

[0025] The diversity factor is used to measure the distribution balance of different scenes (such as day / night, city / country), modality combination or label category, and reflects the breadth of semantic coverage. The calculation method is as follows: ; wherein, represents a semantic category, such as different scenes or label combinations, represents the proportion of each sample. The higher the diversity factor, the richer the scene type.

[0026] The robustness factor is used to measure the influence of data on model performance when subjected to disturbance (such as blur, occlusion, noise or modality loss), and reflects the anti-interference ability of data. The calculation method is as follows: feature extraction is performed on each sample, and the sample feature before disturbance is recorded as , the corresponding sample feature after disturbance is , wherein j=1, 2, 3,... , the distance between the features before and after disturbance of each sample is calculated as : = , set the maximum distance of the feature space for normalization, then the calculation formula of the robustness factor is: ; the closer the robustness factor score is to 1, the lower the sensitivity of the data to disturbance, and the stronger the robustness.

[0027] The consistency factor is used to measure the expression consistency between multi-modal data or multi-label results, and reflects the coordination and reliability between data. The calculation method is as follows: ; wherein, represents a sample The total number of samples in two modalities is The higher the consistency factor, the more consistent the multi-modal data is in describing the same semantic target.

[0028] The complexity factor is used to measure the structural complexity of the feature space, reflecting the density change, boundary ambiguity and expression difficulty of the sample distribution. The calculation method is to extract the features of the sample set and calculate the average distance of each sample to its nearest k neighbors , the calculation formula is as follows: ; wherein, ; is the feature vector of the sample; N is the total number of samples. The higher the complexity factor, the more complex the feature distribution.

[0029] According to the above method, the corresponding quality scores are calculated according to the integrity factor, the diversity factor, the robustness factor, the consistency factor and the complexity factor .

[0030] In step S2, the dimension weight is calculated based on the score fluctuation of the historical quality score, and the specific method is: ; wherein, represents the weight corresponding to the th dimension, represents the standard deviation of the score of the th dimension. It is ensured that the integrity and consistency are given priority in the early training, and more attention is paid to the complexity and robustness in the later period. The calculation method of the comprehensive score is: ; wherein, represents the quality score of the th dimension.

[0031] According to the comprehensive score, it can be divided into four categories: High quality (comprehensive score ≥ 90): direct output; Acceptable (comprehensive score 75-89): need to be slightly optimized; Need optimization (comprehensive score 60-74): enter the enhancement processing flow; Not available (comprehensive score < 60): marked as waste or recycling.

[0032] S3: Obtain all quality scores of the multi-modal data to be optimized and the corresponding weights, concatenate them into a joint input vector, input them into the trained factor-aware optimization network, output the optimization strategy suggestion value, optimize the multi-modal data to be optimized according to the optimization strategy suggestion value, and calculate the corresponding quality scores through the five dimension factors again according to the optimized multi-modal data and the unoptimized multi-modal data.

[0033] In step S3, the quality scores of the multi-modal data to be optimized calculated through the five dimension factors are obtained, and a quality score vector is generated, denoted as: The weights corresponding to the quality scores are obtained, and a weight vector is generated, denoted as: The quality score vector and the weight vector are concatenated into a joint input vector , which is input into the trained factor-aware optimization network. The network is a lightweight multi-layer perception structure, which models the interaction mode between the factor scores, and generates a personalized optimization strategy suggestion according to the quality scores and the corresponding weights. The specific structure is as follows: the network input layer is composed of the quality score vector (a total of five dimensions, corresponding to integrity, diversity, robustness, consistency and complexity) and the corresponding weight vector, which are concatenated into a 10-dimensional input vector. The first layer is a fully connected layer with an input dimension of 10 and an output dimension of 64, and the activation function is ReLU. The second layer is a fully connected layer with an input dimension of 64 and an output dimension of 32, and the activation function is ReLU. The third layer is the output layer with a dimension of 5, corresponding to the five optimization strategies (supplementary sampling, enhancement, noise addition, alignment and balanced sampling), and the activation function is Sigmoid, which is used to represent the execution intensity of each strategy (range 0~1). In the data processing process, in the training stage, the factor-aware optimization network takes the quality score vector and the corresponding weight vector of the sample as input, and outputs the recommended intensity of each optimization strategy, with a value range of 0 to 1. In the training process, the sample quality improvement effect of each optimization strategy in the historical data is used as a supervision signal to construct a 5-dimensional continuous label vector, and the mean square error is used as the loss function to train the network to predict the execution intensity of different strategies. In the deployment stage, the quality score and the weight of the current sample are input, and the network can quickly output the recommended execution intensity of the five optimization strategies. The system sorts the output values and selects the strategy with high recommended intensity for sample optimization, thereby realizing a personalized data processing process. The output optimization strategy suggestion value , the output de1 optimization strategy suggestion value Each component in the output optimization strategy suggestion value The strength distribution of each strategy is called, the corresponding optimization module is called, and execution is performed according to the strength distribution. After optimization, the sample data and the optimization action are recorded in the log. The optimization operation changes the sample data state. These optimized multi-modal data, together with other multi-modal data (high-quality, acceptable, and discarded samples) that are not optimized, are again calculated for the corresponding quality scores by the five dimension factors, and the next round of evaluation is started. Subsequent rounds will continue to update the dynamic weights according to the optimized scores and re-enter the strategy analysis until the convergence condition is met.

[0034] S4: Obtain the comprehensive scores before and after optimization, obtain the comprehensive score change and marginal benefit, and determine whether the convergence condition is met. If the convergence condition is met, optimization is completed, otherwise the optimization is continued.

[0035] In step S4, the convergence condition is expressed as follows: , if the two consecutive rounds meet, it is determined that the multi-round optimization iteration has converged, and the entire closed-loop optimization process is terminated, and the optimization is completed, wherein, represents the comprehensive score change before and after optimization, represents the change threshold, represents the marginal benefit, represents the marginal benefit threshold; wherein ; represents the current comprehensive score, represents the comprehensive score of the previous round. In this embodiment, = data set sample number x 0.5%, indicating that when the average score change of the full data set is less than 0.5% of the total sample number, the optimization is considered to be saturated. = current score x 1%, indicating the marginal benefit threshold, to avoid excessive optimization in the high score stage.

[0036] If the convergence condition is not met, the next round of evaluation optimization cycle is automatically started.

[0037] Embodiment Two In this embodiment, a multi-modal data quality evaluation and optimization system for Internet of Vehicles is provided, which includes the following modules: A data acquisition module configured to acquire multi-modal data and pre-process the multi-modal data; A quality factor calculation module configured to calculate the corresponding quality scores of the pre-processed multi-modal data by five dimension factors respectively, and to obtain a comprehensive score by weighting and fusing the quality scores and the corresponding weights, and to obtain multi-modal data to be optimized according to the comprehensive score; The multi-round optimization module is configured to: acquire all quality scores of the multi-modal data to be optimized and corresponding weights, splice the quality scores and the weights into a joint input vector, input the joint input vector into the trained factor-aware optimization network, output an optimization strategy suggestion value, optimize the multi-modal data to be optimized according to the optimization strategy suggestion value, and calculate corresponding quality scores through five dimension factors respectively according to the optimized multi-modal data and the unoptimized multi-modal data again. The convergence control and feedback module is configured to: acquire comprehensive scores before and after optimization, obtain a comprehensive score change amount and a marginal benefit, and judge whether a convergence condition is met, if the convergence condition is met, optimization is completed, otherwise, the optimization is continued.

[0038] The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for evaluating and optimizing the quality of multimodal data in an Internet of Vehicles, characterized in that: include: Acquire multimodal data and preprocess the multimodal data; The pre-processed multimodal data is calculated through the five dimensional factors to obtain the corresponding quality score, and the quality score is weighted and fused with the corresponding weight to obtain a comprehensive score. Based on the comprehensive score, the multimodal data to be optimized is obtained; Obtain all quality scores and corresponding weights of the multimodal data to be optimized, concatenate them into a joint input vector, input it into the trained factor-aware optimization network, output the optimization strategy recommendation value, optimize the multimodal data to be optimized based on the optimization strategy recommendation value, and calculate the corresponding quality scores again using the five dimensional factors based on the optimized and unoptimized multimodal data; Obtain the comprehensive scores before and after optimization, obtain the change in comprehensive scores and marginal benefits, and determine whether the convergence conditions are met. If the convergence conditions are met, the optimization is completed, otherwise the optimization cycle continues.

2. The method for evaluating and optimizing the quality of multimodal data of an Internet of Vehicles according to claim 1, wherein: The multimodal data includes image data, lidar point cloud, motion state data and inertial navigation data; the preprocessing method is: first, the multimodal data is time synchronized and format converted to a standard data structure, then abnormal frames, empty frames and invalidly annotated multimodal data are removed, and finally, metadata tags are extracted and attached.

3. The method for evaluating and optimizing multimodal data quality of an Internet of Vehicles according to claim 1, wherein: The five dimensional factors include integrity factor, diversity factor, robustness factor, consistency factor and complexity factor.

4. The method for evaluating and optimizing multimodal data quality of an Internet of Vehicles according to claim 1, wherein: The weight is calculated by calculating the weight of each dimension based on the fluctuation of the historical quality score. The specific method is as follows: ;in, Indicates the The weight corresponding to each dimension, Indicates the The standard deviation of scores for each dimension.

5. The method for evaluating and optimizing the quality of multimodal data of an Internet of Vehicles according to claim 1, wherein: The composite score is calculated as follows: ;in, Indicates the Dimension quality scores, Indicates the The weight corresponding to each dimension.

6. The method for evaluating and optimizing multimodal data quality of an Internet of Vehicles according to claim 1, wherein: Obtain the quality score of the multimodal data to be optimized by calculating the five dimensional factors and generate a quality score vector, which is expressed as: Get the weight corresponding to the quality score and generate a weight vector, which is expressed as: The quality score vector and the weight vector are concatenated into a joint input vector, which is then fed into the trained factor-aware optimization network to output the optimization strategy recommendation. The optimization strategy recommendation includes supplementing missing samples, data augmentation, noise enhancement, spatiotemporal alignment verification, and category balanced sampling.

7. The method for evaluating and optimizing multimodal data quality of an Internet of Vehicles according to claim 1, wherein: The convergence condition is expressed as: , ;in, Indicates the change in comprehensive score before and after optimization, Indicates the change threshold, represents marginal revenue, represents the marginal benefit threshold; where , ; Indicates the current comprehensive score. Indicates the comprehensive score of the previous round.

8. A multimodal data quality assessment and optimization system for Internet of Vehicles, characterized by: Includes the following modules: The data acquisition module is configured to: acquire multimodal data and preprocess the multimodal data; The quality factor calculation module is configured to: calculate the corresponding quality scores of the pre-processed multimodal data through the five dimensional factors, and perform weighted fusion of the quality scores with the corresponding weights to obtain a comprehensive score, and obtain the multimodal data to be optimized based on the comprehensive score; The multi-round optimization module is configured to obtain all quality scores and corresponding weights of the multimodal data to be optimized, concatenate them into a joint input vector, input it into the trained factor-aware optimization network, output an optimization strategy recommendation, optimize the multimodal data to be optimized based on the optimization strategy recommendation, and again calculate the corresponding quality scores based on the five dimensional factors of the optimized and unoptimized multimodal data. The convergence control and feedback module is configured to obtain the comprehensive scores before and after optimization, obtain the change in the comprehensive scores and the marginal benefits, and determine whether the convergence conditions are met. If the convergence conditions are met, the optimization is completed; otherwise, the optimization cycle continues.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the multimodal data quality assessment and optimization method of the Internet of Vehicles as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the multimodal data quality assessment and optimization method of the Internet of Vehicles as described in any one of claims 1 to 7 are implemented.

Images