Automatic driving data processing method and device, storage medium and computer device

Abstract

This application provides an autonomous driving data processing method, apparatus, storage medium, and computer device. The method includes: segmenting an acquired multi-source dataset to generate multiple data subsets; determining the probability distribution value corresponding to at least one data feature on each data subset; generating a robust cross-entropy value corresponding to at least one data feature based on the multiple probability distribution values ​​corresponding to the at least one data feature; and generating a feature subset based on the robust cross-entropy value corresponding to the at least one data feature. This enables the evaluation of feature distribution differences on different datasets using cross-entropy, allowing assessment of which features are more susceptible to changes in conditions, outputting features with significant differences between multi-source datasets, and reducing the risks caused by data differences.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically to an autonomous driving data processing method, apparatus, storage medium, and computer device. Background Technology

[0002] In the field of autonomous driving, the diversity of datasets is an unavoidable reality. Vehicles collect information about their surroundings through sensors such as radar, LiDAR, and cameras. During this information collection process, data may be gathered from different geographical locations, time periods, and weather conditions. Each dataset may have unique distribution characteristics, and the differences in feature distribution across different datasets directly affect the generalization ability of algorithms. This poses numerous challenges to the design, testing, and deployment of autonomous driving systems. However, related technologies do not provide a scheme for evaluating the differences in feature distribution across different datasets, increasing the risks arising from data discrepancies. Summary of the Invention

[0003] In view of this, this application provides an autonomous driving data processing method, apparatus, storage medium, and computer device to address the problem in related technologies that there is no scheme for accurately assessing the differences in feature distributions of different datasets, which increases the risks caused by data differences.

[0004] In a first aspect, embodiments of this application provide an autonomous driving data processing method, including:

[0005] The acquired multi-source dataset is segmented to generate multiple data subsets;

[0006] Determine the probability distribution value of at least one data feature on each subset of data;

[0007] Generate a robust cross-entropy value corresponding to at least one of the data features based on multiple probability distribution values ​​corresponding to at least one of the data features;

[0008] A feature subset is generated based on the robust cross-entropy value corresponding to at least one of the data features.

[0009] In one possible implementation, generating a robust cross-entropy value corresponding to at least one of the data features based on multiple probability distribution values ​​corresponding to at least one of the data features includes:

[0010] A test set and at least one training set are determined from multiple data subsets, and a preliminary cross-entropy value of the data feature on each training set is generated based on the probability distribution value of the data feature on the test set and the probability distribution value of the data feature on each training set, until each data subset has appeared as a test set;

[0011] Based on the multiple preliminary cross-entropy values ​​corresponding to the data features, a robust cross-entropy value corresponding to the data features is generated.

[0012] In one possible implementation, determining a test set and at least one training set from multiple data subsets, and generating a preliminary cross-entropy value for the data feature on each training set based on the probability distribution value of the data feature on the test set and the probability distribution value of the data feature on each training set, includes:

[0013] A test set and at least one training set are determined from the multiple subsets of data;

[0014] The probability distribution value of the data feature in the test set is used as the predicted probability distribution value, and the probability distribution value of the data feature in the training set is used as the baseline probability distribution value.

[0015] Using the cross-entropy calculation formula, at least one preliminary cross-entropy value corresponding to the baseline probability distribution value is generated based on at least one baseline probability distribution value corresponding to the data feature and the predicted probability distribution value.

[0016] In one possible implementation, the robust cross-entropy value includes at least one of the following: average cross-entropy value, standard deviation cross-entropy value, median cross-entropy value, quartile cross-entropy value, extreme value cross-entropy value, and confidence interval cross-entropy value.

[0017] In one possible implementation, the segmentation of the acquired multi-source dataset to generate multiple data subsets includes:

[0018] The multi-source dataset is preprocessed to generate a standardized multi-source dataset;

[0019] The standardized multi-source dataset is segmented to generate multiple data subsets.

[0020] In one possible implementation, the segmentation of the acquired multi-source dataset to generate multiple data subsets includes:

[0021] The multi-source dataset is divided into multiple subsets of the same size by average splitting.

[0022] In one possible implementation, the preprocessing includes at least one of checking data integrity, standardizing data format, standardizing image format, and standardizing data range.

[0023] In one possible implementation, generating a feature subset based on the robust cross-entropy value corresponding to at least one of the data features includes:

[0024] Based on the robust cross-entropy value corresponding to at least one of the data features, at least one of the data features is filtered to obtain at least one target data feature;

[0025] The feature subset is generated based on at least one of the target data features.

[0026] In one possible implementation, when the number of data features is equal to 1, the step of filtering at least one data feature based on the robust cross-entropy value corresponding to at least one data feature to obtain at least one target data feature includes:

[0027] Based on the robust cross-entropy and the cross-entropy threshold corresponding to the robust cross-entropy, the data feature is determined as the target data feature.

[0028] In one possible implementation, when the number of data features is greater than 1, the step of filtering at least one data feature based on the robust cross-entropy value corresponding to at least one data feature to obtain at least one target data feature includes:

[0029] Based on the robust cross-entropy values ​​corresponding to multiple data features, the multiple data features are sorted to generate a feature sorting result;

[0030] The feature ranking results are subjected to correlation exclusion filtering to obtain at least one target data feature.

[0031] In one possible implementation, the relevance exclusion filtering process performed on the feature ranking results to obtain at least one target data feature includes:

[0032] The relationship of the feature ranking results is evaluated by statistical methods to obtain at least one feature pair. The data features other than the target data feature in the feature pair are removed. The target data feature is the data feature corresponding to the largest robust cross-entropy value in the feature pair.

[0033] And / or, by using redundancy reduction techniques, at least one redundant feature is identified from the feature sorting results; the redundant feature in the feature sorting results is removed, and the data feature other than the redundant feature in the feature sorting results is taken as the target data feature.

[0034] In one possible implementation, the data features include discrete features, and determining the probability distribution value corresponding to at least one data feature on each subset of data includes:

[0035] The probability distribution value corresponding to the discrete feature on each subset of data is determined.

[0036] In one possible implementation, the data features include continuous features, and the probability distribution values ​​include continuous probability distribution values; determining the probability distribution value corresponding to at least one data feature on each subset of data includes:

[0037] The continuous features are segmented to generate continuous segmentation features;

[0038] The continuous probability distribution value corresponding to the continuous segmentation feature on each of the data subsets is determined.

[0039] Secondly, embodiments of this application provide an autonomous driving data processing device, comprising:

[0040] The first generation module is used to segment the acquired multi-source dataset and generate multiple data subsets;

[0041] A determining module is used to determine the probability distribution value corresponding to at least one data feature on each subset of data;

[0042] The second generation module is used to generate at least one robust cross-entropy value corresponding to at least one of the data features based on multiple probability distribution values ​​corresponding to at least one of the data features.

[0043] The third generation module is used to generate a feature subset based on the robust cross-entropy value corresponding to at least one of the data features.

[0044] Thirdly, embodiments of this application provide a computer device, including a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the computer device is triggered to execute the method described in any of the first aspects above.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the method described in any of the first aspects.

[0046] This invention provides an autonomous driving data processing method, apparatus, storage medium, and computer device. The method involves segmenting an acquired multi-source dataset to generate multiple data subsets; determining the probability distribution value corresponding to at least one data feature in each data subset; generating a robust cross-entropy value corresponding to at least one data feature based on the multiple probability distribution values; and generating a feature subset based on the robust cross-entropy value corresponding to at least one data feature. This allows for the evaluation of feature distribution differences across different datasets using cross-entropy, identifying which features are more susceptible to changes in conditions, and outputting features that are significantly different between the multi-source datasets, thereby reducing the risks associated with data differences. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating an autonomous driving data processing method provided in this application embodiment;

[0049] Figure 2 A schematic diagram of a statistical verification process timeline provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of an autonomous driving data processing device provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0052] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0053] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0054] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0055] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0056] In related technologies, firstly, the differences in feature distribution among different datasets can affect the generalization ability of algorithms. For example, a model trained on traffic sign data collected under sunny weather conditions in a certain region may have difficulty accurately identifying the same signs in rainy or foggy weather in other regions. Changes in feature distribution reduce the recognition performance of the model, seriously threatening driving safety.

[0057] Secondly, for vehicles, autonomous driving needs to handle various unexpected situations and usually relies on robust predictive models. If the model performs well only under specific types of road or traffic conditions but fails when faced with new or unseen scenarios, such as navigation failure in complex urban environments, it will threaten driving safety.

[0058] Furthermore, the control system of an autonomous vehicle must be able to make timely decisions based on data received from sensors. If sensors drift due to inconsistent calibration on different devices or the effects of day-night temperature differences, they may generate misleading data, leading to incorrect operating commands.

[0059] Furthermore, the development of autonomous driving requires a large amount of simulated test data, which relies on data generated in a virtual environment. If the simulated test data cannot accurately reflect the characteristics and distribution of the real world, then the developed autonomous driving system may not be able to adapt to real road conditions.

[0060] Finally, to improve the robustness of autonomous driving systems, it is often necessary to integrate datasets from different sources into a unified framework for training and testing. Inconsistent distributions make dataset integration even more complex.

[0061] Therefore, in the field of autonomous driving, accurately assessing the differences in feature distributions of different datasets and finding effective fusion methods are crucial steps to ensure algorithm performance and system safety.

[0062] To assess the differences in feature distributions across different datasets and reduce the risks associated with data discrepancies. Figure 1 A flowchart of an autonomous driving data processing method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0063] Step 101: The computer device segments the acquired multi-source dataset to generate multiple data subsets.

[0064] In some embodiments, the computer device includes, but is not limited to, a server with computing capabilities.

[0065] A multi-source dataset comprises multiple datasets, with the number of datasets being greater than or equal to two. For example, a multi-source dataset might include multi-sensor data collected by vehicles under different scenarios (such as city streets, suburban roads, and highways) and different weather conditions (such as sunny days, rainy days, and foggy days). Computer equipment collects data samples, gathering multi-sensor data from vehicles at various predetermined locations within a specified time period. This multi-sensor data includes at least one of the following: vehicle operating status information and external environmental information. Vehicle operating status information includes speed, acceleration, steering wheel angle, etc., while external environmental information includes camera images, radar echoes, and lidar point cloud data, etc.

[0066] The process of dividing a multi-source dataset into k subsets is called "folding," where k is greater than or equal to 2. Computer devices can partition a multi-source dataset equally according to its size, resulting in subsets of the same size. For example, if a multi-source dataset includes dataset A and dataset B, the computer device can partition dataset A and dataset B as a whole, generating subset A and subset B; alternatively, the computer device can partition dataset A and dataset B separately, and then arbitrarily combine the partitioned datasets A and B to obtain subset A and subset B.

[0067] The data can be randomly segmented according to the size of the dataset, or it can be segmented according to the data content, or it can be segmented manually by the user, or it can be segmented according to the user's needs, etc. This embodiment of the invention does not limit the operation of segmenting multi-source datasets.

[0068] Step 102: The computer device determines the probability distribution value corresponding to at least one data feature on each subset of data.

[0069] In some embodiments, taking a multi-source dataset collected from a vehicle as an example, data features include speed V (km / h), acceleration A (m / s²), steering wheel angle, camera image edge features, camera image color histogram, or camera image brightness histogram LH (20 bins), etc. For example, a computer device determines the probability distribution values ​​of speed on data subset A, speed on data subset B, acceleration on data subset A, and acceleration on data subset B, etc.

[0070] Step 103: The computer device generates a robust cross-entropy value corresponding to at least one data feature based on multiple probability distribution values ​​corresponding to at least one data feature.

[0071] In some embodiments, cross-entropy originates from the concept of entropy in information theory, originally used in coding theory to estimate the average code length. In machine learning, cross-entropy is widely used in loss functions, particularly in classification problems, to measure the similarity between the model's predicted probability distribution and the probability distribution of the true labels. In this embodiment of the invention, cross-entropy is a measure of the difference between two probability distributions.

[0072] The computer device generates a robust cross-entropy value corresponding to each data feature based on multiple probability distribution values ​​corresponding to that data feature. For example, the computer device generates a robust cross-entropy value for speed based on the probability distribution values ​​of speed on data subset A and data subset B. Similarly, it can generate robust cross-entropy values ​​for data features such as acceleration, steering wheel angle, camera image edge features, or camera image brightness histogram.

[0073] Step 104: The computer device generates a feature subset based on the robust cross-entropy value corresponding to at least one data feature.

[0074] In some embodiments, a computer device evaluates and selects at least one data feature based on the robust cross-entropy value corresponding to at least one data feature, thereby obtaining a set of features with the strongest discriminative power. The feature list is used as a feature subset, and the data features in the feature subset have the highest discriminative efficiency among different datasets.

[0075] This invention provides an autonomous driving data processing method that segments an acquired multi-source dataset to generate multiple data subsets; determines the probability distribution value corresponding to at least one data feature on each data subset; generates a robust cross-entropy value corresponding to at least one data feature based on the multiple probability distribution values ​​corresponding to the at least one data feature; and generates a feature subset based on the robust cross-entropy value corresponding to the at least one data feature. This allows for the evaluation of feature distribution differences across different datasets using cross-entropy, identifying which features are more susceptible to changes in conditions, and outputting features that are significantly different between the multi-source datasets, thereby reducing the risks associated with data differences.

[0076] In one possible implementation, step 101 is preceded by: a computer device acquiring a multi-source dataset.

[0077] In some embodiments, multiple datasets may come from different data sources, including data from different traffic roads, different weather conditions, etc. For example, data source A is data collected on city streets on sunny days, and data source B is data collected on highways on rainy days. The computer device randomly selects 1,000 records at 1000 time points from data source A as dataset A, and randomly selects 1,000 records at 1000 time points from data source B as dataset B.

[0078] In one possible implementation, step 101 may specifically include: a computer device preprocessing the multi-source dataset to generate a standardized multi-source dataset; and segmenting the standardized multi-source dataset to generate multiple data subsets.

[0079] In some embodiments, preprocessing includes at least one of checking data integrity, standardizing data format, standardizing image format, and standardizing data range. Checking data integrity includes removing damaged or missing records; standardizing data format includes converting all timestamps to a uniform format; standardizing image format includes cropping camera images, scaling them to the same resolution, and color normalizing them; standardizing data range includes dimensional consistency and range standardization of sensor values ​​(such as velocity, acceleration, etc.). A common approach is average partitioning, where the computer device performs average partitioning on the standardized multi-source dataset to generate multiple data subsets of equal size. For example, the multi-source dataset can be treated as a whole and averaged into k equal-sized data subsets; alternatively, each dataset can be divided into k equal parts, and one part of each dataset can be combined to obtain k equal-sized data subsets.

[0080] For example, ensure that timestamps are converted to Coordinated Universal Time (UTC) standard time format; scale images to 256*256 pixels, which can be obtained later when calculating the brightness histogram by calculating the frequency of brightness for each pixel; remove outliers from velocity and acceleration data, and normalize velocity and acceleration to the [0,1] interval by subtracting the minimum value from the maximum value and dividing by the range. After the computer device performs standardization preprocessing on multiple datasets, it then splits the multiple datasets to generate multiple data subsets.

[0081] Alternatively, after step 101, the computer device can perform a standardization step, preprocessing multiple data subsets to generate multiple standardized data subsets. This preprocessing calibrates the differences between different data sources, standardizes the data, and ensures data consistency.

[0082] In one possible implementation, the data features include discrete features; step 102 may specifically include: the computer device determining the probability distribution value corresponding to the discrete features on each data subset; or, the data features include continuous features, and the probability distribution value includes continuous probability distribution value; step 102 may specifically include: the computer device segmenting the continuous features to generate continuous segmentation features; and determining the continuous probability distribution value corresponding to the continuous segmentation features on each data subset.

[0083] In some embodiments, taking multi-sensor data collected from a vehicle as an example, the computer device extracts features for each sensor, such as camera image edge features and camera image color histograms from camera images; then it calculates the frequency of each data feature on each data subset and constructs an empirical probability distribution; for discrete features, the frequency of discrete features on the data subset can be directly counted to obtain the probability distribution value of discrete features on the data subset, and the sum of multiple probability distribution values ​​corresponding to discrete features is 1; for continuous features, continuous features can be segmented by binning or kernel density estimation (KDE) to generate multiple continuous segmentation features, and for each continuous segmentation feature, the continuous probability distribution value corresponding to the continuous segmentation feature on each data subset is determined.

[0084] For example, speed is a continuous feature, and its range is [30, 50]. Speed ​​is segmented into two continuous segmentation features: a speed range of 30-40 km / h and a speed range of 40-50 km / h. Table 1 shows the correspondence between continuous probability distribution values ​​of speed ranges on different data subsets provided in this application embodiment, as shown in Table 1 below.

[0085] Table 1

[0086] Speed ​​range (km / h) Data subset A Data subset B 30-40 20％ 10％ 40-50 30％ 40％

[0087] As shown in Table 1 above, the continuous probability distribution value for the speed range of 30-40 km / h is 20% in data subset A and 10% in data subset B; the continuous probability distribution value for the speed range of 40-50 km / h is 30% in data subset A and 40% in data subset B.

[0088] For example, the luminance histogram LH is a continuous feature. Segmenting the luminance histogram generates five continuous segmentation features: luminance interval bin1, luminance interval bin2, luminance interval bin3, luminance interval bin4, and luminance interval bin5. Table 2 shows the correspondence between continuous probability distribution values ​​of luminance intervals on different data subsets provided in the embodiments of this application.

[0089] Table 2

[0090] Brightness range Data subset A Data subset B bin1 5％ 20％ bin2 15％ 25％ … … … bin5 20％ 5％

[0091] As shown in Table 2 above, the continuous probability distribution value of brightness interval bin1 is 5% in data subset A and 20% in data subset B. The continuous probability distribution values ​​of other brightness intervals in different data subsets will not be elaborated here.

[0092] In one possible implementation, step 103 may specifically include:

[0093] Step 1031: The computer device determines a test set and at least one training set from multiple data subsets, and generates a preliminary cross-entropy value for each training set based on the probability distribution value of the data feature on the test set and the probability distribution value of the data feature on each training set, until each data subset has appeared as a test set.

[0094] In some embodiments, step 1031 may specifically include:

[0095] Step 1031A: The computer device determines a test set and at least one training set from multiple subsets of data.

[0096] In some embodiments, the test set may be identified by "active" or "crit".

[0097] For example, multiple data subsets include data subset A and data subset B. The computer device determines the training set as data subset A and the test set as data subset B from data subset A and data subset B.

[0098] Step 1031B: The computer device uses the probability distribution value of the data features in the test set as the predicted probability distribution value and the probability distribution value of the data features in the training set as the baseline probability distribution value.

[0099] In some embodiments, the baseline probability distribution value P is used to determine the probability distribution of the data feature in the training set, serving as the baseline distribution of the data feature; the predicted probability distribution value Q is used to estimate the probability distribution of the data feature in the training set, serving as the estimated distribution of the data feature. For example, if data subset A is the training set and data subset B is the test set, the computer device uses the probability distribution value of the data feature corresponding to data subset A as the baseline probability distribution value P, and uses the probability distribution value of the data feature corresponding to data subset B as the predicted probability distribution value Q.

[0100] The baseline probability distribution value P includes the baseline continuous probability distribution value P(X). i The predicted probability distribution value Q includes the predicted continuous probability distribution value Q(X). i For continuous segmentation features, the computer device uses the continuous probability distribution value of the continuous segmentation feature in the test set as the predicted continuous probability distribution value Q(X). i The continuous probability distribution values ​​of the continuous segmentation features in the training set are used as the baseline continuous probability distribution values ​​P(X). i ).

[0101] Taking Table 1 above as an example, the continuous probability distribution value corresponding to the speed range of 30-40 km / h in data subset A is taken as the baseline continuous probability distribution value P(V1); the continuous probability distribution value corresponding to the speed range of 40-50 km / h in data subset A is taken as the baseline continuous probability distribution value P(V1); the continuous probability distribution value corresponding to the speed range of 30-40 km / h in data subset B is taken as the predicted continuous probability distribution value Q(V2); and the continuous probability distribution value corresponding to the speed range of 40-50 km / h in data subset B is taken as the predicted continuous probability distribution value Q(V2).

[0102] Step 1031C: The computer device generates a preliminary cross-entropy value corresponding to at least one baseline probability distribution value based on the cross-entropy calculation formula, according to at least one baseline probability distribution value and the predicted probability distribution value corresponding to the data feature.

[0103] The excellent characteristic of the cross-entropy loss function in related technologies is that the smaller the cross-entropy value and the smaller the loss, the closer the predicted probability is to the true probability of the target class; conversely, the larger the loss, the closer the predicted probability is to the true probability of the target class. Therefore, it provides a good optimization target for training efficient classifiers. In some embodiments of this invention, the core function of cross-entropy is to evaluate the difference between a given true baseline probability distribution value P and a predicted probability distribution value Q. If the initial cross-entropy value is low, it means that the baseline probability distribution value P and the predicted probability distribution value Q are very close; if the initial cross-entropy value is high, the difference between the baseline probability distribution value P and the predicted probability distribution value Q is large.

[0104] The formulas for calculating cross-entropy include discrete cross-entropy, continuous cross-entropy, and split cross-entropy.

[0105] For discrete features, the computer device iteratively executes the steps of generating preliminary cross-entropy values ​​based on the baseline probability distribution value P and the predicted probability distribution value Q corresponding to the discrete feature using the discrete cross-entropy calculation formula, until the preliminary cross-entropy value corresponding to each baseline probability distribution value P is generated. The discrete cross-entropy calculation formula can be defined as summing all possible events X to obtain the preliminary cross-entropy value based on the baseline probability distribution value P and the predicted probability distribution value Q corresponding to the discrete feature. For example, the computer device performs a logarithmic operation on the predicted probability distribution value Q using the discrete cross-entropy calculation formula to obtain the first logarithmic result; multiplies the first logarithmic result by the baseline probability distribution value P to obtain the first product result; within event X, the first product result is accumulated to obtain the accumulated result, and the negative of the accumulated result is used to obtain the preliminary cross-entropy value; the discrete cross-entropy calculation formula can be expressed as H(P,Q)=-∑ x P(x)log(Q(x)), where H(P,Q) represents the initial cross-entropy value.

[0106] For continuous features, when the computer device does not segment the continuous features, the probability density function of the baseline probability distribution value P can be expressed as p(x), and the probability density function of the predicted probability distribution value Q can be expressed as q(x). The probability density function can be estimated from a subset of data using methods such as kernel density estimation (KDE). The computer device iteratively executes the step of generating the preliminary cross-entropy value corresponding to the baseline probability distribution value P based on the probability density function p(x) of the baseline probability distribution value P and the probability density function q(x) of the predicted probability distribution value Q, using the continuous cross-entropy calculation formula, until the preliminary cross-entropy value corresponding to each baseline probability distribution value P is generated. The continuous cross-entropy calculation formula can be defined as integrating over all possible events X to obtain the preliminary cross-entropy value based on the probability density function p(x) of the baseline probability distribution value P and the probability density function q(x) of the predicted probability distribution value Q corresponding to the continuous feature. For example, a computer device uses the continuous cross-entropy calculation formula to perform a logarithmic operation on the probability density function q(x) of the predicted probability distribution value Q to obtain a second logarithmic result; the second logarithmic result is then multiplied by the baseline probability distribution value P to obtain a second product result; within event X, the second product result is integrated and the negative number of the integral result is taken to obtain the preliminary cross-entropy value; the continuous cross-entropy calculation formula can be expressed as H(p,q)=-∫p(x)log(q(x))dx, where H(p,q) represents the preliminary cross-entropy value.

[0107] For continuous features, when a computer device segments continuous features, it iteratively executes the segmentation cross-entropy calculation formula, based on multiple baseline continuous probability distribution values ​​P(X) included in the baseline probability distribution value P. i The predicted probability distribution value Q includes multiple predicted continuous probability distribution values ​​Q(X). i The step of generating the preliminary cross-entropy value corresponding to the baseline probability distribution value P is repeated until the preliminary cross-entropy value corresponding to each baseline probability distribution value P is generated. The formula for calculating the split cross-entropy can be defined as summing over all possible events X, based on multiple baseline continuous probability distribution values ​​P(X). i ) and multiple predicted continuous probability distribution values ​​Q(X) i The initial cross-entropy value is obtained. For example, continuous features include multiple continuous segmentation features. The computer device uses the segmentation cross-entropy calculation formula to predict the continuous probability distribution value Q(X) corresponding to the continuous segmentation features on the test set. i Perform a logarithmic operation to obtain the third logarithm result; then compare the third logarithm result with the baseline continuous probability distribution value P(X) corresponding to the continuous segmentation feature on the training set. i Multiply the results to obtain the third product. Within event X, accumulate the third product results and then take the negative of the accumulated results to obtain the preliminary cross-entropy value. The formula for calculating the split cross-entropy can be expressed as H(P,Q)=-∑ x P(X i )log(Q(X i H(P,Q) represents the initial cross-entropy value, and i represents the identifier corresponding to the continuous segmentation feature.

[0108] When the data feature is speed, and speed includes multiple speed intervals, event X is speed V, and i represents the index of the speed interval, with the first speed interval numbered 1 and the Nth speed interval numbered N; for the test set and any training set, the formula for calculating the split cross-entropy can be expressed as: That is, within N speed intervals, the predicted continuous probability distribution value Q(V) on the test set for each speed interval. i Perform a logarithmic operation to obtain the third logarithm result; then compare the third logarithm result with the baseline continuous probability distribution value P(X) corresponding to this speed range on the training set. i Multiply the results to obtain the third product; sum the third product results corresponding to each speed interval, and then take the negative of the sum to obtain the initial cross-entropy value corresponding to the speed. Taking Table 1 as an example, the initial cross-entropy of speed V is H(P,Q)=-(20%*log(10%)+30%*log(40%)).

[0109] Furthermore, other modifications to the above formula are all within the scope of protection of this invention. For example, since the natural logarithm is used in the above formula, the predicted probability distribution value Q cannot be 0. Therefore, the predicted probability distribution value Q can be added to a very small constant before performing the logarithmic operation to avoid mathematical errors. This invention does not limit the formula for calculating cross-entropy.

[0110] When the number of data subsets is k, the computer device performs k iterations, each iteration being independent of the others. Each iteration selects a different data subset as the test set, and the data subsets other than the test set are used as the training set. The computer device iteratively executes steps 1031A to 1031C until every data subset has been used as a test set. For example, taking Table 1 above as an example, in the first iteration, data subset A is used as the training set, and data subset B is used as the test set. The initial cross-entropy of velocity V is H(P,Q) = -(20% * log(10%) + 30% * log(40%)). In the second iteration, data subset A is used as the test set, and data subset B is used as the training set. The initial cross-entropy of velocity V is H(P,Q) = -(10% * log(20%) + 40% * log(30%)).

[0111] Step 1032: The computer device generates a robust cross-entropy value corresponding to the data feature based on multiple preliminary cross-entropy values ​​corresponding to the data feature.

[0112] In some embodiments, robust cross-entropy is a robustness assessment of the initial cross-entropy value, and robust cross-entropy includes at least one of the following: average cross-entropy, standard deviation cross-entropy, median cross-entropy, quartile cross-entropy, extreme value cross-entropy, and confidence interval cross-entropy.

[0113] The average cross-entropy is the average of multiple preliminary cross-entropy values ​​corresponding to a data feature. The average cross-entropy reflects the expected estimate of the cross-entropy of the data feature and is used to provide a central tendency measure of the model's predictive performance.

[0114] The standard deviation cross-entropy is the standard deviation of multiple initial cross-entropy values ​​corresponding to a data feature. It measures the degree to which the initial cross-entropy values ​​vary around the average cross-entropy value, reflecting their reliability or volatility. A smaller standard deviation cross-entropy value indicates greater stability of the cross-entropy value across different test sets; a larger standard deviation cross-entropy value indicates greater volatility in the initial cross-entropy value, potentially suggesting inconsistent model performance across different subsets. The standard deviation cross-entropy value can assess the variability or stability of the initial cross-entropy value across different test sets.

[0115] The median cross-entropy is the median of multiple preliminary cross-entropy values ​​corresponding to a data feature. The median cross-entropy is not sensitive to outliers and is sometimes used to measure central tendency.

[0116] The quartile cross-entropy is the value of each point after sorting and dividing the multiple preliminary cross-entropy values ​​corresponding to the data feature into four equal parts. These typically include the first quartile (25th percentile), the second quartile (median, 50th percentile), and the third quartile (75th percentile). The interquartile range (IQR) is the difference between the third quartile and the first quartile. The quartile cross-entropy value can be used to evaluate the dispersion of the data feature distribution.

[0117] Extreme cross-entropy values ​​include the maximum and / or minimum values ​​among multiple preliminary cross-entropy values ​​corresponding to data features. Extreme cross-entropy values ​​are used to help identify potential outliers or determine the range of cross-entropy.

[0118] The confidence interval cross-entropy value is usually estimated based on the average cross-entropy value and the standard deviation cross-entropy value. The confidence interval cross-entropy value reflects the probability that the true cross-entropy value falls within the confidence interval at a certain confidence level.

[0119] Figure 2 This is a schematic diagram of a statistical verification process timeline provided in an embodiment of the present invention, such as... Figure 2 As shown, a Gantt chart illustrates the timeline and sequence of steps for calculating robust cross-entropy. The computer first splits the multi-source dataset into k subsets. Then, iterative cross-validation is performed k times. In each iteration, a test set is selected from the k subsets, and the remaining (k-1) subsets are used as the training set. Preliminary cross-entropy values ​​corresponding to the data features are then calculated, resulting in (k-1) preliminary cross-entropy values ​​for each data feature. This process continues until each data subset serves as a test set, at which point the iteration ends. Finally, the cross-validation results are summarized, resulting in k*(k-1) preliminary cross-entropy values ​​corresponding to the data features. The mean and standard deviation of these k*(k-1) preliminary cross-entropy values ​​are then calculated.

[0120] Robust cross-entropy values ​​typically include the average cross-entropy value and the standard deviation cross-entropy value. This ensures that the final cross-entropy evaluation result has high reliability and effectively reflects the model's generalization ability on different datasets. Through multiple iterations, the robust cross-entropy value can more accurately estimate the ability of data features to distinguish different datasets, thereby improving the robustness of autonomous driving models.

[0121] In one possible implementation, step 104 may specifically include:

[0122] Step 1041: The computer device filters at least one data feature based on the robust cross-entropy value corresponding to at least one data feature to obtain at least one target data feature.

[0123] In some embodiments, when the number of data features is equal to 1, step 1041 may specifically include: the computer device may determine the data feature as the target data feature based on the robust cross-entropy and the cross-entropy threshold corresponding to the robust cross-entropy.

[0124] When the number of data features is greater than 1, step 1041 may specifically include:

[0125] Step 1041A: The computer device sorts the multiple data features based on the robust cross-entropy values ​​corresponding to the multiple data features, and generates a feature sorting result.

[0126] In some embodiments, robust cross-entropy values ​​include average cross-entropy values. The computer device uses multiple average cross-entropy values ​​to sort multiple data features in ascending order, with the data features at the top of the list being the more discriminative features. Alternatively, they can be sorted in descending order, with the data features at the bottom being the more discriminative features. Or, the sorting can be based on standard deviation cross-entropy values, median cross-entropy values, quartile cross-entropy values, extreme value cross-entropy values, or confidence interval cross-entropy values, etc. Other cases will not be elaborated upon in this embodiment of the invention.

[0127] Step 1041B: The computer equipment performs correlation elimination and screening on the feature ranking results to obtain at least one target data feature.

[0128] In some embodiments, the computer device evaluates the relationship of the feature ranking results using statistical methods to obtain at least one feature pair, and removes data features other than the target data feature from the feature pair. The target data feature is the data feature corresponding to the largest robust cross-entropy value in the feature pair. And / or, the computer device identifies at least one redundant feature from the feature ranking results using redundancy reduction techniques; removes the redundant feature from the feature ranking results, and uses the data feature other than the redundant feature in the feature ranking results as the target data feature.

[0129] Statistical methods include Pearson correlation coefficient and Spearman rank correlation coefficient. Computer equipment considers the correlation between features and uses statistical methods to evaluate the linear or non-linear relationship between features, obtaining highly correlated feature pairs. Since multiple data features in a feature pair provide similar information, only the data feature with the strongest discriminative power can be retained based on the feature ranking result to reduce redundancy.

[0130] Redundancy reduction techniques include regularization techniques and feature selection algorithms. Feature selection algorithms include recursive feature elimination (RFE), which can eliminate redundant features, reduce redundancy, and help select the best-performing data features.

[0131] Step 1042: The computer device generates a feature subset based on at least one target data feature.

[0132] In some embodiments, the computer device combines ranking results with correlation analysis to remove redundant and weakly discriminative features from multiple data features, selecting target data features with strong discriminative power and low correlation among them. The feature subset includes at least one target data feature. This results in a feature subset that effectively distinguishes the dataset while being as concise as possible. This feature subset provides maximum information about the differences between multi-source datasets while keeping the complexity of subsequent models as low as possible. This is crucial for improving the model's generalization and interpretability, playing a vital role in subsequent model training or data analysis.

[0133] In one possible implementation, step 104 is followed by: the computer device training a machine learning model based on a subset of features to generate an applicable model.

[0134] In some embodiments, the machine learning model includes a supervised learning model, such as a decision tree, support vector machine, neural network, etc.

[0135] Computer equipment trains machine learning models based on a subset of features. These models can be applied to any domain; for example, in the case of machine learning models related to autonomous driving, the computer equipment uses a subset of features as input data to train one or more machine learning models related to autonomous driving. During training, cross-validation techniques can be used to evaluate the model's generalization ability and to select hyperparameters.

[0136] After training the machine learning model, it needs to be tested and validated. The model's performance can be evaluated on independent test datasets, which should ideally include a variety of datasets to represent the model's future real-world operating environment. Based on these independent test datasets, different metrics, such as accuracy, recall, and F1 score, can be used to verify whether the model truly possesses cross-dataset generalization ability.

[0137] After the computer equipment tests and validates the machine learning model, test results are obtained. Based on these results, the parameters of the machine learning model are adjusted. Parameter adjustment can be done manually by the user operating the computer equipment, or it can be done automatically, such as through grid search, random search, or Bayesian optimization. If the model performs poorly on certain datasets, it may be necessary to return to the feature selection stage to reconsider which features to include, or to consider introducing new features.

[0138] To improve a model's generalization ability, various methods can be employed, including data partitioning, regularization, early stopping, dropout, data augmentation, and ensemble learning. These methods allow for continuous monitoring of the model's performance in real-world environments and further optimization based on feedback. For example, data partitioning involves dividing the dataset into training, validation, and test sets. The validation set is used for model selection and parameter tuning, while the test set is used to evaluate the model's generalization ability. Regularization techniques, such as L1 or L2 regularization, penalize model complexity and limit model capacity. Early stopping stops training when the loss on the validation set stops, preventing overfitting. Dropout randomly discards a portion of neurons during neural network training, reducing reliance on local features and increasing generalization ability. Data augmentation enhances generalization by increasing the diversity of training data, such as through rotation, scaling, and pruning. Ensemble learning combines the predictions of multiple models to reduce overfitting.

[0139] Once the performance of the machine learning model reaches the expected target, it is deployed as an applicable model. For example, a machine learning model related to autonomous driving can be deployed as an applicable model into an autonomous driving system. The deployment process may involve converting the applicable model into a production-ready format, such as TensorFlow Lite for mobile or embedded devices. After deployment, continuous monitoring and maintenance of the applicable model are necessary to ensure that it maintains good performance when faced with new data. Therefore, the autonomous driving data processing method provided in this embodiment of the invention can be applied to optimize autonomous driving systems, helping to enhance the robustness and accuracy of the model, thereby improving the safety and reliability of autonomous vehicles in real-world road environments.

[0140] However, the steps of training the model can also be performed on other devices. The computer device sends a subset of features to other devices, including but not limited to computers, mobile phones, vehicles, wearable devices, chips, servers, etc.

[0141] By evaluating the differences in feature distribution across different datasets, we can identify which features are more susceptible to changes in conditions and adjust the model or data collection strategy accordingly to enhance its robustness and improve its performance. Cross-entropy provides a quantitative indicator of data consistency, enabling precise assessment and correction of distribution biases in multi-data source merging or transfer learning scenarios, thus improving data consistency. By minimizing cross-entropy, the model can better generalize to unseen data and environments, reducing the risk of overfitting, increasing the model's adaptability to new scenarios, and improving its generalization ability. Cross-entropy can help researchers understand the importance of different features in model decisions, contributing to improved model design or a deeper understanding of the underlying generation mechanisms of data, thereby enhancing model interpretability.

[0142] The above examples illustrate that the method described in this invention can be applied to the field of autonomous driving, ensuring that the autonomous driving system can maintain stable and reliable performance in various environments, reducing the risks caused by data differences, and accelerating the migration of autonomous driving technology to the real world. However, the method of this invention is not limited to the field of autonomous driving and can be applied to multiple fields, which will not be elaborated here.

[0143] This invention provides an autonomous driving data processing method that segments an acquired multi-source dataset to generate multiple data subsets; determines the probability distribution value of at least one data feature on each data subset; generates a robust cross-entropy value corresponding to at least one data feature based on the multiple probability distribution values ​​corresponding to the at least one data feature; and generates a feature subset based on the robust cross-entropy value corresponding to the at least one data feature. This allows for the evaluation of feature distribution differences across different datasets using cross-entropy, identifying which features are more susceptible to changes in conditions, and outputting features with significant differences between multi-source datasets, thus reducing the risk caused by data differences. Furthermore, since the feature distribution differences between different datasets directly affect the algorithm's generalization ability, the model or collection strategy can be adjusted accordingly to enhance the model's robustness and improve its performance robustness. The model can better generalize to unseen data and environments, reducing the risk of overfitting, increasing the model's adaptability to new scenarios, and improving its generalization ability. It can also help researchers understand the importance of different features in model decision-making, contributing to improved model design or a deeper understanding of the underlying generation mechanisms of data, and improving model interpretability.

[0144] Figure 3 This is a schematic diagram of the structure of an autonomous driving data processing device provided in an embodiment of the present invention, as shown below. Figure 3As shown, the device includes: a first generation module 11, a determining module 12, a second generation module 13, and a third generation module 14. The first generation module 11 is connected to the determining module 12, the determining module 12 is connected to the second generation module 13, and the second generation module 13 is connected to the third generation module 14.

[0145] The first generation module 11 is used to segment the acquired multi-source dataset to generate multiple data subsets; the determination module 12 is used to determine the probability distribution value corresponding to at least one data feature on each data subset; the second generation module 13 is used to generate a robust cross-entropy value corresponding to at least one data feature based on the multiple probability distribution values ​​corresponding to at least one data feature; the third generation module 14 is used to generate a feature subset based on the robust cross-entropy value corresponding to at least one data feature.

[0146] In one possible implementation, the second generation module 13 includes an iterative submodule 131 and a first generation submodule 132. The iterative submodule 131 is connected to the first generation submodule 132.

[0147] The iterative submodule 131 is used to determine a test set and at least one training set from multiple data subsets, and generate a preliminary cross-entropy value for each training set based on the probability distribution value of the data feature on the test set and the probability distribution value of the data feature on each training set, until each data subset has appeared as a test set; the first generation submodule 132 is used to generate a robust cross-entropy value for the data feature based on the multiple preliminary cross-entropy values ​​corresponding to the data feature.

[0148] In one possible implementation, the iterative submodule 131 is specifically used to determine a test set and at least one training set from multiple data subsets; to use the probability distribution value of the data feature in the test set as the predicted probability distribution value and the probability distribution value of the data feature in the training set as the baseline probability distribution value; and to generate an initial cross-entropy value corresponding to at least one baseline probability distribution value based on the cross-entropy calculation formula, according to the at least one baseline probability distribution value and the predicted probability distribution value corresponding to the data feature.

[0149] In one possible implementation, robust cross-entropy values ​​include at least one of the following: average cross-entropy, standard deviation cross-entropy, median cross-entropy, quartile cross-entropy, extreme value cross-entropy, and confidence interval cross-entropy.

[0150] In one possible implementation, the first generation module 11 includes a second generation submodule 111 and a third generation submodule 112. The second generation submodule 111 and the third generation submodule 112 are connected.

[0151] The second generation submodule 111 is used to preprocess the multi-source dataset to generate a standardized multi-source dataset; the third generation submodule 112 is used to segment the standardized multi-source dataset to generate multiple data subsets.

[0152] In one possible implementation, the first generation module 11 or the third generation submodule 112 is specifically used to perform average division of the multi-source dataset to generate multiple data subsets of the same size.

[0153] In one possible implementation, preprocessing includes at least one of checking data integrity, standardizing data format, standardizing image format, and standardizing data range.

[0154] In one possible implementation, the third generation module 14 includes a obtaining submodule 141 and a fourth generation submodule 142. The obtaining submodule 141 and the fourth generation submodule 142 are connected.

[0155] The first submodule 141 is used to filter at least one data feature based on the robust cross-entropy value corresponding to at least one data feature to obtain at least one target data feature; the second submodule 142 is used to generate a feature subset based on at least one target data feature.

[0156] In one possible implementation, when the number of data features is equal to 1, the submodule 141 is specifically used to determine the data feature as the target data feature based on the robust cross-entropy and the cross-entropy threshold corresponding to the robust cross-entropy.

[0157] In one possible implementation, when the number of data features is greater than 1, submodule 141 is specifically used to sort multiple data features based on the robust cross-entropy values ​​corresponding to multiple data features, generate feature sorting results, and perform correlation exclusion and filtering processing on the feature sorting results to obtain at least one target data feature.

[0158] In one possible implementation, the submodule 141 is specifically used to evaluate the relationship of the feature ranking result by statistical methods, obtain at least one feature pair, and remove the data features other than the target data feature in the feature pair, wherein the target data feature is the data feature corresponding to the largest robust cross-entropy value in the feature pair.

[0159] And / or, by using redundancy reduction techniques, at least one redundant feature is identified from the feature sorting results; the redundant feature in the feature sorting results is removed, and the data feature other than the redundant feature in the feature sorting results is taken as the target data feature.

[0160] In one possible implementation, the data features include discrete features, and the determining module 12 is specifically used to determine the probability distribution value corresponding to the discrete features on each subset of data.

[0161] In one possible implementation, the data features include continuous features, and the probability distribution values ​​include continuous probability distribution values; the determining module 12 is specifically used to segment the continuous features to generate continuous segmentation features; and to determine the continuous probability distribution values ​​corresponding to the continuous segmentation features on each of the data subsets.

[0162] This invention provides an autonomous driving data processing device, comprising a first generation module for segmenting an acquired multi-source dataset to generate multiple data subsets; a determination module for determining the probability distribution value corresponding to at least one data feature on each data subset; a second generation module for generating a robust cross-entropy value corresponding to at least one data feature based on the multiple probability distribution values ​​corresponding to the at least one data feature; and a third generation module for generating a feature subset based on the robust cross-entropy value corresponding to the at least one data feature. This allows for the evaluation of feature distribution differences across different datasets using cross-entropy, identifying which features are more susceptible to changes in conditions, and outputting features with significant differences between multi-source datasets, thus reducing the risk caused by data differences. Furthermore, since the feature distribution differences between different datasets directly affect the generalization ability of the algorithm, the model or collection strategy can be adjusted accordingly to enhance the robustness of the model, improving its performance robustness. The model can better generalize to unseen data and environments, reducing the risk of overfitting, increasing the model's adaptability to new scenarios, and improving its generalization ability. It can also help researchers understand the importance of different features in model decision-making, contributing to improved model design or a deeper understanding of the underlying generation mechanisms of data, and improving model interpretability.

[0163] Corresponding to the above embodiments, this application also provides a computer device. Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 800 may include a processor 801, a memory 802, and a communication unit 803. These components communicate through one or more buses. Those skilled in the art will understand that the structure of the computer device shown in the figure does not constitute a limitation on the embodiment of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0164] The communication unit 803 is used to establish a communication channel, enabling the computer device to communicate with other devices. It receives user data from other devices or sends user data to other devices.

[0165] The processor 801 serves as the control center of the computer device, connecting various parts of the device via interfaces and lines. It executes software programs, instructions, and / or modules stored in the memory 802, and accesses data stored in the memory to perform various functions and / or process data. The processor may be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 801 may consist only of a central processing unit (CPU). In this embodiment, the CPU may have a single processing core or include multiple processing cores.

[0166] The memory 802 is used to store the execution instructions of the processor 801. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0167] When the execution instructions in memory 802 are executed by processor 801, the computer device 800 is able to perform its functions. Figure 1 Some or all of the steps in the illustrated embodiments.

[0168] In a specific implementation, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, which, when executed, may include some or all of the steps of the various embodiments of the autonomous driving data processing method provided by the present invention. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0169] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0170] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the descriptions of the apparatus embodiments and computer device embodiments are relatively simple since they are basically similar to the method embodiments, and the relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. An automatic driving data processing method, characterized by, include: The acquired multi-source dataset is segmented to generate multiple data subsets; Determine the probability distribution value of at least one data feature on each subset of data; Generate a robust cross-entropy value corresponding to at least one of the data features based on multiple probability distribution values ​​corresponding to at least one of the data features; A feature subset is generated based on the robust cross-entropy value corresponding to at least one of the data features.

2. The method of claim 1, wherein, The step of generating a robust cross-entropy value corresponding to at least one of the data features based on multiple probability distribution values ​​corresponding to at least one of the data features includes: A test set and at least one training set are determined from multiple data subsets, and a preliminary cross-entropy value of the data feature on each training set is generated based on the probability distribution value of the data feature on the test set and the probability distribution value of the data feature on each training set, until each data subset has appeared as a test set; Based on the multiple preliminary cross-entropy values ​​corresponding to the data features, a robust cross-entropy value corresponding to the data features is generated.

3. The method of claim 2, wherein, The step of determining a test set and at least one training set from multiple data subsets, and generating a preliminary cross-entropy value for the data feature on each training set based on the probability distribution value of the data feature on the test set and the probability distribution value of the data feature on each training set, includes: A test set and at least one training set are determined from the multiple subsets of data; The probability distribution value of the data feature in the test set is used as the predicted probability distribution value, and the probability distribution value of the data feature in the training set is used as the baseline probability distribution value. Using the cross-entropy calculation formula, at least one preliminary cross-entropy value corresponding to the baseline probability distribution value is generated based on at least one baseline probability distribution value corresponding to the data feature and the predicted probability distribution value.

4. The method according to claim 1 or 2, characterized in that, The robust cross-entropy value includes at least one of the following: average cross-entropy value, standard deviation cross-entropy value, median cross-entropy value, quartile cross-entropy value, extreme value cross-entropy value, and confidence interval cross-entropy value.

5. The method according to claim 1, characterized in that, The process of segmenting the acquired multi-source dataset to generate multiple data subsets includes: The multi-source dataset is preprocessed to generate a standardized multi-source dataset; The standardized multi-source dataset is segmented to generate multiple data subsets.

6. The method according to claim 1 or 5, characterized in that, The process of segmenting the acquired multi-source dataset to generate multiple data subsets includes: The multi-source dataset is divided into multiple subsets of the same size by average splitting.

7. The method according to claim 5, characterized in that, The preprocessing includes at least one of the following: checking data integrity, standardizing data format, standardizing image format, and standardizing data range.

8. The method according to claim 1, characterized in that, The step of generating a feature subset based on the robust cross-entropy value corresponding to at least one of the data features includes: Based on the robust cross-entropy value corresponding to at least one of the data features, at least one of the data features is filtered to obtain at least one target data feature; The feature subset is generated based on at least one of the target data features.

9. The method according to claim 8, characterized in that, When the number of data features is equal to 1, the step of filtering at least one data feature based on the robust cross-entropy value corresponding to at least one data feature to obtain at least one target data feature includes: Based on the robust cross-entropy and the cross-entropy threshold corresponding to the robust cross-entropy, the data feature is determined as the target data feature.

10. The method according to claim 8, characterized in that, When the number of data features is greater than 1, the step of filtering at least one data feature based on the robust cross-entropy value corresponding to at least one data feature to obtain at least one target data feature includes: Based on the robust cross-entropy values ​​corresponding to multiple data features, the multiple data features are sorted to generate a feature sorting result; The feature ranking results are subjected to correlation exclusion filtering to obtain at least one target data feature.

11. The method according to claim 10, characterized in that, The relevance exclusion and filtering process performed on the feature ranking results yields at least one target data feature, including: The relationship of the feature ranking results is evaluated by statistical methods to obtain at least one feature pair. The data features other than the target data feature in the feature pair are removed. The target data feature is the data feature corresponding to the largest robust cross-entropy value in the feature pair. And / or, by using redundancy reduction techniques, at least one redundant feature is identified from the feature sorting results; the redundant feature in the feature sorting results is removed, and the data feature other than the redundant feature in the feature sorting results is taken as the target data feature.

12. The method according to claim 1, characterized in that, The data features include discrete features. Determining the probability distribution value corresponding to at least one data feature on each subset of data includes: The probability distribution value corresponding to the discrete feature on each subset of data is determined.

13. The method according to claim 1, characterized in that, The data features include continuous features. The probability distribution values ​​include continuous probability distribution values; determining the probability distribution value corresponding to at least one data feature on each subset of data includes: The continuous features are segmented to generate continuous segmentation features; The continuous probability distribution value corresponding to the continuous segmentation feature on each of the data subsets is determined.

14. An autonomous driving data processing device, characterized in that, include: The first generation module is used to segment the acquired multi-source dataset and generate multiple data subsets; A determining module is used to determine the probability distribution value corresponding to at least one data feature on each subset of data; The second generation module is used to generate at least one robust cross-entropy value corresponding to at least one of the data features based on multiple probability distribution values ​​corresponding to at least one of the data features. The third generation module is used to generate a feature subset based on the robust cross-entropy value corresponding to at least one of the data features.

15. A computer device, characterized in that, The device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the computer device performs the autonomous driving data processing method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the autonomous driving data processing according to any one of claims 1 to 13.

Images