Data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a readable storage medium. The method comprises the steps that multiple to-be-processed driving data pieces are acquired; deleting the driving data pieces with abnormal values from the plurality of driving data pieces to obtain a first data set; if the values among the plurality of first driving data pieces do not meet the value corresponding relation, deleting the plurality of first driving data pieces from the first data set to obtain a second data set; and if the ratio of the exceeding value in the second driving data piece exceeds the first threshold value, deleting the second driving data piece from the second data set to obtain a target data set. According to the method provided by the invention, the accuracy of the values in the target data set obtained after screening is relatively high.

Description

Technical Field

[0001] The present application relates to the field of electronic information technology, and more specifically, to a data processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] As machine learning becomes more mature, the quality of data used for machine learning becomes increasingly important for the training process of machine learning models, which directly affects the accuracy, reliability, and efficiency of machine learning model training.

[0003] Currently, the data to be processed is screened through a set rule library to select data that meets the rules for training machine learning models.

[0004] However, using existing data processing methods, the quality of the screened data is poor, resulting in poor recognition effect of the machine learning model trained with the screened data. Summary of the invention

[0005] The present application proposes a data processing method, device, electronic device and computer-readable storage medium to improve the above-mentioned defects.

[0006] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0007] Acquire a plurality of driving data slices to be processed, wherein one driving data slice includes a value of a driving parameter at at least one collection time point within a collection period;

[0008] Deleting a driving data slice having an abnormal value from the plurality of driving data slices to obtain a first data set;

[0009] If the values ​​between the multiple first driving data slices do not satisfy the value correspondence relationship, the multiple first driving data slices are deleted from the first data set to obtain a second data set; the multiple first driving data slices include driving data slices corresponding to the multiple associated driving parameters in the first data set in the same collection period; the multiple associated driving parameters refer to driving parameters that have corresponding values ​​in the first data set and have a value correspondence relationship;

[0010] If the proportion of the exceeded values ​​in the second driving data slice exceeds the first threshold, the second driving data slice is deleted from the second data set to obtain the target data set; the second driving data slice is any driving data slice in the second data set.

[0011] In a second aspect, an embodiment of the present application further provides a data processing device, the device comprising:

[0012] An acquisition module, used to acquire a plurality of driving data slices to be processed, wherein a driving data slice includes a value of a driving parameter at at least one acquisition time point within a acquisition period;

[0013] A first deleting module, used to delete the driving data slices with abnormal values ​​from the multiple driving data slices to obtain a first data set;

[0014] A second deletion module is used to delete the multiple first driving data pieces from the first data set to obtain a second data set if the values ​​between the multiple first driving data pieces do not satisfy the value correspondence relationship; the multiple first driving data pieces include driving data pieces corresponding to the multiple associated driving parameters in the first data set in the same collection period; the multiple associated driving parameters refer to driving parameters that have corresponding values ​​in the first data set and have a value correspondence relationship;

[0015] The third deletion module is used to delete the second driving data slice from the second data set to obtain the target data set if the proportion of the exceeded values ​​in the second driving data slice exceeds the first threshold; the second driving data slice is any driving data slice in the second data set.

[0016] In a third aspect, an embodiment of the present application further provides an electronic device, characterized in that the electronic device comprises: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above method.

[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the above method.

[0018] The present application provides a data processing method, device, electronic device and computer-readable storage medium. In the present application, first, the driving data slices with abnormal values ​​are deleted, then the first driving data slices whose values ​​do not satisfy the value correspondence relationship are deleted, and then the second driving data slices whose proportion of excessive values ​​exceeds the first threshold are deleted to obtain a target data set. It can be seen that in the present application, the multiple driving data slices to be processed are screened from three dimensions: abnormal values, value correspondence relationship and proportion of excessive values, thereby realizing multi-dimensional and comprehensive screening of the multiple driving data slices to be processed, avoiding the situation where the filtered data is inaccurate due to the single setting of rules when filtering data only by setting rules, so that the values ​​in each driving data slice in the target data set obtained after screening are more accurate and effective, thereby improving the accuracy of each driving data slice in the target data set, thereby improving the recognition effect of the machine learning model trained according to the values ​​in each driving data slice in the target data set.

[0019] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flow chart of a data processing method proposed in one embodiment of the present application is shown.

[0022] Figure 2 Shows Figure 1 The steps after step S140 of the corresponding embodiment are a flowchart in an embodiment.

[0023] Figure 3 A schematic diagram showing a data processing process in an embodiment of the present application.

[0024] Figure 4 A structural block diagram of a data processing device proposed in one embodiment of the present application is shown.

[0025] Figure 5 A structural block diagram of an electronic device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0027] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0028] See also Figure 1 , Figure 1 A flow chart of a data processing method proposed in one embodiment of the present application is shown, which is used in an electronic device. The method includes:

[0029] S110: Acquire multiple driving data slices to be processed.

[0030] A driving data slice includes the value of a driving parameter at at least one collection time point in a collection period. A driving parameter may correspond to multiple driving data slices, and each driving data slice belongs to a row. Driving parameters refer to parameters involved in the vehicle driving process. For example, driving parameters may be the vehicle speed, acceleration, angular velocity, steering wheel angle, lighting, and air conditioning temperature.

[0031] The collection time point refers to the time point used to collect the driving parameters of the vehicle. When each collection time point is reached, the vehicle collects the values ​​of each driving parameter as the values ​​of the vehicle at the collection time point.

[0032] After the vehicle is powered on, the values ​​of driving parameters can be collected according to the set collection cycle. Each time point when the collection cycle is reached can be used as a collection time point. For example, if the collection cycle is 100ms, the 100ms, 200ms, 300ms... after the vehicle is powered on will all be used as collection time points.

[0033] The collection time period can be a time period of any length set according to demand. The lengths of different collection time periods can be the same. The collection time period includes at least two collection time points. For example, if the length of the collection time period is 10s and the collection cycle is 100ms, the collection time period includes 100 collection time points.

[0034] The value of a driving parameter within a collection time period can be determined from the data collected by the vehicle as a driving data slice. The data collected by the vehicle includes the values ​​of each driving parameter within a longer specified time period (the duration of the specified time period is longer than the duration of the collection time period), and the driving data slices of each driving parameter can be intercepted from the data collected by the vehicle according to the set duration of the collection time period.

[0035] The collection time period of a driving data slice of a driving parameter includes at least two collection time points. The vehicle may not have collected the value of the driving parameter at one or more collection time points. Therefore, there may be one or more collection time points in the driving data slice where no value is collected (but not all collection time points where no value is collected. If no value is collected, then the driving data slice does not contain data and does not exist). Therefore, the driving data slice includes the value of the driving parameter at at least one collection time point within a collection time period.

[0036] For example, the driving parameter is vehicle speed, the collection time period includes 100 collection time points, and a driving data slice corresponding to the driving parameter includes 99 vehicle speed values ​​(no value is collected at one collection time point).

[0037] S120 , deleting driving data slices with abnormal values ​​from the multiple driving data slices to obtain a first data set.

[0038] An abnormal value may refer to a value that is not generated during normal driving of the vehicle. The presence of an abnormal value in a driving data slice indicates that there is a value in the driving data slice that is not generated during normal driving of the vehicle, and the value in the driving data slice is inaccurate, so the driving data slice needs to be deleted. The driving data slices with abnormal values ​​are deleted from the multiple driving data slices, and the remaining driving data slices after deletion are obtained as a set of aggregated data as the first data set.

[0039] In some implementations, for each form data slice, if the driving data slice satisfies an abnormal value condition, it is determined that the driving data slice has an abnormal value. Wherein, satisfying the abnormal value condition includes at least one of the following: there is a missing value in the driving data slice; the difference between the values ​​of two adjacent collection time points in the driving data slice within the corresponding collection period is greater than a second threshold; there is a non-differentiable value in the driving data slice; there is a value in the driving data slice whose positive and negative signs do not match the standard positive and negative signs of the driving parameters corresponding to the driving data slice.

[0040] In this embodiment, the values ​​in a driving data slice can be stored in the form of a data packet. The driving data slice can be unpacked into the dataframe data format of the Pandas library, and the df.isna().any() function in the Pandas library can be used to detect missing values ​​in the driving data slice. If missing values ​​are detected, it is determined that the driving data slice has at least one collection time point with missing values ​​within the corresponding collection period, thereby determining that there are missing values ​​in the driving data slice.

[0041] For each driving data slice, a clustering algorithm (such as the K-nearest neighbor algorithm (KNN)) can be used to determine multiple first clusters based on the values ​​in the driving data slice, and determine the relative distance between each value and the corresponding first cluster center. Based on the relative distance between each value and the corresponding first cluster center, it is determined whether there is a mutation value in the driving data slice.

[0042] Specifically, for each driving data slice, after determining the relative distance between each value in the driving data slice and the corresponding first cluster center, the relative distances of all points in the multiple first clusters in the driving data slice can be averaged to obtain the average relative distance, and then the value whose relative distance exceeds twice the average relative distance is determined as the mutation value.

[0043] For each driving data slice, the values ​​in the driving data slice can be differentiated by a derivative function (such as numpy's diff function). If one or more values ​​cannot be differentiated by the derivative function, it is determined that the driving data slice has non-differentiable values.

[0044] Each driving data slice corresponds to a driving parameter. The positive and negative signs of each driving parameter during the normal driving process of the vehicle are used as the standard positive and negative signs of each driving parameter. The standard positive and negative signs of each driving parameter during the normal driving process of the vehicle are usually fixed. For example, the driving parameter vehicle speed is non-negative during the driving process of the vehicle. Mismatch means that the positive and negative signs are inconsistent. Therefore, if a driving data slice contains a value whose positive and negative signs do not match the standard positive and negative signs of the driving parameter corresponding to the driving data slice, it means that the value whose positive and negative signs do not match the standard positive and negative signs of the driving parameter corresponding to the driving data slice does not conform to the normal form of the vehicle. This value is taken as an abnormal value, thereby determining that the driving data slice contains an abnormal value.

[0045] It is worth mentioning that the abnormal driving data slice with abnormal values ​​can be repaired to obtain a repaired driving data slice, and the repaired driving data slice is added to the first data set; the repair processing includes but is not limited to supplementing missing values, correcting abnormal values, etc.

[0046] For example, for abnormal driving data pieces with missing values, values ​​are inserted through interpolation to make up for the missing values. For another example, for abnormal driving data pieces with mutation values, values ​​within the normal range are determined to replace the mutation values. For another example, for non-differentiable values, differentiable values ​​are determined to replace the non-differentiable values. For another example, for values ​​whose signs do not match the standard signs of the driving parameters corresponding to the driving data pieces, the sign of the value and the absolute value of the value are adjusted so that the sign of the adjusted value matches the standard sign of the driving parameters corresponding to the driving data piece.

[0047] It can be understood that, during the execution of S120, the value of the driving parameter itself is actually verified and analyzed, so S120 is to implement data screening at the parameter level.

[0048] S130: If the values ​​between the multiple first driving data pieces do not satisfy the value correspondence relationship, the multiple first driving data pieces are deleted from the first data set to obtain a second data set.

[0049] Among them, the multiple first driving data slices include driving data slices corresponding to multiple associated driving parameters in the first data set in the same collection period; the multiple associated driving parameters refer to driving parameters that have corresponding values ​​and have a value correspondence relationship in the first data set.

[0050] The value correspondence refers to the correspondence between the values ​​of different driving parameters. For example, the value correspondence may include a calculus relationship or a conversion through a preset rule, wherein the calculus relationship includes but is not limited to a calculus relationship between vehicle speed and acceleration, a calculus relationship between vehicle speed and distance, etc. The preset rule may include a conversion relationship between the coordinate value in the Ego coordinate system and the longitude and latitude coordinate value, and the preset rule may also include a conversion relationship between the coordinate value of the radar coordinate system in the vehicle and the vehicle's own vehicle tilt angle and heading angle. For example, the values ​​of vehicle speed and acceleration are differential and integral to each other.

[0051] First, the driving parameters that have values ​​and the aforementioned corresponding relationship in the first data set can be determined as the associated driving parameters. For example, the values ​​of the vehicle speed and the acceleration are in a calculus relationship, and the vehicle speed and the acceleration are both associated driving parameters; then the driving data slices of the associated driving parameters in the same collection time period are determined as the first driving data slice. The values ​​in a driving data slice are the values ​​at the collection time point in a collection time period, and the driving data slice is determined to be the driving data slice in the collection time period.

[0052] For example, the driving data slices in the first data set involve collection time periods including a1, a2 and a3, wherein vehicle speed and acceleration are associated driving parameters, and the driving data slice of vehicle speed in collection time period a1 and the driving data slice of acceleration in collection time period a1 are taken as the first driving data slice.

[0053] During normal vehicle driving, the values ​​of the associated driving parameters with a value correspondence relationship in the same collection time period should satisfy the value correspondence relationship. For example, the values ​​of the vehicle speed and the acceleration in the same collection time period a1 should be in a calculus relationship with each other: the vehicle speed is the integral of the acceleration, and the acceleration is the differential of the vehicle speed. If the values ​​of the associated driving parameters with a value correspondence relationship in the same collection time period do not satisfy the value correspondence relationship, it is determined that the values ​​of the associated driving parameters in the same collection time period are abnormal (not collected during normal vehicle driving, or caused by a sensor failure for collection, etc.), therefore, it can be determined that the values ​​included in the first driving data slice of the associated driving parameters in the same collection time period are no longer accurate, and the first driving data slice is deleted.

[0054] It can be understood that, when the value correspondence relationship includes a calculus relationship or can be converted through preset rules, the values ​​between multiple first driving data pieces do not satisfy the value correspondence relationship, including at least one of the following: the values ​​between the multiple first driving data pieces are not in a calculus relationship; the values ​​between the multiple first driving data pieces cannot be converted through preset rules.

[0055] In this embodiment, TFDV (TensorFlow Data Verification) has a built-in speed-related module, which can fit the vehicle's speed prediction curve according to the vehicle's displacement value. The speed prediction curve and the real speed curve obtained by fitting the real speed value of the vehicle are similar. When the similarity is lower than a certain curve phase speed threshold, it is considered that the calculus relationship between speed and displacement is not satisfied. The curve phase speed threshold can be set based on demand, for example, the curve similarity threshold can be 0.7.

[0056] It can be understood that, in the process of executing S130, the relationship between the values ​​of the driving parameters having a relationship (that is, a value corresponding relationship) is actually verified, and therefore, S130 is to implement data screening at the relationship layer.

[0057] S140: If the proportion of the exceeding-standard values ​​in the second driving data slice exceeds the first threshold, the second driving data slice is deleted from the second data set to obtain a target data set.

[0058] The second driving data slice is any driving data slice in the second data set. The over-standard value refers to the value of the median of the second driving data slice exceeding the standard corresponding to the second driving data slice. The standard corresponding to the second driving data slice may refer to the mean, weighted sum, and mean square value of the median of the second driving data slice. The first threshold may be a value set based on demand, for example, the first threshold is 0.3%.

[0059] If the proportion of exceeded values ​​in the second driving data slice exceeds the first threshold, it means that there are too many exceeded values ​​in the second driving data slice, resulting in inaccurate values ​​in the second driving data slice, so the second driving data slice needs to be deleted; if the proportion of exceeded values ​​in the second driving data slice does not exceed the first threshold, it means that there are not many exceeded values ​​in the second driving data slice, and the values ​​in the second driving data slice are relatively accurate, so the second driving data slice can be retained, and all retained second driving data slices are obtained and summarized into a set as the target data set.

[0060] In this embodiment, for each second driving data slice, the interquartile range corresponding to the second driving data slice can be determined according to the value in the second driving data slice; the standard score corresponding to each value in the second driving data slice can be determined according to the interquartile range; for any value in the second driving data slice, if the absolute value of the standard score of the value is greater than the second threshold, the value is determined to be an over-standard value. The second threshold can be set based on demand, for example, 3.

[0061] The interquartile range (IQR), also known as the four-point difference, is a statistic that measures the distribution range of the middle 50% of the values ​​in the second driving data slice. It is calculated as the difference between the 75% (Q3) and the 25% (Q1). It is generally considered that any value lower than Q1-1.5IQR or higher than Q3+1.5IQR is abnormal because these values ​​are far away from the distribution area of ​​most data vectors.

[0062] The standard score (Z-Score) is a measure of the deviation of a value in the second driving data slice from the average value, and the unit is standard deviation; the absolute value of the standard score is greater than the second threshold and is considered abnormal.

[0063] Exemplarily, in the present application, after deleting multiple first driving data slices that do not satisfy the value correspondence relationship, the values ​​in the remaining driving data slices can be normalized to obtain a set of driving data slices with normalized values ​​as the second data set. At this time, the values ​​in each driving data slice in the second data set are all normalized values, so that the set second threshold can be 3, which is used to determine the exceeded value.

[0064] According to the rules of statistics, only about 0.3% of the data under the normal distribution will fall outside three standard deviations from the mean. Therefore, the first threshold set in the present application can be 0.3%, so as to determine the second driving data slice whose proportion of excess values ​​exceeds the first threshold, and delete the second driving data slice whose proportion of excess values ​​exceeds the first threshold to obtain the remaining driving data slices as a set as the target data set.

[0065] In addition, in the present application, after determining the second driving data slice in which the proportion of excess values ​​exceeds the first threshold, the second driving data slice in which the proportion of excess values ​​exceeds the first threshold can also be analyzed to determine the reason why the proportion of excess values ​​in the second driving data slice exceeds the first threshold. For example, the reason may be an error in data collection, a failure of the collection equipment, or other external factors; so that after determining the reason, the law of excess can be summarized according to the reason, so as to directly determine the vehicle failure, etc. by using the law of excess.

[0066] It is understandable that, in the process of executing S140, the values ​​of the driving parameters are actually counted to obtain standard scores, so as to filter out the excessive values ​​through the standard scores. Therefore, S140 is to implement data screening at the statistical layer.

[0067] In summary, S120-S140 actually implements data screening at the parameter layer, relationship layer and statistical layer respectively to obtain the final target data set.

[0068] For example, different data unpacking and input codes can be written according to different driving parameters, and the values ​​extracted from the driving data slice corresponding to the driving parameters are temporarily stored in the memory in the dataframe data structure in the python pandas package, and then the values ​​in the driving data slice are detected according to the three screening methods involved in the above S120-S140. At the same time, the detected abnormal values ​​can also be repaired, and the repair process refers to the repair process involved in the above S120.

[0069] After screening, the data without problems (that is, the target data set) can be classified as valid data for model training. At the same time, the problematic data (such as exceeded values) can be used to summarize the rules to obtain the aforementioned exceeded rules for subsequent troubleshooting, etc.

[0070] In this embodiment, the driving data slices with abnormal values ​​are first deleted, and then the first driving data slices whose values ​​do not satisfy the value correspondence relationship are deleted. Thereafter, the second driving data slices whose excess values ​​account for more than a first threshold value are deleted to obtain the target data set. It can be seen that in the present application, the multiple driving data slices to be processed are screened from three dimensions: abnormal values, value correspondence relationships, and excess value proportions, thereby achieving multi-dimensional and comprehensive screening of the multiple driving data slices to be processed, avoiding the situation where the filtered data is inaccurate due to a single set rule when filtering data only by setting rules, so that the values ​​in each driving data slice in the target data set obtained after screening are more accurate and effective, thereby improving the accuracy of each driving data slice in the target data set, thereby improving the recognition effect of the machine learning model trained according to the values ​​in each driving data slice in the target data set.

[0071] Secondly, by setting specific outlier conditions, when screening driving data slices with outliers, the driving data slices with outliers determined are more accurate, thereby improving the accuracy of the first data set after deleting the driving data slices with outliers, thereby improving the accuracy of the determined target data set.

[0072] Furthermore, a specific value correspondence is set so that the first driving data piece that does not satisfy the value correspondence is determined to be more accurate, thereby improving the accuracy of the second data set obtained after deleting the first driving data piece that satisfies the value correspondence, thereby improving the accuracy of the determined target data set.

[0073] In addition, by determining the interquartile range to determine the standard score, and using the standard score to determine whether a value is an excess value, the accuracy of the determined excess values ​​is improved, thereby improving the accuracy of the second driving data slice in which the proportion of excess values ​​exceeds the first threshold, thereby making the accuracy of the target data set after deleting the second driving data slice in which the proportion of excess values ​​exceeds the first threshold higher.

[0074] In one embodiment, if Figure 2 As shown, after S140, the method further includes:

[0075] S210 , dividing the driving data slices in the target data set to obtain scene data sets under multiple driving scenes.

[0076] Each scene data set includes at least one driving data piece in the target data set; and the driving data pieces included in a scene data set belong to the same driving scene.

[0077] In the present application, a driving scenario may refer to any scenario involved in the vehicle driving process. For example, a driving scenario may include a turning scenario, a lane changing scenario, an acceleration scenario, a deceleration scenario, a parking scenario, and an overtaking scenario.

[0078] When the vehicle collects the values ​​of each driving parameter, the driving scene to which the value of each driving parameter belongs can be determined according to the value of the driving parameter, so that for a driving data slice corresponding to any driving parameter, the driving scene to which the value in the driving data slice belongs can be determined as the driving scene corresponding to the driving data slice.

[0079] S220. Delete the driving data slices containing the candidate scene outliers from the candidate scene data set corresponding to the candidate driving scene, and obtain the first scene data set corresponding to the candidate driving scene.

[0080] Among them, the candidate driving scene is any driving scene; the candidate scene outlier refers to the scene outlier corresponding to the candidate driving scene; the scene data set corresponding to the candidate driving scene is the candidate scene data set.

[0081] A scene outlier value in a certain driving scenario refers to a value determined to be an outlier in the driving scenario. Generally speaking, excluding the constraints of the driving scenario, a value is a scene outlier value, which may not be an outlier. For example, a vehicle speed value of 100km / h is not an outlier value of the vehicle speed. However, in a turning scenario, a vehicle speed value of 100km / h is regarded as a scene outlier value (because the turning speed generally cannot exceed 100km / h).

[0082] Generally, different driving scenarios have different corresponding scene outlier values. For example, a turning scenario and an overtaking scenario have different corresponding scene outlier values.

[0083] It can be understood that the candidate driving scene is any driving scene, that is, for the scene data set under each driving scene, the driving data slices with corresponding scene outliers are deleted, and the remaining driving data slices are summarized to obtain a set as the first scene data set under the driving scene, that is, for each candidate driving scene, there is a corresponding first scene data set.

[0084] In some embodiments, if the value in the fourth driving data slice satisfies the candidate scene outlier condition corresponding to the candidate driving scene, it is determined that the fourth driving data slice has a candidate scene outlier; the fourth driving data slice is any driving data slice in the candidate scene data set; wherein, satisfying the candidate scene outlier condition includes at least one of the following: there is a value in the fourth driving data slice that exceeds the reference scene standard value; the reference scene standard value refers to the standard value of the driving parameter to which the fourth driving data slice belongs in the candidate driving scene; there is a value in the fourth driving data slice whose positive and negative signs after differentiation are inconsistent with the positive and negative signs of the reference scene standard; the reference scene standard positive and negative signs refer to the scene standard positive and negative signs of the driving parameter to which the fourth driving data slice belongs in the candidate driving scene.

[0085] The standard value of the driving parameter in the driving scenario serves as the scene standard value of the driving parameter in the driving scenario. The scene standard value of the driving parameter in the driving scenario can be set based on demand. The scene standard value of the same driving parameter in different driving scenarios may be different. For example, the scene standard value (also called standard vehicle speed) corresponding to the vehicle speed in the turning scenario and the speeding scenario may be different.

[0086] The standard positive and negative signs of the driving parameters in the driving scenario serve as the scene standard positive and negative signs of the driving parameters in the driving scenario. The scene standard positive and negative signs of the driving parameters in the driving scenario can be set based on demand. The scene standard positive and negative signs of the same driving parameter in different driving scenarios may be different. For example, the scene standard positive and negative signs corresponding to the vehicle speed in the turning scenario and the speeding scenario may be different.

[0087] For the convenience of description, any driving data piece in the candidate scene data set is taken as the fourth driving data piece.

[0088] For any fourth driving data slice, determine the driving parameter to which the value included therein belongs, and then obtain a scene standard value of the driving parameter to which it belongs under the candidate driving scene as the reference scene standard value. If there is a value in the fourth driving data slice that exceeds the reference scene standard value, determine that the value in the fourth driving data slice meets the candidate scene abnormal value condition corresponding to the candidate driving scene.

[0089] For any fourth driving data slice, determine the driving parameter to which the value included in it belongs, and then obtain a scene standard positive and negative sign of the driving parameter to which it belongs in the candidate driving scene, as the reference scene standard positive and negative sign, and calculate the first-order inverse of the value in the fourth driving data slice. If the fourth driving data slice contains a value whose positive and negative sign after differentiation is inconsistent with the positive and negative sign of the reference scene standard (different positive and negative signs mean inconsistency), it is determined that the value in the fourth driving data slice meets the candidate scene anomaly condition corresponding to the candidate driving scene.

[0090] Of course, there may be values ​​in the four driving data slices that exceed the standard value of the reference scene, and if the sign of the derivative in the fourth driving data slice is inconsistent with the standard sign of the reference scene, it is determined that the value in the fourth driving data slice meets the candidate scene outlier condition corresponding to the candidate driving scene.

[0091] In the present application, the difference of the values ​​in the fourth driving data slice may be calculated by using the di ff function of numpy, thereby implementing a derivative operation on the values ​​in the fourth driving data slice.

[0092] For any candidate driving scenario, the content of the candidate scenario abnormal value conditions can be the same (for example, both include at least one of the aforementioned reference scenario standard value screening and reference scenario standard positive and negative sign screening). The difference between the candidate scenario abnormal value conditions is that the reference scenario standard value and the reference scenario standard positive and negative sign can be different. For example, when the candidate scenario abnormal value conditions are the aforementioned two combinations, the reference scenario standard value and the reference scenario standard positive and negative sign for the vehicle speed in the turning scenario are different from the reference scenario standard value and the reference scenario standard positive and negative sign for the vehicle speed in the overtaking scenario.

[0093] After determining that there are driving data slices with candidate scene outliers in the candidate scene data set, the driving data slices with the candidate scene outliers are deleted to obtain a set of the remaining driving data slices as the first scene data set corresponding to the candidate driving scene.

[0094] It can be understood that, during the execution of S220, the value of the driving parameter itself is actually analyzed under the constraints of the driving scenario to determine whether the value is abnormal. Therefore, S220 is to implement data screening at the parameter layer.

[0095] S230: If the values ​​between the plurality of third driving data slices do not satisfy the candidate scene value correspondence relationship, the plurality of third driving data slices are deleted from the first scene data set to obtain a second scene data set corresponding to the candidate driving scene.

[0096] Among them, multiple third driving data slices refer to the driving data slices corresponding to multiple scene-associated driving parameters in the first scene data set under the candidate driving scenario in the same collection period; multiple scene-associated driving parameters refer to driving parameters that have corresponding values ​​in the first scene data set and have candidate scene value correspondences; the candidate scene value correspondences are the scene value correspondences corresponding to the candidate driving scenarios; different driving scenarios have their own scene value correspondences.

[0097] In different driving scenarios, the corresponding scene value correspondence may be different. For example, the scene value correspondence between the vehicle speed and the steering wheel angle value in a turning scenario may be different from the scene value correspondence between the vehicle speed and the steering wheel angle value in a parking scenario.

[0098] In the first scene data set, the driving parameters that have corresponding values ​​and have a corresponding relationship with the candidate scene values ​​corresponding to the candidate driving scene are used as scene-associated driving parameters. For example, if the candidate driving scene is a straight-ahead scene, and the driving parameters that have corresponding values ​​in the first scene data set and have a corresponding relationship with the scene values ​​corresponding to the straight-ahead scene include vehicle speed and acceleration, then the vehicle speed and acceleration are both used as scene-associated driving parameters.

[0099] The driving data slices corresponding to the multiple scene-related driving parameters in the first scene data set in the same collection period are used as multiple third driving data slices. For example, the collection time periods involved in the driving data slices in the first scene data set include b1, b2, and b3, wherein the vehicle speed and acceleration are scene-related driving parameters, and the driving data slice of the vehicle speed in the collection time period b1 and the driving data slice of the acceleration in the collection time period b1 are used as the third driving data slice.

[0100] In this embodiment, the scene value correspondence relationship corresponding to different driving scenes can be manually constructed through the TFDV library of Tensorflow, and this application does not make specific limitations. For example, the scene value correspondence relationship can include the change of heading angle to define the turning scene, or define the turning scene by the lateral displacement of the vehicle.

[0101] Of course, other means may be used to write a scene value correspondence relationship different from the scene value correspondence relationship manually constructed by the TFDV library, so as to judge the value in the third driving data slice by the same scene value correspondence relationship constructed by different methods, so as to improve the accuracy of the analysis of the value in the third driving data slice.

[0102] During normal driving of the vehicle, the values ​​between multiple third driving data slices satisfy the corresponding candidate scene value correspondence relationship. If the values ​​between multiple third driving data slices do not satisfy the corresponding candidate scene value correspondence relationship, it means that the values ​​between the multiple third driving data slices are not collected during normal driving of the vehicle, or that the collection equipment used to collect the driving parameters belonging to the third driving data slices is faulty, resulting in abnormal values ​​in the third driving data slices. Therefore, the multiple third driving data slices that do not satisfy the corresponding candidate scene value correspondence relationship can be deleted to obtain the second scene data set under the candidate driving scene.

[0103] It can be understood that, for each driving scenario, the respective second scenario data set can be determined according to the means described in the aforementioned S230.

[0104] It can be understood that during the execution of S230, the relationship between the values ​​of the driving parameters having a relationship (scenario value correspondence relationship) is actually determined under the constraints of the driving scenario to determine whether the corresponding relationship is satisfied. Therefore, S230 is to implement data screening at the relationship layer.

[0105] S240. Delete the value of each driving parameter in the second scene data set at the exceeding-standard collection time point to obtain a target scene data set corresponding to the candidate driving scene.

[0106] The exceeding-standard collection time point refers to the collection time point whose corresponding value in the second scene data set is the scene exceeding-standard value.

[0107] In this embodiment, after determining the time point of exceeding the standard collection, all values ​​in the second scene data set at the time point of exceeding the standard collection (including the values ​​of various driving parameters) can be deleted to obtain a set of driving data slices with deleted data summarized as the target scene data set corresponding to the candidate driving scene.

[0108] The over-standard collection time point refers to the time when the corresponding value (including the value of each driving parameter) exceeds the value of the scene value index of the candidate driving scene. In this application, the scene value index corresponding to different driving scenes may be different, for example, the scene value index corresponding to the turning scene and the straight driving scene may be different. The scene value index under the driving scene is used to indicate the respective values ​​of each driving parameter when the vehicle is driving normally under the driving scene.

[0109] In some embodiments, a data vector can be constructed based on the values ​​of each driving parameter at the same collection time point in the fifth driving data slice to obtain multiple data vectors corresponding to the fifth driving data slice; the fifth driving data slice is each driving data slice corresponding to any collection time period in the second scene data set; one collection time point in the fifth driving data slice corresponds to a data vector; the multiple data vectors corresponding to the fifth driving data slice are clustered to obtain the cluster center corresponding to the fifth driving data slice; the average distance between the cluster center corresponding to the fifth driving data slice and the multiple data vectors corresponding to the fifth driving data slice is determined; the candidate scene value index corresponding to the candidate driving scene is determined based on the average distance; for any one of the multiple collection time points corresponding to the fifth driving data slice, if the distance between the data vector corresponding to the collection time point and the cluster center corresponding to the fifth driving data slice is greater than the candidate scene value index, the collection time point is determined to be an over-standard collection time point.

[0110] For the convenience of description, each driving data slice corresponding to any collection time period in the second scene data set can be taken as the fifth driving data slice, and each collection time period determines its own fifth driving data slice; for example, the driving parameters involve acceleration, vehicle speed and displacement, and the driving data slice d1 of the vehicle speed corresponding to the collection time period c1 in the second scene data set, the driving data slice d3 of the acceleration and the driving data slice d3 corresponding to the displacement, the driving data slice d1, the driving data slice d2 and the driving data slice d3 are all obtained as the fifth driving data slice.

[0111] Then, the values ​​of the driving parameters at the same collection time point in the fifth driving data slice are used to construct a vector indicating the values ​​of the driving parameters as the data vector at the collection time point, and one data vector corresponds to each collection time point of the fifth driving data slice. For example, the driving parameters involve acceleration, vehicle speed, and displacement, and the collection time end c2 corresponding to the fifth driving data slice includes 100 collection time points. For each of the 100 collection time points, a vector indicating the values ​​of acceleration, vehicle speed, and displacement is constructed according to the values ​​of acceleration, vehicle speed, and displacement at the collection time point as the data vector at the collection time point, thereby constructing 100 data vectors.

[0112] In the present application, the values ​​of each driving parameter at the same collection time point in the fifth driving data slice can be encoded through a deep learning pre-training model (such as LSTM (Long Short-Term Memory)) to obtain an encoded vector as a data vector.

[0113] Afterwards, the K-means clustering method can be used based on the vector distance between data vectors (such as cosine distance, Euclidean distance, etc.) to determine the cluster center corresponding to the fifth driving data piece (including K cluster centers, K can be set based on demand, and is not limited in this application, for example, K is 10); for each collection time period, determine the cluster center corresponding to the fifth driving data piece in the collection time period.

[0114] For the fifth driving data slice in each collection time period, the vector distances between each data vector in the fifth driving data slice and the corresponding cluster center are averaged to obtain the average distance corresponding to each data vector involved in the fifth driving data slice. Afterwards, the candidate scene value index corresponding to the candidate driving scene to which the fifth driving data slice belongs can be determined based on the average distance. For example, half of the average distance can be determined as the candidate scene value index corresponding to the candidate driving scene.

[0115] Afterwards, for any one of the multiple collection time points corresponding to the fifth driving data slice, if the distance between the data vector corresponding to the collection time point and the cluster center corresponding to the fifth driving data slice is greater than the candidate scene value index, it is determined that the corresponding value of the collection time point (including the value of each driving parameter at the collection time point) exceeds the scene value index, and the collection time point is further determined to be an over-standard collection time point, and then the value of each driving parameter at the over-standard collection time point in the second scene data set corresponding to the candidate driving scene is deleted to obtain the target scene data set corresponding to the candidate driving scene.

[0116] For any one of the multiple collection time points corresponding to the fifth driving data slice, if the distance between the data vector corresponding to the collection time point and the cluster center corresponding to the fifth driving data slice is not greater than the candidate scene value index, it is determined that the corresponding value of the collection time point (including the value of each driving parameter at the collection time point) does not exceed the scene value index, and it is further determined that the collection time point is not an excessive collection time point, and then the value of each driving parameter in the second scene data set corresponding to the candidate driving scene at the non-excessive collection time point is retained.

[0117] It is worth mentioning that each driving scene can be used as a candidate driving scene. After deleting the data in the aforementioned S220-S240 method, the target scene data set corresponding to the candidate driving scene is obtained, thereby realizing data screening of the driving data slices under each driving scene.

[0118] In addition, in the present application, after determining the time point of exceeding the standard collection, the values ​​of each driving parameter at the time point of exceeding the standard collection can be analyzed to determine the reason why the value of each driving parameter at the time point of exceeding the standard collection is abnormal. For example, the reason may be an error in data collection, a failure of the collection equipment, or other external factors; so that after determining the reason, the scene exceeding standard law can be summarized according to the reason, so as to directly determine the vehicle failure, etc. by using the scene exceeding standard law.

[0119] It can be understood that in the process of executing S240, the values ​​of driving parameters are actually statistically analyzed under the constraints of the driving scenario to obtain the scene value index, so as to filter out the excessive values ​​through the scene value index and further determine the time point of excessive collection. Therefore, S240 is to realize data screening at the statistical layer.

[0120] In summary, S220-S240 actually implements data screening from the parameter layer, relationship layer and statistical layer respectively under the constraints of the formal scene to obtain the final target scene data set.

[0121] After screening, the data without problems (that is, the target scenario data set) can be classified as valid data (which can be stored in the database corresponding to each driving scenario) for model training. At the same time, the problematic data (such as the value at the time of exceeding the standard collection) can be used to summarize the rules to obtain the exceeding standard rules of the aforementioned scenarios, which can be used for subsequent troubleshooting, etc.

[0122] For the data without problems after screening (that is, the target scene data set), a certain proportion of the data can be extracted regularly, and after the data is checked (which can be checked manually or automatically according to the aforementioned process S220-S240), it is transmitted to the model training engine corresponding to the driving scene for retraining of the model under the driving scene.

[0123] In this embodiment, the data processing process is as follows: Figure 3 As shown, in Figure 3 The original data includes multiple driving data slices to be processed. The original data is checked by the aforementioned S120-S140 method to determine abnormal data (all data deleted by the aforementioned S120-S140 method can be regarded as abnormal data), and the abnormal data is deleted. At the same time, the problem data (driving data slices with excessive values) can be determined, and the pattern of exceeding the standard can be summarized through the problem data.

[0124] For the data pieces that are determined to have abnormal values, data correction can also be performed, and the corrected data can be put into the original data and the screening steps S130 - S140 can be re-executed to obtain the target data set.

[0125] Afterwards, the data in the target data set can be divided to obtain scene data (including a scene data set for each driving scene), and a scene check can be performed on the scene data (screened according to the aforementioned steps S220-S240) to obtain a target scene data set for each driving scene, and then the target scene data set for each driving scene can be stored in a corresponding scene database to facilitate iterative optimization of the machine learning model based on the data in the scene database.

[0126] In this embodiment, after obtaining the target data set, it can also be divided into scene data sets for each driving scenario, and then the scene data sets for each driving scenario are deleted in turn, including driving data slices with abnormal values ​​of the candidate scenes, driving data slices that do not satisfy the corresponding relationship of the candidate scene values, and values ​​at the time point of exceeding the standard collection, to obtain the target scene data set corresponding to each driving scene, thereby realizing multi-dimensional and comprehensive screening of the data for each driving scenario, so that the values ​​in each driving data slice in the target scene data set obtained after screening are more accurate and effective, thereby improving the accuracy of each driving data slice in the target scene data set, thereby further improving the recognition effect of the machine learning model trained according to the values ​​in each driving data slice in the target scene data set.

[0127] Secondly, by setting specific candidate scene outlier conditions to screen out driving data slices with scene outliers, the determined driving data slices with scene outliers are more accurate, which improves the accuracy of the first scene data set after deleting the driving data slices with scene outliers, thereby improving the accuracy of the determined target scene data set.

[0128] In addition, the candidate scene value index is determined by determining the vector distance from the cluster center, and the over-standard collection time point is determined by the candidate scene value index. The determined candidate scene value index accurately indicates the normal level of the value of the candidate driving scene, thereby improving the accuracy of the over-standard collection time point determined according to the candidate scene value index, thereby making the target scene data set obtained after deleting the value at the over-standard collection time point more accurate.

[0129] See attached Figure 4 , Figure 4 The structure block diagram of a data processing device proposed in one embodiment of the present application is shown. For electronic equipment, the device 800 includes:

[0130] An acquisition module 810 is used to acquire a plurality of driving data slices to be processed, wherein a driving data slice includes a value of a driving parameter at at least one acquisition time point within a acquisition period;

[0131] A first deleting module 820, configured to delete a driving data slice having an abnormal value from the plurality of driving data slices to obtain a first data set;

[0132] The second deletion module 830 is used to delete the multiple first driving data pieces from the first data set to obtain a second data set if the values ​​between the multiple first driving data pieces do not satisfy the value correspondence relationship; the multiple first driving data pieces include driving data pieces corresponding to the multiple associated driving parameters in the first data set in the same collection period; the multiple associated driving parameters refer to driving parameters that have corresponding values ​​in the first data set and have a value correspondence relationship;

[0133] The third deletion module 840 is used to delete the second driving data slice from the second data set to obtain the target data set if the proportion of the exceeded values ​​in the second driving data slice exceeds the first threshold; the second driving data slice is any driving data slice in the second data set.

[0134] Optionally, the first deletion module 820 is also used to determine whether the driving data slice has an abnormal value if the driving data slice satisfies the abnormal value condition; the abnormal value condition includes at least one of the following: there are missing values ​​in the driving data slice; there are mutation values ​​in the driving data slice; there are non-differentiable values ​​in the driving data slice; there are values ​​in the driving data slice whose positive and negative signs do not match the standard positive and negative signs of the driving parameters corresponding to the driving data slice.

[0135] Optionally, the third deletion module 830 is also used to determine the interquartile range corresponding to the second driving data slice based on the values ​​in the second driving data slice; determine the standard score corresponding to each value in the second driving data slice based on the interquartile range; for any value in the second driving data slice, if the absolute value of the standard score of the value is greater than the second threshold, determine that the value is an exceeded value.

[0136] Optionally, the device also includes a scene data processing module, which is used to divide the driving data slices in the target data set to obtain scene data sets under multiple driving scenarios; each scene data set includes at least one driving data slice in the target data set; the driving data slices with candidate scene outliers are deleted from the candidate scene data set corresponding to the candidate driving scene to obtain a first scene data set corresponding to the candidate driving scene; the candidate driving scene is any driving scene; the candidate scene outlier refers to the scene outlier value corresponding to the candidate driving scene; if the values ​​between the multiple third driving data slices do not satisfy the candidate scene value correspondence relationship, the multiple third driving data slices are deleted from the first scene data set to obtain the candidate driving scene. A second scene data set corresponding to the scene; multiple third driving data slices refer to the driving data slices corresponding to multiple scene-related driving parameters in the first scene data set in the same collection period; multiple scene-related driving parameters refer to driving parameters that have corresponding values ​​in the first scene data set and have candidate scene value correspondences; the candidate scene value correspondences are the scene value correspondences corresponding to the candidate driving scenes; different driving scenes correspond to their respective scene value correspondences; the values ​​of each driving parameter in the second scene data set at the exceeded collection time point are deleted to obtain the target scene data set corresponding to the candidate driving scene; the exceeded collection time point refers to the collection time point whose corresponding value in the second scene data set is the scene exceeded value.

[0137] Optionally, the scene data processing module is also used to determine that the fourth driving data slice has a candidate scene outlier if the value in the fourth driving data slice meets the candidate scene outlier condition corresponding to the candidate driving scene; the fourth driving data slice is any driving data slice in the candidate scene data set; wherein, satisfying the candidate scene outlier condition includes at least one of the following: the fourth driving data slice contains a value that exceeds the reference scene standard value; the reference scene standard value refers to the standard value of the driving parameter to which the fourth driving data slice belongs in the candidate driving scene; the fourth driving data slice contains a value whose positive and negative sign after derivative is inconsistent with the positive and negative sign of the reference scene standard; the reference scene standard positive and negative sign refers to the scene standard positive and negative sign of the driving parameter to which the fourth driving data slice belongs in the candidate driving scene.

[0138] Optionally, the scene data processing module is also used to construct a data vector based on the values ​​of each driving parameter at the same collection time point in the fifth driving data slice, and obtain multiple data vectors corresponding to the fifth driving data slice; the fifth driving data slice is each driving data slice corresponding to any collection time period in the second scene data set; one collection time point in the fifth driving data slice corresponds to a data vector; clustering the multiple data vectors corresponding to the fifth driving data slice to obtain the cluster center corresponding to the fifth driving data slice; determining the average distance between the cluster center corresponding to the fifth driving data slice and the multiple data vectors corresponding to the fifth driving data slice; determining the candidate scene value index corresponding to the candidate driving scene based on the average distance; for any one of the multiple collection time points corresponding to the fifth driving data slice, if the distance between the data vector corresponding to the collection time point and the cluster center corresponding to the fifth driving data slice is greater than the candidate scene value index, determine that the collection time point is an over-standard collection time point.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0140] In addition, each function in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software function module.

[0141] In addition, each function in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software function module.

[0142] Please refer to Figure 5 , which shows a structural block diagram of an electronic device provided according to an embodiment of the present application. The electronic device 500 can be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, and a vehicle. The electronic device 500 in the present application may include one or more of the following components: a processor 510, a memory 520, and one or more applications. One or more applications may be stored in the memory 520 and configured to be executed by one or more processors 510, and one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0143] The processor 510 may include one or more processing cores. The processor 510 uses various interfaces and lines to connect various parts of the entire electronic device 500, and executes various functions and processes data of the electronic device 500 by running or executing instructions, programs, code sets or instruction sets stored in the memory 520, and calling data stored in the memory 520. Optionally, the processor 510 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 510 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 510, but may be implemented separately through a communication chip.

[0144] The memory 520 may include a random access memory (RAM) or a read-only memory (ROM). The memory 520 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 520 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 500 during use.

[0145] In addition, each function in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software function module.

[0146] On the other hand, the present application also provides a computer-readable storage medium, in which program code is stored. The program code can be called by a processor to execute the method described in the above method embodiment.

[0147] The computer readable storage medium can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a cluster of ROMs. Optionally, the computer readable storage medium includes a non-transitory computer-readable storage medium. The computer readable storage medium has storage space for program codes that execute any of the method steps in the above method. These program codes can be read from or written into one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire a plurality of driving data slices to be processed, wherein one driving data slice includes a value of a driving parameter at at least one collection time point within a collection period; Deleting the driving data slices with abnormal values ​​from the multiple driving data slices to obtain a first data set; If the values ​​between the multiple first driving data pieces do not satisfy the value correspondence relationship, the multiple first driving data pieces are deleted from the first data set to obtain a second data set; the multiple first driving data pieces include driving data pieces corresponding to the multiple associated driving parameters in the first data set in the same collection period; the multiple associated driving parameters refer to driving parameters that have corresponding values ​​in the first data set and have the value correspondence relationship; If the proportion of exceeded values ​​in the second driving data slice exceeds the first threshold, the second driving data slice is deleted from the second data set to obtain a target data set; the second driving data slice is any driving data slice in the second data set.

2. The method according to claim 1, characterized in that Before deleting the driving data slices with abnormal values ​​from the plurality of driving data slices to obtain the first data set, the method further includes: If the driving data slice meets the abnormal value condition, determining that the driving data slice has an abnormal value; The outlier condition includes at least one of the following: The driving data slice has missing values; The driving data slice has a mutation value; There are non-differentiable values ​​in the driving data slice; The driving data slice contains a value whose sign does not match the standard sign of the driving parameter corresponding to the driving data slice.

3. The method according to claim 1, characterized in that Not satisfying the value correspondence relationship includes at least one of the following: The values ​​of the plurality of first driving data pieces are not in a calculus relationship with each other; The values ​​between the plurality of first driving data pieces cannot be converted according to a preset rule.

4. The method according to claim 1, characterized in that: If the proportion of the exceeded values ​​in the second driving data slice exceeds the first threshold, the second driving data slice is deleted from the second data set, and before obtaining the target data set, the method further includes: determining, according to the value in the second driving data slice, an interquartile range corresponding to the second driving data slice; Determining, according to the interquartile range, a standard score corresponding to each value in the second driving data slice; For any value in the second driving data slice, if the absolute value of the standard score of the value is greater than a second threshold, the value is determined to be an out-of-standard value.

5. The method according to claim 1, characterized in that If the proportion of the exceeded values ​​in the second driving data slice exceeds the first threshold, the second driving data slice is deleted from the second data set, and after the target data set is obtained, the method further includes: Dividing the driving data slices in the target data set to obtain scene data sets under multiple driving scenes; each of the scene data sets includes at least one driving data slice in the target data set; Deleting a driving data piece having a candidate scene outlier value from a candidate scene data set corresponding to a candidate driving scene, to obtain a first scene data set corresponding to the candidate driving scene; the candidate driving scene is any one of the driving scenes; the candidate scene outlier value refers to a scene outlier value corresponding to the candidate driving scene; If the values ​​between the multiple third driving data slices do not satisfy the candidate scene value correspondence relationship, the multiple third driving data slices are deleted from the first scene data set to obtain the second scene data set corresponding to the candidate driving scene; the multiple third driving data slices refer to the driving data slices corresponding to the multiple scene-associated driving parameters in the first scene data set in the same collection period; the multiple scene-associated driving parameters refer to the driving parameters that have corresponding values ​​in the first scene data set and have the candidate scene value correspondence relationship; the candidate scene value correspondence relationship is the scene value correspondence relationship corresponding to the candidate driving scene; different driving scenes correspond to their respective scene value correspondence relationships; The values ​​of each driving parameter in the second scene data set at the exceeding-standard collection time point are deleted to obtain the target scene data set corresponding to the candidate driving scene; the exceeding-standard collection time point refers to the collection time point whose corresponding value in the second scene data set is the scene exceeding-standard value.

6. The method according to claim 5, characterized in that Before deleting the driving data slices containing the candidate scene outliers from the candidate scene data set corresponding to the candidate driving scene to obtain the first scene data set corresponding to the candidate driving scene, the method further comprises: If the value in the fourth driving data slice meets the candidate scene outlier condition corresponding to the candidate driving scene, it is determined that the fourth driving data slice has a candidate scene outlier; the fourth driving data slice is any one of the driving data slices in the candidate scene data set; The candidate scene outlier condition is satisfied including at least one of the following: The fourth driving data slice contains a value exceeding a reference scene standard value; the reference scene standard value refers to a standard value of the driving parameter to which the fourth driving data slice belongs in the candidate driving scene; The fourth driving data slice contains a value whose sign after differentiation is inconsistent with the sign of the reference scene standard; the reference scene standard sign refers to the scene standard sign of the driving parameter to which the fourth driving data slice belongs in the candidate driving scene.

7. The method according to claim 5, characterized in that Before deleting the value of each driving parameter in the second scene data set at the exceeding-standard collection time point to obtain the target scene data set corresponding to the target driving scene, the method further includes: A data vector is constructed according to the values ​​of each driving parameter at the same collection time point in the fifth driving data slice, and a plurality of data vectors corresponding to the fifth driving data slice are obtained; the fifth driving data slice is each driving data slice corresponding to any collection period in the second scene data set; and one collection time point in the fifth driving data slice corresponds to one data vector; Clustering a plurality of data vectors corresponding to the fifth driving data slice to obtain a cluster center corresponding to the fifth driving data slice; Determine an average distance between a cluster center corresponding to the fifth driving data piece and a plurality of data vectors corresponding to the fifth driving data piece; Determine a candidate scene value index corresponding to the candidate driving scene based on the average distance; For any one of the multiple collection time points corresponding to the fifth driving data slice, if the distance between the data vector corresponding to the collection time point and the cluster center corresponding to the fifth driving data slice is greater than the candidate scene value index, the collection time point is determined to be an out-of-standard collection time point.

8. A data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a plurality of driving data slices to be processed, wherein a driving data slice includes a value of a driving parameter at at least one acquisition time point within a acquisition period; A first deleting module, configured to delete the driving data slices with abnormal values ​​from the plurality of driving data slices to obtain a first data set; A second deletion module is configured to delete the plurality of first driving data pieces from the first data set to obtain a second data set if the values ​​between the plurality of first driving data pieces do not satisfy the value correspondence relationship; the plurality of first driving data pieces include driving data pieces corresponding to the plurality of associated driving parameters in the first data set in the same collection period; the plurality of associated driving parameters refer to driving parameters that have corresponding values ​​in the first data set and have the value correspondence relationship; The third deletion module is used to delete the second driving data slice from the second data set to obtain the target data set if the proportion of the exceeded values ​​in the second driving data slice exceeds the first threshold; the second driving data slice is any one driving data slice in the second data set.

9. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the method according to any one of claims 1 to 7.

Images