Test method, device, equipment, medium and product

Abstract

The invention discloses a test method, device and equipment, a medium and a product. The method comprises the following steps: acquiring original CAN bus data acquired by a real vehicle road side; the original CAN bus data are screened based on a to-be-measured algorithm, a positive scene slice set and a negative scene slice set of at least one driving scene corresponding to the to-be-measured algorithm are obtained, scene slices in the positive scene slice set are correctly recognized based on the to-be-measured algorithm, and scene slices in the negative scene slice set are correctly recognized based on the to-be-measured algorithm. The scene slices in the negative scene slice set are scene slices with errors identified based on a to-be-measured algorithm; sending input data and an output result of a to-be-measured algorithm corresponding to the negative scene slice set to a target terminal, and receiving an updated to-be-measured algorithm fed back by the target terminal; verifying the updated to-be-measured algorithm based on the positive scene slice set and the negative scene slice set of the at least one driving scene; and if the verification is passed, replacing the to-be-measured algorithm deployed on the real vehicle with the updated to-be-measured algorithm.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of vehicle technology, and in particular to a testing method, apparatus, equipment, medium and product. Background Technology

[0002] In recent years, the intelligent driving vehicle industry has developed rapidly, with increasingly mature development technologies. Internet companies and automakers have invested heavily in R&D in the field of intelligent driving. However, traditional real-vehicle testing faces two major challenges: first, high testing costs and long cycles, with difficulty in efficiently reproducing problematic conditions and extreme scenarios after software updates; second, the sheer volume of real-vehicle road test data, and although data backfeeding has become a research focus for automakers and Tier 1 suppliers, the coverage and application efficiency of data backfeeding still need optimization. Against this backdrop, how to fully, efficiently, and purposefully utilize raw real-vehicle road test data to conduct intelligent driving function evaluations has become a key technical problem that the industry urgently needs to overcome. Summary of the Invention

[0003] This invention provides a testing method, apparatus, device, medium, and product for testing algorithms under test.

[0004] According to one aspect of the present invention, a testing method is provided, comprising:

[0005] Acquire raw CAN bus data collected from the roadside of a real vehicle, wherein the algorithm to be tested is deployed on the real vehicle;

[0006] The original CAN bus data is filtered based on the algorithm under test to obtain a positive scene slice set and a negative scene slice set for at least one driving scenario corresponding to the algorithm under test. The scene slices in the positive scene slice set are the scene slices correctly identified based on the algorithm under test, and the scene slices in the negative scene slice set are the scene slices incorrectly identified based on the algorithm under test.

[0007] The input data and output results of the algorithm to be tested corresponding to the negative scene slice set are sent to the target terminal, and the updated algorithm to be tested is received from the target terminal.

[0008] The updated algorithm to be tested is verified based on the positive scene slice set and the negative scene slice set of the at least one driving scenario;

[0009] If the verification passes, the algorithm under test deployed on the actual vehicle will be replaced with the updated algorithm under test.

[0010] According to another aspect of the present invention, a testing apparatus is provided, the testing apparatus comprising:

[0011] The acquisition module is used to acquire raw CAN bus data collected from the roadside of the actual vehicle, wherein the algorithm to be tested is deployed on the actual vehicle;

[0012] The filtering module is used to filter the original CAN bus data based on the algorithm under test to obtain a positive scene slice set and a negative scene slice set for at least one driving scenario corresponding to the algorithm under test. The scene slices in the positive scene slice set are scene slices correctly identified based on the algorithm under test, and the scene slices in the negative scene slice set are scene slices incorrectly identified based on the algorithm under test.

[0013] The sending module is used to send the input data and output results of the algorithm under test corresponding to the negative scene slice set to the target terminal, and to receive the updated algorithm under test fed back by the target terminal.

[0014] The verification module is used to verify the updated algorithm under test based on the positive scene slice set and the negative scene slice set of the at least one driving scenario;

[0015] The replacement module is used to replace the algorithm under test deployed on the actual vehicle with the updated algorithm under test if the verification passes.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the test method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the test method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the testing method as described in any of the embodiments of the present invention.

[0022] This invention acquires raw CAN bus data collected from a roadside vehicle, where an algorithm under test is deployed. Based on the algorithm under test, the raw CAN bus data is filtered to obtain a positive scene slice set and a negative scene slice set corresponding to at least one driving scenario. The scene slices in the positive scene slice set are correctly identified by the algorithm under test, and the scene slices in the negative scene slice set are incorrectly identified by the algorithm under test. The input data and output results of the algorithm under test corresponding to the negative scene slice set are sent to a target terminal, and the updated algorithm under test is received from the target terminal. The updated algorithm under test is verified based on the positive and negative scene slice sets of the at least one driving scenario. If the verification is successful, the algorithm under test deployed on the vehicle is replaced with the updated algorithm under test, thus enabling the testing of the algorithm under test.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a testing method in an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of a method for constructing a scene slice library for evaluating and testing intelligent driving functions according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of a testing device according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0032] Example 1

[0033] Figure 1 This is a flowchart of a testing method provided in an embodiment of the present invention. This embodiment is applicable to testing algorithms under test. The method can be executed by the testing device in this embodiment, which can be implemented in software and / or hardware, such as... Figure 1 As shown, the method specifically includes the following steps:

[0034] S110 acquires raw CAN bus data collected from the roadside of the actual vehicle.

[0035] In this embodiment, the algorithm to be tested is deployed on the actual vehicle. The algorithm to be tested can be any one of the following: perception fusion algorithm, sensor perception algorithm, localization and mapping algorithm, behavior decision algorithm, path planning algorithm, trajectory planning algorithm, longitudinal control algorithm, lateral control algorithm, and chassis coordination control algorithm.

[0036] In this embodiment, the raw CAN bus data collected by the roadside of the actual vehicle includes: the input data and output results of each algorithm under test deployed on the vehicle.

[0037] S120, based on the algorithm under test, the original CAN bus data is filtered to obtain a positive scene slice set and a negative scene slice set corresponding to at least one driving scenario of the algorithm under test.

[0038] In this embodiment, the scene slices in the positive scene slice set are scene slices that are correctly identified based on the algorithm under test, and the scene slices in the negative scene slice set are scene slices that are incorrectly identified based on the algorithm under test.

[0039] In this embodiment, the method of filtering the original CAN bus data based on the algorithm under test to obtain the positive scene slice set and negative scene slice set of at least one driving scenario corresponding to the algorithm under test can be as follows: obtain the target output result corresponding to each driving scenario corresponding to the algorithm under test, and filter the original CAN bus data based on the target output result corresponding to each driving scenario to obtain the positive scene slice set and negative scene slice set of at least one driving scenario corresponding to the algorithm under test.

[0040] In a specific example, the algorithm to be tested is a perception fusion algorithm, and the driving scenarios corresponding to the perception fusion algorithm include: the scenario of a vehicle cutting in front, the scenario of a vehicle stopping in front, and the scenario of a vehicle cutting out in front.

[0041] Optionally, the raw CAN bus data is filtered based on the algorithm under test to obtain a positive scene slice set and a negative scene slice set for at least one driving scenario corresponding to the algorithm under test, including:

[0042] Obtain the target output results corresponding to each driving scenario of the algorithm under test.

[0043] In this embodiment, the target output result is the expected result corresponding to each driving scenario of the algorithm under test. For example, the algorithm under test may be a perception fusion algorithm, and the driving scenarios corresponding to the perception fusion algorithm include: a scenario where a vehicle cuts in from the front. The target output result corresponding to the scenario where a vehicle cuts in from the front is CutInStatus = 1. Here, CutInStatus is the vehicle cutting status.

[0044] Based on the target output results corresponding to each driving scenario, the original CAN bus data is filtered to obtain scenario slices for each driving scenario.

[0045] In this embodiment, the method for filtering the original CAN bus data based on the target output results corresponding to each driving scenario to obtain scenario slices for each driving scenario can be as follows: extract the times in the original CAN bus data where the output result is the target output result to obtain a time list; filter the original CAN bus data based on each time in the time list to obtain the CAN bus data corresponding to each time in the time list; and generate scenario slices for each driving scenario based on the CAN bus data corresponding to each time in the time list.

[0046] Obtain the video data corresponding to each scene slice.

[0047] In this embodiment, the method for obtaining video data corresponding to each scene slice can be as follows: continuously collect video data through the camera, query the above video data, obtain the video data corresponding to each time in the time list, and use the video data corresponding to each time in the time list as the video data corresponding to each scene slice.

[0048] The video data corresponding to the scene slices of each driving scenario are identified to obtain the identification results of the scene slices of each driving scenario.

[0049] In this embodiment, the method for identifying the video data corresponding to the scene slices of each driving scenario and obtaining the identification results of the scene slices of each driving scenario can be as follows: based on the target output results corresponding to each driving scenario, the video data corresponding to the scene slices of each driving scenario is identified to obtain the identification results of the scene slices of each driving scenario.

[0050] In a specific example, the algorithm under test is a perception fusion algorithm. The driving scenario corresponding to the perception fusion algorithm includes: a vehicle cutting in from the front. The target output result corresponding to the vehicle cutting in from the front scenario is CutInStatus = 1. All moments when CutInStatus = 1 are extracted to generate a cutting time table. If the signal remains 1, only the first moment when CutInStatus = 1 is retained. Then, based on these moments, the chassis data, radar data, and camera data are extracted for 5 seconds before and after each moment to form scene slices, which are stored in a scene slice library for subsequent analysis. Video data corresponding to each moment is acquired, and the video data is identified to determine whether a vehicle cutting in from the front actually exists in the video, obtaining the identification result, which includes: a vehicle cutting in from the front exists or a vehicle cutting in from the front does not exist.

[0051] Based on the recognition results of scene slices for each driving scenario and the target output results, the scene slices for each driving scenario are divided to obtain a positive scene slice set and a negative scene slice set.

[0052] In this embodiment, the scene slices of each driving scenario are divided into positive scene slice sets and negative scene slice sets based on the recognition results of the scene slices of each driving scenario and the target output results. The method is as follows: scene slices whose recognition results are the same as the target output results are added to the positive scene slice set, and scene slices whose recognition results are different from the target output results are added to the negative scene slice set.

[0053] In this embodiment, the recognition results of scene slices in the positive scene slice set are the same as the target output result, while the recognition results of scene slices in the negative scene slice set are different from the target output result.

[0054] In this embodiment, taking a driving scenario as an example of a vehicle cutting in from the front, the scene slices are compared with the video data at the corresponding time. Based on whether a vehicle actually cuts in from the front in the video, the scene slices are divided into a positive scene slice set and a negative scene slice set. Simultaneously, the input data and output results of the algorithm under test that fails to identify the error are recorded and fed back to the R&D personnel for algorithm modification.

[0055] In a specific example, the data in the scene slice library is compared with the corresponding video clips to accurately determine whether a vehicle actually cuts into the video frame. Based on the verification results, scene slices in the video frame where a vehicle actually cuts into the frame are categorized into the positive scene slice set; while scene slices in the video frame where no vehicle cutting into the frame is shown are uniformly included in the negative scene slice set, thereby achieving accurate classification and effective management of scene data. Simultaneously, detailed logs of the algorithm's operation and output deviation data under erroneous scenarios are recorded to facilitate subsequent feedback to the R&D team for algorithm modification.

[0056] In this embodiment of the invention, a systematic scene slice data system is constructed. Through standardized collection and classification technology, positive scene slice sets and negative scene slice sets are established to achieve in-depth mining and refined management of erroneous scene data in real vehicle road test data. This effectively releases the data value that is ignored in traditional processes and significantly improves the efficiency of data resource utilization.

[0057] In terms of functional evaluation, this invention not only applies to perception fusion algorithms but comprehensively covers the evaluation and testing of all functions involved in intelligent driving systems, including perception, decision-making, and control. Regarding the construction of the scene slice library, it is not limited to specific entry scene slices but fully encompasses all scene types, including various traffic environments, road conditions, and weather conditions, that can be labeled and recognized by intelligent driving algorithms. Based on actual scene classification, different scene slice libraries are created, including but not limited to entry scene slice libraries, exit scene slice libraries, overtaking scene slice libraries, merging scene slice libraries, and emergency braking scene slice libraries. Within each scene slice library, based on the comparison between the recognition judgment and the judgment results from real video, it is further divided into positive scene slice sets and negative scene slice sets.

[0058] Optionally, based on the target output results corresponding to each driving scenario, the original CAN bus data is filtered to obtain scene slices for each driving scenario and video data corresponding to each scene slice, including:

[0059] The original CAN bus data is filtered based on the input data corresponding to the algorithm under test to obtain the first CAN bus data.

[0060] In this embodiment, the method of filtering the original CAN bus data according to the input data corresponding to the algorithm under test to obtain the first CAN bus data can be as follows: obtain the data type of the input data corresponding to the algorithm under test, filter the original CAN bus data according to the data type of the input data corresponding to the algorithm under test, and obtain the first CAN bus data.

[0061] Obtain the target data format corresponding to the algorithm under test.

[0062] In this embodiment, the target data format corresponding to the algorithm under test is a data format that can be read by the algorithm under test.

[0063] The first CAN bus data is converted into the target data format to obtain the second CAN bus data.

[0064] In this embodiment, data frame interpolation can also be performed according to the actual data situation.

[0065] Based on the target output results corresponding to each driving scenario, the second CAN bus data is filtered to obtain scenario slices for each driving scenario.

[0066] Obtain the video data corresponding to each scene slice of each driving scenario.

[0067] In this embodiment, the raw CAN bus data is processed, which mainly includes filtering chassis signals, radar signals and camera signals according to the actual input signal requirements of the algorithm under test, converting them into a data format that the algorithm under test can read, and performing data frame interpolation processing according to the actual data situation.

[0068] Optionally, the at least one driving scenario includes: a scenario where a vehicle cuts in from the front, and the target output result corresponding to the scenario where a vehicle cuts in from the front is: the status signal of the vehicle cutting in from the front is a preset value.

[0069] In this embodiment, the preset value can be 1.

[0070] In this embodiment, taking the perception fusion algorithm as an example, the cut-in scene data is filtered according to the definition of the input data and output results of the perception fusion algorithm. Taking the perception fusion output signal CutInStatus as an example, this signal is used to indicate the cut-in status of the vehicle in front: when CutInStatus = 0, it means the signal is invalid; when CutInStatus = 1, it means the vehicle in front is cutting in; when CutInStatus = 2, it means the vehicle in front is cutting out; when CutInStatus = 3, it means there is no cut-in or cut-out action.

[0071] Based on the target output results corresponding to each driving scenario, the original CAN bus data is filtered to obtain scenario slices for each driving scenario, including:

[0072] Extract the times when the forward vehicle cut-in status signal in the original CAN bus data is a preset value to obtain a time list.

[0073] Based on each time point in the time list, the original CAN bus data is filtered to obtain the chassis data, radar data, and camera data corresponding to each time point in the time list.

[0074] In this embodiment, the data types of the input data corresponding to the algorithm under test include: chassis data, radar data, and camera data; the original CAN bus data is filtered according to the data types of the input data corresponding to the algorithm under test and the times in the time list to obtain the chassis data, radar data, and camera data corresponding to each time in the time list.

[0075] Based on the chassis data, radar data, and camera data corresponding to each moment, a scene slice of the scene where the vehicle ahead cuts in is generated.

[0076] In this embodiment, a scene slice of the scene of the vehicle ahead cutting in is generated based on the chassis data, radar data, and camera data corresponding to each time in the time list of the original CAN bus data.

[0077] In this embodiment, all moments when CutInStatus = 1 are first extracted from the original CAN bus data to generate a cut-in timetable. If the signal remains at 1, only the first moment when CutInStatus = 1 is retained. Then, based on these moments, the chassis data, radar data, and camera data are extracted for 5 seconds before and after each moment to form scene slices, which are stored in the cut-in scene slice library for subsequent analysis. The scene slice library stores scenes selected based on markers that can be recognized by the intelligent driving algorithm. It is divided into different scene slice libraries according to the actual scene classification, including but not limited to cut-in scene slice library, cut-out scene slice library, overtaking scene slice library, merging scene slice library, and emergency braking scene slice library. Each scene slice library is divided into a positive scene slice set and a negative scene slice set based on the comparison between the recognition judgment and the judgment result of the real video.

[0078] This embodiment provides a method for constructing a scenario slice library for the evaluation and testing of intelligent driving functions, so as to make full use of the original data from real vehicle road tests, explore the scenarios that trigger misidentification by the intelligent driving system, and conduct a more comprehensive evaluation and testing of the intelligent driving algorithm.

[0079] S130, the input data and output results of the algorithm to be tested corresponding to the negative scene slice set are sent to the target terminal, and the updated algorithm to be tested is received from the target terminal.

[0080] S140, the updated algorithm to be tested is verified based on the positive scene slice set and the negative scene slice set of the at least one driving scenario.

[0081] In this embodiment, data back-injection technology is used to first import a positive scene slice set to verify the basic functions of the algorithm; then, a negative scene slice set is back-injected to verify the algorithm improvement effect and prevent the algorithm from being triggered erroneously.

[0082] In a specific example, the R&D team modified and optimized the algorithm based on issues raised during real-vehicle road tests and data back-injection feedback. Using data back-injection technology, scene slices from the positive scene slice library were imported into the updated perception fusion algorithm to verify whether the updated algorithm could correctly identify the scene being cut off. Then, the negative scene slice set was back-injected to test the improved effect of the updated perception fusion algorithm and prevent misidentification. If problems still arose after the back-injection, the issues were reported back to the R&D team, and the algorithm was modified again.

[0083] Optionally, the updated test algorithm is validated based on the positive and negative scene slice sets of the at least one driving scenario, including:

[0084] The scene slices in the positive scene slice set of the at least one driving scenario are sequentially input into the updated test algorithm to obtain the output results of the scene slices in the positive scene slice set of each driving scenario.

[0085] If the output results of the scene slices in the positive scene slice set of each driving scenario match the target output results corresponding to each driving scenario, then the scene slices in the negative scene slice set of at least one driving scenario are sequentially input into the updated test algorithm to obtain the output results of the scene slices in the negative scene slice set of each driving scenario.

[0086] The updated algorithm to be tested is verified based on the output results of the scene slices in the negative scene slice set of each driving scenario and the target output results corresponding to each driving scenario.

[0087] In a specific example, data back-injection technology is used to convert and reorganize the real-vehicle road test data from the positive scene slice set according to the algorithm input requirements, and then inject it into the perception fusion algorithm's runtime environment. During this process, it is recorded whether the algorithm's output cut-in signal matches the scene characteristics to determine if the perception fusion algorithm correctly identifies the cut-in scene. If it fails to identify correctly, the problem and related data are recorded and fed back to the R&D personnel for algorithm modification. Data back-injection is then carried out on the negative scene slice set. The negative scene slice set includes abnormal operating condition data such as algorithm recognition errors and misidentifications. This data is input into the perception fusion algorithm through an adapted back-injection process. By comparing the algorithm's output results with the actual scene conditions, the cut-in signal output is determined. If recognition errors occur, the algorithm's operation log and output deviation data under the erroneous scenario are recorded in detail, providing key improvement directions for subsequent targeted optimization. Based on the specific functional requirements of the perception fusion algorithm, a portion of data is selected from the scene slice library for targeted back-injection testing. For example, to verify the algorithm's response to specific targets (such as vehicles braking suddenly or pedestrians crossing the road), slice data containing such scenarios is selected. To evaluate the algorithm's performance in specific environments (such as nighttime or tunnels), slice sets corresponding to the operating conditions are extracted. By precisely back-injecting a subset of data, in-depth testing is conducted on the core functional modules of the algorithm, avoiding resource waste caused by redundant testing. If the output data does not meet the testing standards, the issue is reported back to the R&D personnel for modification.

[0088] This invention establishes a working condition consistency algorithm comparison mechanism. Based on the standardized processing results of scene slice sets, it constructs a unified algorithm testing benchmark environment, eliminates the interference of external variables on algorithm performance evaluation, and realizes accurate comparison of new and old algorithms under completely equivalent working conditions, providing reliable technical support for the scientific exploration of algorithm performance boundaries.

[0089] In this embodiment of the invention, a two-way cross-validation evaluation system is designed. Based on positive and negative scenario slice sets, a two-way data validation model is established to accurately define the application boundaries of real vehicle road test data and the applicable scenarios of the algorithm. By improving the scientificity and accuracy of algorithm iteration evaluation, the R&D cycle is effectively shortened, the cost of ineffective road tests is reduced, and an efficient and economical algorithm continuous optimization technology solution is formed.

[0090] S150, if the verification is successful, the algorithm under test deployed on the actual vehicle is replaced with the updated algorithm under test.

[0091] In this embodiment, after confirming that the back-injection results are all correct, real vehicle road test data is then back-injected according to requirements to test algorithm functionality and enrich the scene slice library.

[0092] In this embodiment, after the data back-injection results of both the positive and negative scene slice sets are verified to be correct, other real-vehicle road test data are selected for data back-injection based on the testing requirements of the perception fusion algorithm to verify the algorithm's operational function. During this process, S120 is executed synchronously and repeatedly, that is, the cut signal is selected from the back-injection results of the updated algorithm under test, and the signal is compared and identified with the corresponding video content. Based on the comparison results, the relevant scene slices are respectively assigned to the positive or negative scene slice set, thereby continuously expanding and enriching the data content of the scene slice library.

[0093] In this embodiment, the updated algorithm to be tested is integrated into the vehicle-side algorithm of the intelligent driving vehicle, and real vehicle road tests are conducted. S110 is repeated to continuously enrich and improve the scene slice library, providing reliable data support for the iteration of intelligent driving algorithms.

[0094] In this embodiment, the updated algorithm under test is integrated into the vehicle-side algorithm module of the intelligent driving vehicle, and multi-scenario real-vehicle road tests are conducted. The perception, decision-making, and control functions of the algorithm are comprehensively verified under different operating conditions, including urban roads, highways, and rural areas. During the test, step S110 is repeated to collect sensor data, algorithm logs, and vehicle status information in real time. After data processing and filtering of input signals, scene slices are generated and classified into positive and negative scene slice sets. The slice library is continuously expanded to increase its size and improve data coverage accuracy, providing reliable data support for the iterative optimization of the intelligent driving algorithm and enhancing its adaptability in complex scenarios.

[0095] In a specific example, the updated algorithm under test is integrated into the intelligent driving vehicle's on-board algorithm for real-world field and road testing. This achieves deep integration between the updated algorithm and the intelligent driving vehicle's on-board system. First, the algorithm is adapted and optimized based on the vehicle's hardware architecture (e.g., sensor interfaces, computing unit configuration) and software environment (operating system, communication protocols) to ensure seamless integration between the updated algorithm and the vehicle system. After integration, testing is conducted in phases: initially, low-speed, controllable scenario testing (e.g., fixed obstacle avoidance, simulated following) is performed in a closed environment to verify the stability of the updated algorithm's basic functions; then, the testing is gradually expanded to open roads, testing the algorithm's comprehensive processing capabilities for complex traffic flow, dynamic obstacles, and traffic rules in real-world scenarios such as urban traffic and highways. Vehicle operation data and algorithm output results are recorded throughout the testing process, providing empirical evidence for subsequent performance evaluation and problem analysis. Real-world road test data is processed to enrich the positive and negative scenario slice sets. Systematic processing and analysis are carried out based on the data collected from real-world road tests. The raw data undergoes data processing and filtering to extract signals, generating scene slices. These slices are then incorporated into positive and negative scene slice sets through video comparison and classification, enriching the scene diversity and data dimensions of the slice sets and providing richer, more targeted training and validation data for algorithm iteration. Researchers modify the algorithm based on feedback. Targeted optimizations are conducted based on algorithmic issues exposed during prior data back-injection and real-vehicle testing. First, a deep review of the recorded algorithm running logs, output deviation data, and scene slices is performed to pinpoint the root cause of the problem (e.g., algorithm model defects, improper parameter configuration, insufficient data coverage). If the problem originates from the algorithm model, the network structure is adjusted or the algorithm logic is optimized; if it is a parameter configuration issue, the parameters are recalibrated through a combination of simulation and field testing; if misjudgment is caused by insufficient data coverage, the corresponding scene slices are prioritized for addition to the training set. The modified algorithm undergoes further data back-injection verification and real-vehicle testing, forming a closed-loop iterative process of "problem analysis - algorithm optimization - effect verification," gradually improving algorithm performance and reliability.

[0096] Optionally, if the verification passes, the algorithm under test deployed on the actual vehicle is replaced with the updated algorithm under test, including:

[0097] If the output results of the scene slices in the negative scene slice set of each driving scenario match the target output results corresponding to each driving scenario, then the verification is successful.

[0098] Replace the algorithm under test deployed on the actual vehicle with the updated algorithm under test.

[0099] In a specific example, such as Figure 2 As shown, the method for constructing a scenario slice library for the evaluation and testing of intelligent driving functions involves the following specific steps:

[0100] Step 1: Analyze the raw CAN bus data collected from the actual vehicle road test and sort out the various signals.

[0101] Step 2: Filter the entry signals output by the perception fusion algorithm, accurately locate and slice the identified entry scene, generate entry scene slices, and store them in the entry scene slice library.

[0102] Step 3: Compare the cut-in scene slices with the corresponding time-time video data. Based on whether the cut-in scene actually exists in the video, divide them into positive scene slice sets and negative scene slice sets. Simultaneously, record the input data and output results of identification errors, and provide feedback to the R&D personnel so that the algorithm can be modified, resulting in an updated algorithm for testing.

[0103] Step 4: The updated algorithm under test is used to first import the positive scene slice set to verify the basic function of the algorithm using data back-injection technology; then the negative scene slice set is back-injected to check the effect of the algorithm improvement and prevent the algorithm from being triggered erroneously.

[0104] Step 5: After confirming that all backfill results are correct, backfill real vehicle road test data according to requirements to test algorithm functions and enrich the scene slice library.

[0105] Step 6: Integrate the tested algorithm into the intelligent driving vehicle for real-vehicle road testing. Repeat the above processing steps for newly collected data to continuously enrich and improve the scene slice library, providing reliable data support for the iteration of intelligent driving algorithms.

[0106] This invention constructs a systematic scene slice data system, employing standardized collection and classification techniques to establish positive and negative scene slice sets. It deeply mines easily overlooked identification error scenarios in real-world road tests, achieving refined data management and improving data resource utilization efficiency. Based on this, a working condition consistency algorithm comparison mechanism is established. Relying on the standardized scene slice sets, a unified algorithm testing benchmark environment is constructed to eliminate external variable interference, ensuring accurate comparison between new and old algorithms under completely equivalent working conditions, providing technical support for probing algorithm performance boundaries. Furthermore, a bidirectional cross-validation evaluation system is designed, building a positive and negative bidirectional data validation model based on the two types of slice sets to accurately define the application boundaries of road test data and the applicable scenarios of the algorithm. This system significantly improves the scientific rigor and accuracy of algorithm iteration evaluation, greatly shortens the R&D cycle, reduces the number of invalid road tests, lowers R&D costs, and forms a high-efficiency and economical algorithm continuous optimization technical solution, providing strong support for the development of intelligent driving algorithms.

[0107] In this embodiment, the raw CAN bus data collected during real-vehicle road tests is first parsed, and scene slices are generated based on the input signals output by the perception fusion algorithm. These scene slices are then compared with video data at corresponding times, and categorized into positive and negative scene slice sets according to whether the corresponding scene actually exists in the video. The original data and output information of any identification errors are recorded and fed back to the R&D personnel for algorithm modification. For the updated algorithm under test, data back-injection technology is used: first, the positive scene slice set is imported to verify the algorithm's basic functions, and then the negative scene slice set is back-injected to verify the effectiveness of the algorithm improvement, significantly shortening the R&D cycle. Then, specific raw data is selected for data back-injection based on testing requirements and integrated into the intelligent driving vehicle-side algorithm for real-vehicle road testing. The above processing procedure is repeated for newly collected data, continuously expanding and improving the scene slice library, providing a solid data guarantee for the iterative optimization of intelligent driving algorithms.

[0108] Example 2

[0109] Figure 3 This is a schematic diagram of a testing device provided in an embodiment of the present invention. This embodiment is applicable to the testing of algorithms. The device can be implemented using software and / or hardware, and can be integrated into any device that provides testing functionality, such as… Figure 3 As shown, the testing device specifically includes: an acquisition module 310, a filtering module 320, a sending module 330, a verification module 340, and a replacement module 350.

[0110] The acquisition module is used to acquire raw CAN bus data collected from the roadside of the actual vehicle, wherein the algorithm to be tested is deployed on the actual vehicle.

[0111] The filtering module is used to filter the original CAN bus data based on the algorithm under test to obtain a positive scene slice set and a negative scene slice set for at least one driving scenario corresponding to the algorithm under test. The scene slices in the positive scene slice set are scene slices correctly identified based on the algorithm under test, and the scene slices in the negative scene slice set are scene slices incorrectly identified based on the algorithm under test.

[0112] The sending module is used to send the input data and output results of the algorithm under test corresponding to the negative scene slice set to the target terminal, and to receive the updated algorithm under test fed back by the target terminal.

[0113] The verification module is used to verify the updated algorithm under test based on the positive scene slice set and the negative scene slice set of the at least one driving scenario;

[0114] The replacement module is used to replace the algorithm under test deployed on the actual vehicle with the updated algorithm under test if the verification passes.

[0115] The above-described products can perform the methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects for performing the methods.

[0116] Example 3

[0117] Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0118] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0119] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as test methods.

[0121] In some embodiments, the test method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the test method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the test method by any other suitable means (e.g., by means of firmware).

[0122] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0124] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0126] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0127] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0128] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0129] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the testing method according to any embodiment of the invention.

[0130] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A testing method, characterized in that, include: Acquire raw CAN bus data collected from the roadside of a real vehicle, wherein the algorithm to be tested is deployed on the real vehicle; The original CAN bus data is filtered based on the algorithm under test to obtain a positive scene slice set and a negative scene slice set for at least one driving scenario corresponding to the algorithm under test. The scene slices in the positive scene slice set are the scene slices correctly identified based on the algorithm under test, and the scene slices in the negative scene slice set are the scene slices incorrectly identified based on the algorithm under test. The input data and output results of the algorithm to be tested corresponding to the negative scene slice set are sent to the target terminal, and the updated algorithm to be tested is received from the target terminal. The updated algorithm to be tested is verified based on the positive scene slice set and the negative scene slice set of the at least one driving scenario; If the verification passes, the algorithm under test deployed on the actual vehicle will be replaced with the updated algorithm under test.

2. The method according to claim 1, characterized in that, Based on the algorithm under test, the original CAN bus data is filtered to obtain at least one positive scene slice set and one negative scene slice set corresponding to the algorithm under test, including: Obtain the target output results corresponding to each driving scenario of the algorithm under test; Based on the target output results corresponding to each driving scenario, the original CAN bus data is filtered to obtain scenario slices for each driving scenario; Obtain the video data corresponding to each scene slice; The video data corresponding to the scene slices of each driving scenario are identified to obtain the identification results of the scene slices of each driving scenario; Based on the recognition results of scene slices for each driving scenario and the target output result, the scene slices for each driving scenario are divided into a positive scene slice set and a negative scene slice set. The recognition results of scene slices in the positive scene slice set are the same as the target output result, while the recognition results of scene slices in the negative scene slice set are different from the target output result.

3. The method according to claim 2, characterized in that, The updated algorithm to be tested is validated based on the positive scene slice set and the negative scene slice set of the at least one driving scenario, including: The scene slices in the positive scene slice set of the at least one driving scenario are sequentially input into the updated test algorithm to obtain the output results of the scene slices in the positive scene slice set of each driving scenario. If the output results of the scene slices in the positive scene slice set of each driving scenario match the target output results corresponding to each driving scenario, then the scene slices in the negative scene slice set of at least one driving scenario are sequentially input into the updated test algorithm to obtain the output results of the scene slices in the negative scene slice set of each driving scenario. The updated algorithm to be tested is verified based on the output results of the scene slices in the negative scene slice set of each driving scenario and the target output results corresponding to each driving scenario.

4. The method according to claim 3, characterized in that, If the verification passes, the algorithm under test deployed on the actual vehicle will be replaced with the updated algorithm under test, including: If the output results of the scene slices in the negative scene slice set of each driving scenario match the target output results corresponding to each driving scenario, then the verification is successful. Replace the algorithm under test deployed on the actual vehicle with the updated algorithm under test.

5. The method according to claim 2, characterized in that, Based on the target output results corresponding to each driving scenario, the original CAN bus data is filtered to obtain scene slices for each driving scenario and the corresponding video data for each scene slice, including: The original CAN bus data is filtered according to the input data corresponding to the algorithm under test to obtain the first CAN bus data; Obtain the target data format corresponding to the algorithm under test; The first CAN bus data is converted into the target data format to obtain the second CAN bus data; Based on the target output results corresponding to each driving scenario, the second CAN bus data is filtered to obtain scenario slices for each driving scenario; Obtain the video data corresponding to each scene slice of each driving scenario.

6. The method according to claim 2, characterized in that, The at least one driving scenario includes: a scenario where a vehicle cuts in from ahead, and the target output result corresponding to the scenario where a vehicle cuts in from ahead is: the status signal of the vehicle cutting in from ahead is a preset value; Based on the target output results corresponding to each driving scenario, the original CAN bus data is filtered to obtain scenario slices for each driving scenario, including: Extract the times when the forward vehicle cut-in status signal in the original CAN bus data is a preset value to obtain a time list; Based on each time in the time list, the original CAN bus data is filtered to obtain the chassis data, radar data, and camera data corresponding to each time in the time list; Based on the chassis data, radar data, and camera data corresponding to each moment, a scene slice of the scene where the vehicle ahead cuts in is generated.

7. A testing apparatus, characterized in that, include: The acquisition module is used to acquire raw CAN bus data collected from the roadside of the actual vehicle, wherein the algorithm to be tested is deployed on the actual vehicle; The filtering module is used to filter the original CAN bus data based on the algorithm under test to obtain a positive scene slice set and a negative scene slice set for at least one driving scenario corresponding to the algorithm under test. The scene slices in the positive scene slice set are scene slices correctly identified based on the algorithm under test, and the scene slices in the negative scene slice set are scene slices incorrectly identified based on the algorithm under test. The sending module is used to send the input data and output results of the algorithm under test corresponding to the negative scene slice set to the target terminal, and to receive the updated algorithm under test fed back by the target terminal. The verification module is used to verify the updated algorithm under test based on the positive scene slice set and the negative scene slice set of the at least one driving scenario; The replacement module is used to replace the algorithm under test deployed on the actual vehicle with the updated algorithm under test if the verification passes.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the test method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the test method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the test method according to any one of claims 1-6.

Images