Method for evaluating automatic driving perception model, computer device and storage medium
By acquiring and matching the evaluation data of the autonomous driving perception model with the target environment information, the problem of high development cost in traditional evaluation methods is solved, and efficient evaluation applicable to multiple projects is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 安徽蔚来智驾科技有限公司
- Filing Date
- 2023-02-23
- Publication Date
- 2026-06-12
AI Technical Summary
Traditional methods for evaluating autonomous driving perception models require the development of multiple scripts or programs, resulting in high evaluation development costs and a lack of reusability.
By acquiring the data to be evaluated, determining the target environment information, and using a perception model to process the perception and predict the environment information, the evaluation results are obtained through matching. This method is applicable to a variety of evaluation projects.
It enables cost savings in development across different evaluation projects, while improving evaluation efficiency and applicability.
Smart Images
Figure CN116152637B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous driving technology, and in particular to an evaluation method, apparatus, computer device, storage medium, and computer program product for an autonomous driving perception model. Background Technology
[0002] With the development of autonomous driving technology, perception models have become one of the core technologies of autonomous vehicles. Therefore, the evaluation of perception models is of paramount importance.
[0003] In traditional technologies, when evaluating perception models, different perception modules require the development of multiple scripts or evaluation programs for different evaluation items, such as algorithm evaluation, regression evaluation, version comparison evaluation, edge-cloud consistency evaluation, and data self-consistency evaluation. Furthermore, these scripts or programs lack reusability, resulting in high development costs for evaluation. Summary of the Invention
[0004] Therefore, it is necessary to address the technical problem of high development costs in the aforementioned evaluation process by providing an evaluation method, apparatus, computer equipment, computer-readable storage medium, and computer program product for autonomous driving perception models that can save on evaluation development costs.
[0005] Firstly, this application provides a method for evaluating an autonomous driving perception model. The method includes:
[0006] Acquire the data to be evaluated, which includes the perception data collected during vehicle operation;
[0007] For the target evaluation item, determine the target environmental information of the vehicle corresponding to the data to be evaluated;
[0008] Obtain the predicted environment information of the vehicle after the perception model to be evaluated has processed the data to be evaluated.
[0009] The predicted environmental information and the target environmental information of the vehicle are matched to obtain a matching result;
[0010] The evaluation result of the perception model to be evaluated is determined based on the matching result.
[0011] In one embodiment, acquiring the data to be evaluated includes: acquiring continuously collected perception data during vehicle operation; segmenting the perception data according to a set time interval to obtain multiple segmented perception data segments; and using the multiple perception data segments as the data to be evaluated.
[0012] In one embodiment, the sensing data segment includes multiple frames; after obtaining the segmented multiple sensing data segments, the method further includes: identifying the sensing object of each frame in the sensing data segment; and determining the temporal trajectory of the same sensing object in different frames based on the sensing object of each frame.
[0013] In one embodiment, using the plurality of sensing data segments as the evaluation data includes: establishing a tree structure of the sensing data based on the plurality of sensing data segments, the plurality of frames corresponding to each sensing data segment, the sensing objects of each frame, and the temporal trajectories of the same sensing objects in different frames; and using the sensing data of the tree structure as the evaluation data.
[0014] In one embodiment, after establishing the tree structure of the sensing data, the method further includes: storing the sensing data of the tree structure.
[0015] In one embodiment, obtaining the predicted environment information of the vehicle predicted by the perception model to be evaluated after processing the data to be evaluated includes: for each perception data segment in the data to be evaluated, inputting the perception data segment into the perception model to be evaluated, and obtaining the predicted environment information of the vehicle predicted by the perception model to be evaluated for each perception data segment.
[0016] In one embodiment, the matching process of the predicted environment information and target environment information of the vehicle to obtain a matching result includes: matching the predicted environment information of the vehicle predicted for each of the perception data segments with the corresponding target environment information to obtain a matching degree for each of the perception data segments; and calculating the matching degree of each of the perception data segments in the data to be evaluated to obtain a matching result for the data to be evaluated.
[0017] Secondly, this application also provides an evaluation device for an autonomous driving perception model. The device includes:
[0018] The data acquisition module is used to acquire the data to be evaluated, which includes the perception data collected during the vehicle's operation.
[0019] The target information determination module is used to determine the target environment information of the vehicle corresponding to the data to be evaluated for the target evaluation item.
[0020] The prediction information acquisition module is used to acquire the predicted environment information of the vehicle predicted by the perception model to be evaluated after processing the data to be evaluated.
[0021] The matching module is used to perform matching processing on the predicted environmental information and the target environmental information of the vehicle to obtain the matching result;
[0022] The evaluation result determination module is used to determine the evaluation result of the perception model to be evaluated based on the matching result.
[0023] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in the first aspect above.
[0024] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.
[0025] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.
[0026] The aforementioned evaluation method, apparatus, computer equipment, storage medium, and computer program product for autonomous driving perception models acquire the data to be evaluated, determine the target environment information of the vehicle corresponding to the data for the target evaluation item, acquire the predicted environment information of the vehicle predicted by the perception model after processing the data, then match the predicted environment information of the vehicle with the target environment information to obtain the matching result, and finally determine the evaluation result of the perception model based on the matching result. Because it can determine and match the target environment information of the vehicle corresponding to the data to be evaluated for different target evaluation items during evaluation, it is applicable to the evaluation of various evaluation items and can save on evaluation development costs compared to traditional technologies. Attached Figure Description
[0027] Figure 1 This is a flowchart illustrating an evaluation method for an autonomous driving perception model in one embodiment.
[0028] Figure 2 This is a flowchart illustrating the steps for acquiring perceived data in one embodiment;
[0029] Figure 3 This is a flowchart illustrating the sensing data preprocessing steps in one embodiment;
[0030] Figure 4 This is a flowchart illustrating the sensing data preprocessing step in another embodiment;
[0031] Figure 5 This is a schematic diagram of the tree structure of the sensed data in one embodiment;
[0032] Figure 6 This is a flowchart illustrating the matching process steps in one embodiment;
[0033] Figure 7 This is a structural block diagram of an evaluation device for an autonomous driving perception model in one embodiment;
[0034] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0036] In one embodiment, such as Figure 1 As shown, an evaluation method for an autonomous driving perception model is provided. This embodiment uses the application of this method to a computer device as an example for illustration, and it may specifically include the following steps:
[0037] Step 102: Obtain the data to be evaluated.
[0038] The data to be evaluated includes perception data collected during vehicle operation. Specifically, perception data includes, but is not limited to, sensor data and image data recorded during the vehicle's historical driving process.
[0039] A perception model is a network learning model used in the process of autonomous driving to identify information such as the position, size, category, and speed of perceived objects. In this embodiment, when it is necessary to evaluate the perception model, the computer device first needs to acquire the data to be evaluated.
[0040] Step 104: For the target evaluation item, determine the target environmental information of the vehicle corresponding to the data to be evaluated.
[0041] The target evaluation item can be any test item that needs to be evaluated on the perception model, including but not limited to algorithm evaluation, regression evaluation, version comparison evaluation, edge-cloud consistency evaluation, and data self-consistency evaluation. The target environment information is the ground truth data used to evaluate the perception model, corresponding to the target evaluation item and the data to be evaluated. For example, if the target evaluation item is an algorithm evaluation of the perception model, the target environment information can be the label information of the perceived object's location, size, category, and speed, based on the data to be evaluated. If the target evaluation item is a version comparison evaluation of the perception model, the target environment information can be the location, size, category, and speed of the perceived object in the vehicle's environment, output after processing the data to be evaluated, based on the version of the perception model to be compared. Therefore, the target environment information of the vehicle corresponding to the data to be evaluated will differ for different target evaluation items.
[0042] In this embodiment, the computer device can determine the target environmental information of the vehicle corresponding to the data to be evaluated for the target evaluation item.
[0043] Step 106: Obtain the predicted environment information of the vehicle predicted by the perception model to be evaluated after processing the data to be evaluated.
[0044] The perception model to be evaluated can be the perception model that needs to be evaluated. The predicted environment information can be information about the vehicle's environment output by the perception model after processing the evaluation data, including but not limited to predicted information such as the location, size, category, and speed of perceived objects in the vehicle's environment. Specifically, the computer device can obtain the predicted environment information of the vehicle predicted by the perception model after processing the evaluation data.
[0045] Step 108: Match the predicted environmental information of the vehicle with the target environmental information to obtain the matching result.
[0046] The matching process involves comparing various indicators in the predicted and target environmental information of the vehicle. The matching result is the determination of whether a match has been made after the comparison process. Specifically, each indicator can be the corresponding perceived object and its position, size, category, speed, and other indicators.
[0047] In this embodiment, the computer device obtains the corresponding matching result by matching the predicted environmental information of the vehicle with the target environmental information.
[0048] Step 110: Determine the evaluation result of the perception model to be evaluated based on the matching result.
[0049] The evaluation result refers to the output used to characterize the performance of the perception model after evaluation, such as, but not limited to, accuracy, recall, and error distribution. Specifically, the computer device can determine the evaluation result of the perception model based on the matching results obtained above.
[0050] In the aforementioned evaluation method for autonomous driving perception models, the method acquires the data to be evaluated, determines the target environment information of the vehicle corresponding to the data for the target evaluation item, and acquires the predicted environment information of the vehicle predicted by the perception model after processing the data. Then, the predicted environment information of the vehicle and the target environment information are matched to obtain the matching result. Finally, the evaluation result of the perception model is determined based on the matching result. Because this method can determine the target environment information of the vehicle corresponding to the data to be evaluated for different target evaluation items, it is applicable to the evaluation of various evaluation items and can save development costs compared to traditional technologies.
[0051] In one embodiment, such as Figure 2 As shown, in step 102, the data to be evaluated is obtained, which may specifically include:
[0052] Step 202: Acquire the perception data continuously collected during vehicle operation.
[0053] Since perception data consists of sensor data, image data, etc., recorded during the vehicle's historical driving process, it is usually continuously collected data and is characterized by a large amount of data.
[0054] Step 204: Divide the sensing data into segments according to a set time interval to obtain multiple segments of sensing data.
[0055] The set time interval can be a pre-defined time interval for segmenting the sensed data, such as 5 minutes, 3 minutes, etc., meaning the sensed data is segmented every 5 or 3 minutes to obtain multiple segmented sensed data segments. Since the sensed data is continuously collected, its volume is large, and processing it requires a considerable amount of time. In this embodiment, the computer device can segment the sensed data according to the set time interval to obtain multiple segmented sensed data segments, and then perform parallel evaluation based on these multiple segments to improve evaluation efficiency.
[0056] Step 206: Use multiple sensing data segments as the data to be evaluated.
[0057] Specifically, the computer device can use the multiple segments of the above-mentioned segmented sensing data as the data to be evaluated, that is, the data to be evaluated includes multiple segments of sensing data.
[0058] In the above embodiments, the computer device acquires continuously collected perception data during vehicle operation and segments the perception data according to a set time interval to obtain multiple segments of perception data. These multiple segments of perception data are used as data to be evaluated, thereby enabling parallel evaluation and improving evaluation efficiency.
[0059] In one embodiment, each sensing data segment may include multiple frames. For example... Figure 3 As shown, after obtaining the segmented multiple sensing data segments in step 204, the above method may further include:
[0060] Step 302: Identify the sensing objects in each frame of the sensing data segment.
[0061] The perceived object can be any object in the frame, such as pedestrians, vehicles, obstacles, traffic lights, etc. In this embodiment, the computer device can identify the perceived object in each frame of the perceived data segment.
[0062] Step 304: Determine the temporal trajectory of the same perceived object in different frames based on the perceived object in each frame.
[0063] Since the sensing data segment is obtained by dividing continuously acquired sensing data into segments based on certain time intervals, the multiple frames in each sensing data segment have a certain temporal continuity. Furthermore, since the sensing object in each frame of the sensing data segment has been identified in the above steps, based on the temporal continuity of multiple frames in the sensing data segment and the sensing object in each frame, the temporal trajectory of the same sensing object in different frames can be determined.
[0064] In this embodiment, the computer device identifies the sensing object in each frame of the sensing data segment, and determines the temporal trajectory of the same sensing object in different frames based on the sensing object in each frame, thereby performing evaluation, which can improve evaluation efficiency.
[0065] In one embodiment, such as Figure 4 As shown, in step 206, multiple sensing data segments are used as the data to be evaluated, which may specifically include:
[0066] Step 402: Establish a tree structure of the sensing data based on multiple sensing data segments, multiple frames corresponding to each sensing data segment, sensing objects in each frame, and the temporal trajectories of the same sensing objects in different frames.
[0067] Tree structures, which represent hierarchical relationships, exhibit a one-to-many tree-like relationship between data elements, making them an important type of non-linear data structure. In a tree structure, the root node has no predecessor node, and each of the remaining nodes has exactly one predecessor node. Leaf nodes have no successors, while each of the remaining nodes can have one or more successors. In this embodiment, the tree structure built with the perceived data as the root node is as follows: Figure 5 As shown, in Figure 5 In the tree structure shown, the perceptual data `Dataset` is the root node, the perceptual data segments `Clip` are child nodes of the perceptual data `Dataset`, the frames are child nodes of the perceptual data segments `Clip`, and the perceptual objects `Object` are child nodes of the frames and also leaf nodes of the tree. Specifically, the perceptual data `Dataset` can include multiple perceptual data segments `Clip`, each perceptual data segment `Clip` corresponds to multiple frames, and each frame can include multiple perceptual objects `Object`. Therefore, the temporal trajectory `Trajectory` of the same perceptual object `Object` in different frames can also be determined. This forms a general data structure, namely the tree structure of perceptual data, which can be used for various evaluation projects to evaluate perceptual models.
[0068] Step 404: Use the perceptual data of the tree structure as the evaluation data.
[0069] Specifically, the computer device uses the perceptual data of the aforementioned tree structure as the data to be evaluated, thus making it applicable to various evaluation items and evaluating the perceptual model for various evaluation items.
[0070] In one embodiment, after establishing the tree structure of the perception data in step 402, the method may further include storing the tree-structured perception data. In this embodiment, by storing the tree-structured perception data, it can be reused in other evaluation projects to evaluate the perception model, which not only achieves data universality but also improves the evaluation efficiency of the perception model in other evaluation projects.
[0071] In one embodiment, step 106, obtaining the predicted environment information of the vehicle predicted by the perception model to be evaluated after processing the evaluation data, may further include: for each perception data segment in the evaluation data, inputting the perception data segment into the perception model to be evaluated, and obtaining the predicted environment information of the vehicle predicted by the perception model for each perception data segment. That is, the perception model to be evaluated processes each perception data segment in the evaluation data to obtain the predicted environment information of the vehicle based on the perception data segment.
[0072] In one scenario, because the data to be evaluated also has corresponding target environment information for different evaluation items, and because the data to be evaluated includes multiple sensing data segments, there is also target environment information corresponding to each sensing data segment in the data to be evaluated for different evaluation items, which facilitates subsequent matching processing.
[0073] In one embodiment, such as Figure 6 As shown, in step 108, the predicted environmental information and the target environmental information of the vehicle are matched to obtain the matching result. Specifically, this may also include the following steps:
[0074] Step 602: Match the predicted environment information of the vehicle for each sensing data segment with the corresponding target environment information to obtain the matching degree for each sensing data segment.
[0075] The matching degree can be defined as the degree of fit between the predicted environmental information of the vehicle and the corresponding target environmental information for each perceived data segment.
[0076] Specifically, if the predicted environment information of the vehicle predicted by the model to be evaluated for a certain perception data segment includes the predicted perception object A, and the position AP, size AS, category AK, and speed AV of the perception object A. If the target evaluation item is the algorithm evaluation of the perception model to be evaluated, then the target environment information corresponding to the perception data segment is the label information such as the position, size, category, and speed of the labeled perception object. In this embodiment, the labeled perception object B has a position of BP, a size of BS, a category of BK, and a speed of BV. Then, the predicted environment information of the vehicle predicted for each perception data segment is matched with the corresponding target environment information. Specifically, this can be done by matching the predicted perception object A and the labeled perception object B. For example, the object similarity AB between perception object A and perception object B can be calculated, the category similarity ABK between category AK and BK can be determined, the position fit ABP between position AP and BP can be determined, the size fit ABS between size AS and BS can be determined, and the speed fit ABV between speed AV and BV can be determined. The results are then weighted according to the weights of each indicator to obtain the matching degree for the perception data segment. For example, if the weight of object similarity AB is l, the weight of category similarity ABK is m, the weight of positional fit ABP is n, the weight of size fit ABS is r, and the weight of velocity fit ABV is o, then the matching degree F of the perceived data segment is F = AB × l + ABK × m + ABP × n + ABS × r + ABV × o. Where the sum of l, m, n, r, and o is 1.
[0077] It is understood that the matching method may differ for different evaluation items, and this embodiment does not limit this.
[0078] Step 604: Calculate the matching degree of each sensing data segment in the data to be evaluated to obtain the matching result for the data to be evaluated.
[0079] Specifically, by performing distributed statistics on the matching degree of each sensing data segment in the data to be evaluated, the matching result of the data to be evaluated is obtained. Then, the evaluation result of the sensing model to be evaluated can be determined based on the matching result, so as to achieve a general evaluation of the sensing model.
[0080] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0081] Based on the same inventive concept, this application also provides an evaluation apparatus for implementing the evaluation method of the autonomous driving perception model described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more embodiments of the autonomous driving perception model evaluation apparatus provided below can be found in the limitations of the autonomous driving perception model evaluation method described above, and will not be repeated here.
[0082] In one embodiment, such as Figure 7 As shown, an evaluation device for an autonomous driving perception model is provided, comprising: a data acquisition module 702, a target information determination module 704, a prediction information acquisition module 706, a matching module 708, and an evaluation result determination module 710, wherein:
[0083] The data acquisition module 702 is used to acquire the data to be evaluated, which includes the perception data collected during the vehicle's operation.
[0084] The target information determination module 704 is used to determine the target environment information of the vehicle corresponding to the data to be evaluated for the target evaluation item;
[0085] The prediction information acquisition module 706 is used to acquire the predicted environment information of the vehicle predicted by the perception model to be evaluated after processing the data to be evaluated.
[0086] The matching module 708 is used to perform matching processing on the predicted environmental information and the target environmental information of the vehicle to obtain a matching result;
[0087] The evaluation result determination module 710 is used to determine the evaluation result of the perception model to be evaluated based on the matching result.
[0088] In one embodiment, the data acquisition module is further configured to: acquire continuously collected sensing data during vehicle operation; segment the sensing data according to a set time interval to obtain multiple segmented sensing data segments; and use the multiple sensing data segments as the data to be evaluated.
[0089] In one embodiment, the sensing data segment includes multiple frames; the data acquisition module is further configured to: identify the sensing object in each frame of the sensing data segment; and determine the temporal trajectory of the same sensing object in different frames based on the sensing object in each frame.
[0090] In one embodiment, the data acquisition module is further configured to: establish a tree structure of the sensing data based on multiple sensing data segments of the sensing data, multiple frames corresponding to each sensing data segment, sensing objects in each frame, and the temporal trajectories of the same sensing objects in different frames; and use the sensing data of the tree structure as the data to be evaluated.
[0091] In one embodiment, the device further includes a storage module for storing the perceived data of the tree structure.
[0092] In one embodiment, the prediction information acquisition module is further configured to: input the perception data segment into the perception model to be evaluated for each perception data segment in the data to be evaluated, and obtain the predicted environment information of the vehicle for each perception data segment output by the perception model to be evaluated.
[0093] In one embodiment, the matching module is further configured to: match the predicted environment information of the vehicle predicted for each of the perception data segments with the corresponding target environment information to obtain the matching degree for each of the perception data segments; and calculate the matching degree of each of the perception data segments in the data to be evaluated to obtain the matching result for the data to be evaluated.
[0094] Each module in the aforementioned evaluation device for the autonomous driving perception model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0095] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an evaluation method for an autonomous driving perception model. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0096] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0097] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0098] Acquire the data to be evaluated, which includes the perception data collected during vehicle operation;
[0099] For the target evaluation item, determine the target environmental information of the vehicle corresponding to the data to be evaluated;
[0100] Obtain the predicted environment information of the vehicle after the perception model to be evaluated has processed the data to be evaluated.
[0101] The predicted environmental information and the target environmental information of the vehicle are matched to obtain a matching result;
[0102] The evaluation result of the perception model to be evaluated is determined based on the matching result.
[0103] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring continuously collected perception data during vehicle operation; segmenting the perception data according to a set time interval to obtain multiple segmented perception data segments; and using the multiple perception data segments as the data to be evaluated.
[0104] In one embodiment, the sensing data segment includes multiple frames; when the processor executes the computer program, it further implements the following steps: identifying the sensing object in each frame of the sensing data segment; and determining the temporal trajectory of the same sensing object in different frames based on the sensing object in each frame.
[0105] In one embodiment, when the processor executes the computer program, it further implements the following steps: establishing a tree structure of the sensing data based on multiple sensing data segments of the sensing data, multiple frames corresponding to each of the sensing data segments, sensing objects in each frame, and the temporal trajectories of the same sensing objects in different frames; and using the sensing data of the tree structure as the data to be evaluated.
[0106] In one embodiment, when the processor executes a computer program, it further performs the following step: storing the perception data of the tree structure.
[0107] In one embodiment, when the processor executes the computer program, it further performs the following steps: for each sensing data segment in the data to be evaluated, inputting the sensing data segment into the sensing model to be evaluated, and obtaining the predicted environment information of the vehicle for each sensing data segment output by the sensing model to be evaluated.
[0108] In one embodiment, when the processor executes the computer program, it further performs the following steps: matching the predicted environment information of the vehicle predicted for each of the perception data segments with the corresponding target environment information to obtain the matching degree for each of the perception data segments; and calculating the matching degree of each of the perception data segments in the data to be evaluated to obtain the matching result for the data to be evaluated.
[0109] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0110] Acquire the data to be evaluated, which includes the perception data collected during vehicle operation;
[0111] For the target evaluation item, determine the target environmental information of the vehicle corresponding to the data to be evaluated;
[0112] Obtain the predicted environment information of the vehicle after the perception model to be evaluated has processed the data to be evaluated.
[0113] The predicted environmental information and the target environmental information of the vehicle are matched to obtain a matching result;
[0114] The evaluation result of the perception model to be evaluated is determined based on the matching result.
[0115] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: acquiring continuously collected perception data during vehicle operation; segmenting the perception data according to a set time interval to obtain multiple segmented perception data segments; and using the multiple perception data segments as the data to be evaluated.
[0116] In one embodiment, the sensing data segment includes multiple frames; when the computer program is executed by the processor, it further performs the following steps: identifying the sensing object in each frame of the sensing data segment; and determining the temporal trajectory of the same sensing object in different frames based on the sensing object in each frame.
[0117] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: establishing a tree structure of the sensing data based on multiple sensing data segments of the sensing data, multiple frames corresponding to each of the sensing data segments, the sensing objects of each frame, and the temporal trajectories of the same sensing objects in different frames; and using the sensing data of the tree structure as the data to be evaluated.
[0118] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: storing the perception data of the tree structure.
[0119] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each sensing data segment in the data to be evaluated, inputting the sensing data segment into the sensing model to be evaluated, and obtaining the predicted environment information of the vehicle for each sensing data segment output by the sensing model to be evaluated.
[0120] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: matching the predicted environment information of the vehicle predicted for each of the perception data segments with the corresponding target environment information to obtain the matching degree for each of the perception data segments; and calculating the matching degree of each of the perception data segments in the data to be evaluated to obtain the matching result for the data to be evaluated.
[0121] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:
[0122] Acquire the data to be evaluated, which includes the perception data collected during vehicle operation;
[0123] For the target evaluation item, determine the target environmental information of the vehicle corresponding to the data to be evaluated;
[0124] Obtain the predicted environment information of the vehicle after the perception model to be evaluated has processed the data to be evaluated.
[0125] The predicted environmental information and the target environmental information of the vehicle are matched to obtain a matching result;
[0126] The evaluation result of the perception model to be evaluated is determined based on the matching result.
[0127] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: acquiring continuously collected perception data during vehicle operation; segmenting the perception data according to a set time interval to obtain multiple segmented perception data segments; and using the multiple perception data segments as the data to be evaluated.
[0128] In one embodiment, the sensing data segment includes multiple frames; when the computer program is executed by the processor, it further performs the following steps: identifying the sensing object in each frame of the sensing data segment; and determining the temporal trajectory of the same sensing object in different frames based on the sensing object in each frame.
[0129] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: establishing a tree structure of the sensing data based on multiple sensing data segments of the sensing data, multiple frames corresponding to each of the sensing data segments, the sensing objects of each frame, and the temporal trajectories of the same sensing objects in different frames; and using the sensing data of the tree structure as the data to be evaluated.
[0130] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: storing the perception data of the tree structure.
[0131] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for each sensing data segment in the data to be evaluated, inputting the sensing data segment into the sensing model to be evaluated, and obtaining the predicted environment information of the vehicle for each sensing data segment output by the sensing model to be evaluated.
[0132] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: matching the predicted environment information of the vehicle predicted for each of the perception data segments with the corresponding target environment information to obtain the matching degree for each of the perception data segments; and calculating the matching degree of each of the perception data segments in the data to be evaluated to obtain the matching result for the data to be evaluated.
[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0134] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0135] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0136] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for evaluating an autonomous driving perception model, characterized in that, The method includes: Acquire the data to be evaluated, wherein the data to be evaluated includes perception data continuously collected by the vehicle during driving; For each target evaluation item, target environment information of the vehicle corresponding to the data to be evaluated is determined; wherein, the target environment information is dynamically determined according to the type of the target evaluation item; the target evaluation item includes at least one of algorithm evaluation, regression evaluation, version comparison evaluation, edge-cloud consistency evaluation, and data self-consistency evaluation; The predicted environment information of the vehicle is obtained after the perception model to be evaluated performs perception processing on the data to be evaluated; wherein, the predicted environment information includes at least one of the predicted information of the position, size, category and speed of the perceived objects in the environment where the vehicle is located. The predicted environment information of the vehicle for each sensing data segment is matched with the corresponding target environment information to obtain the matching degree for each sensing data segment. The matching degree of each of the perceived data segments in the data to be evaluated is calculated to obtain the matching result for the data to be evaluated; The evaluation result of the perception model to be evaluated is determined based on the matching result; the evaluation result includes at least one of accuracy, recall, and error distribution.
2. The method according to claim 1, characterized in that, The acquisition of the data to be evaluated includes: The sensed data is segmented according to a set time interval to obtain multiple segmented sensed data segments. The multiple sensing data segments are used as the data to be evaluated.
3. The method according to claim 2, characterized in that, The sensing data segment includes multiple frames; after obtaining the segmented sensing data segments, the method further includes: Identify the perceived object in each frame of the perceived data segment; Based on the perceived object in each frame, determine the temporal trajectory of the same perceived object in different frames.
4. The method according to claim 3, characterized in that, The step of using the plurality of perceived data segments as the data to be evaluated includes: A tree structure of the sensing data is established based on multiple sensing data segments, multiple frames corresponding to each sensing data segment, the sensing objects in each frame, and the temporal trajectories of the same sensing objects in different frames. The perceived data of the tree structure is used as the evaluation data.
5. The method according to claim 4, characterized in that, After establishing the tree structure of the perceived data, the method further includes: Store the perception data of the tree structure.
6. The method according to any one of claims 2 to 5, characterized in that, The step of obtaining the predicted environment information of the vehicle after the perception model to be evaluated has processed the data to be evaluated includes: For each perception data segment in the data to be evaluated, the perception data segment is input into the perception model to be evaluated to obtain the predicted environment information of the vehicle for each perception data segment output by the perception model to be evaluated.
7. An evaluation device for an autonomous driving perception model, characterized in that, The device includes: The data acquisition module is used to acquire the data to be evaluated, wherein the data to be evaluated includes perception data continuously collected by the vehicle during driving. The target information determination module is used to determine the target environment information of the vehicle corresponding to the data to be evaluated for the target evaluation item; wherein, the target environment information is dynamically determined according to the type of the target evaluation item; the target evaluation item includes at least one of algorithm evaluation, regression evaluation, version comparison evaluation, edge-cloud consistency evaluation and data self-consistency evaluation; The prediction information acquisition module is used to acquire the predicted environment information of the vehicle predicted by the perception model to be evaluated after processing the data to be evaluated; wherein, the predicted environment information includes at least one of the predicted information of the position, size, category and speed of the perceived objects in the environment where the vehicle is located. The matching module is used to match the predicted environment information of the vehicle for each perception data segment with the corresponding target environment information to obtain the matching degree for each perception data segment; and to calculate the matching degree of each perception data segment in the data to be evaluated to obtain the matching result for the data to be evaluated. The evaluation result determination module is used to determine the evaluation result of the perception model to be evaluated based on the matching result; the evaluation result includes at least one of accuracy, recall, and error distribution.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.