Evaluation methods, apparatus, equipment, storage media and program products

Abstract

This disclosure provides an evaluation method, apparatus, device, storage medium, and program product, relating to the application of artificial intelligence technology in the vehicle field. The method includes: acquiring the inference result of a functional entity under test based on environmental data, and the corresponding ground truth data; pairing the inference result and the ground truth data to obtain paired data; evaluating the paired data based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results; and displaying a functional entity evaluation report based on the multi-dimensional evaluation results. This method can pair the acquired inference result and ground truth data of the functional entity under test, and perform systematic evaluation based on multiple preset evaluation dimensions, generating multi-dimensional evaluation results and displaying an evaluation report. This achieves comprehensive, objective, and quantitative automated evaluation of the functional entity under test, improving the completeness and reliability of the evaluation results.

Description

Technical Field

[0001] This disclosure relates to the application of artificial intelligence technology in the vehicle field, and in particular to an evaluation method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of computer technology and vehicle intelligence technology, the performance evaluation of driver assistance function entities has become the key to ensuring driving safety and functional iteration. It is very important to conduct accurate and efficient evaluation of driver assistance function entities.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide an evaluation method, apparatus, device, storage medium, and program product.

[0005] According to a first aspect of the present disclosure, an evaluation method is provided, comprising: acquiring a reasoning result of a functional entity under test based on environmental data, and ground truth data corresponding to the environmental data; performing pairing processing on the reasoning result and the ground truth data to obtain paired data; evaluating the paired data based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results; and displaying a functional entity evaluation report based on the multi-dimensional evaluation results.

[0006] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: This disclosure can pair the obtained inference results of the functional entity under test with the true data, and conduct systematic evaluation based on multiple preset evaluation dimensions, generate multi-dimensional evaluation results and display evaluation reports, realize comprehensive, objective and quantitative automated evaluation of the functional entity under test, improve the completeness and credibility of the evaluation results, provide a standardized evaluation framework for the performance evaluation of functional entities, and provide quantitative basis for the optimization of functional entities.

[0007] In some implementations, the step of pairing the inference result and the truth data to obtain paired data includes: aligning the inference result with the truth data based on the timestamp of the environmental data to obtain the inference result and the truth data with matching timestamps, which are then used as the paired data.

[0008] In the above implementation, timestamps can be used to align the inference results with the truth data, ensuring that the two are matched in time sequence, avoiding evaluation errors caused by time mismatch, providing accurate matching data for subsequent multi-dimensional evaluation, and effectively improving the accuracy and reliability of the evaluation results.

[0009] In some implementations, evaluating the paired data based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results includes: filtering the paired data based on preset evaluation dimensions to obtain filtered paired data corresponding to each evaluation dimension; matching the inference results in the filtered paired data corresponding to each evaluation dimension with the ground truth data to generate data comparison results for each evaluation dimension; and determining the multi-dimensional evaluation results based on the data comparison results of the multiple evaluation dimensions.

[0010] In the above implementation, the paired data can be filtered according to the evaluation dimensions first, then the filtered data of each dimension can be matched and compared, and finally the evaluation results can be determined by combining the comparison results of each dimension. This achieves refined evaluation by dimension, ensures the independence and pertinence of each evaluation dimension, effectively improves the granularity and accuracy of the evaluation, and makes the evaluation results more systematic and interpretable.

[0011] In some implementations, the step of matching the inference results in the filtered paired data corresponding to each evaluation dimension with the ground truth data to generate data comparison results for each evaluation dimension includes: calling the evaluation logic corresponding to each evaluation dimension to compare the spatial position, shape, and / or attribute features between the inference results in the filtered paired data and the ground truth data to generate data comparison results for each evaluation dimension.

[0012] In the above implementation, the evaluation logic specific to each evaluation dimension can be invoked to perform specific fine-grained evaluations on the corresponding filtered paired data, effectively improving the accuracy and depth of the data comparison results for each evaluation dimension.

[0013] In some implementations, the evaluation dimensions include at least one of the following: target type dimension, accuracy dimension, safety dimension, comfort dimension, efficiency dimension, compliance dimension, and interaction dimension.

[0014] In the above implementation, the evaluation dimensions can comprehensively cover the key performance of the assisted driving function from multiple perspectives, and achieve comprehensive coverage of the assisted driving function entity in terms of perception, decision-making, execution and other capabilities, thereby effectively improving the comprehensiveness and business relevance of the evaluation and meeting the evaluation needs in different scenarios.

[0015] In some implementations, displaying the functional entity evaluation report based on the multi-dimensional evaluation results includes: determining descriptive tags for the multi-dimensional evaluation results; the descriptive tags include at least one of the following: evaluation dimension tags determined based on the evaluation dimensions corresponding to the multi-dimensional evaluation results, context tags determined based on the environmental data corresponding to the paired data, and functional entity tags determined based on the entity type of the functional entity to which the inference result belongs; and displaying functional entity evaluation reports categorized by tags based on the descriptive tags.

[0016] In the above implementation, the evaluation results can be categorized and displayed using evaluation dimension tags, context tags, and functional entity tags, thus achieving a structured and tagged presentation of the evaluation report. This effectively improves the readability and searchability of the evaluation report, making it easier for users of different roles to view the evaluation results in specific dimensions or scenarios as needed.

[0017] In some implementations, the method further includes: in response to receiving a selection operation of a description tag in the functional entity evaluation report, displaying detailed evaluation data corresponding to the selected tag.

[0018] In the above implementation, the corresponding detailed evaluation data can be displayed in response to the selection operation of the tags in the evaluation report, realizing the drill-down analysis capability of the evaluation report, which makes it easier for users to locate the root cause of the problem of the functional entity and conduct in-depth analysis, effectively improving the interactivity and practicality of the evaluation report.

[0019] According to a second aspect of the present disclosure, an evaluation apparatus is provided, comprising: an acquisition unit, configured to acquire a reasoning result of a functional entity under test based on environmental data, and truth data corresponding to the environmental data; a pairing unit, configured to pair the reasoning result and the truth data to obtain paired data; an evaluation unit, configured to evaluate the paired data based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results; and a display unit, configured to display a functional entity evaluation report based on the multi-dimensional evaluation results.

[0020] In some implementations, the pairing unit performs pairing processing on the inference result and the truth data to obtain paired data, including: aligning the inference result with the truth data based on the timestamp of the environmental data to obtain the inference result and the truth data with matching timestamps, which are then used as the paired data.

[0021] In some implementations, the evaluation unit evaluates the paired data based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results, including: filtering the paired data based on preset evaluation dimensions to obtain filtered paired data corresponding to each evaluation dimension; matching the inference results in the filtered paired data corresponding to each evaluation dimension with the ground truth data to generate data comparison results for each evaluation dimension; and determining the multi-dimensional evaluation results based on the data comparison results of the multiple evaluation dimensions.

[0022] In some implementations, the evaluation unit matches the inference results in the filtered paired data corresponding to each evaluation dimension with the ground truth data to generate data comparison results for each evaluation dimension. This includes: calling the evaluation logic corresponding to each evaluation dimension to compare the spatial position, shape, and / or attribute features between the inference results in the filtered paired data and the ground truth data to generate data comparison results for each evaluation dimension.

[0023] In some implementations, the evaluation dimensions include at least one of the following: target type dimension, accuracy dimension, safety dimension, comfort dimension, efficiency dimension, compliance dimension, and interaction dimension.

[0024] In some implementations, the display unit displays a functional entity evaluation report based on the multi-dimensional evaluation results, including: determining descriptive tags for the multi-dimensional evaluation results; the descriptive tags include at least one of the following: evaluation dimension tags determined based on the evaluation dimensions corresponding to the multi-dimensional evaluation results, context tags determined based on the environmental data corresponding to the paired data, and functional entity tags determined based on the entity type of the functional entity to which the inference result belongs; and displaying functional entity evaluation reports categorized by tags based on the descriptive tags.

[0025] In some implementations, the display unit is further configured to: in response to receiving a selection operation of a description tag in the functional entity evaluation report, display detailed evaluation data corresponding to the selected tag.

[0026] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the evaluation method described above.

[0027] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein instructions in the storage medium are loaded by a processor and executed using the evaluation method described above.

[0028] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described evaluation method.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0031] Figure 1 This is a flowchart illustrating an evaluation method according to some embodiments of the present disclosure.

[0032] Figure 2 This is a schematic diagram illustrating the processing of inference results and truth data in an evaluation method according to some embodiments of the present disclosure.

[0033] Figure 3 This is a flowchart illustrating a method for generating multi-dimensional evaluation results according to some embodiments of the present disclosure.

[0034] Figure 4 This is a schematic diagram of a concurrent evaluation framework in an evaluation method according to some embodiments of the present disclosure.

[0035] Figure 5 This is a flowchart illustrating an evaluation report in an evaluation method according to some embodiments of the present disclosure.

[0036] Figure 6 This is a schematic diagram illustrating a functional entity evaluation report in an evaluation method according to some embodiments of the present disclosure.

[0037] Figure 7 This is a flowchart illustrating an evaluation method according to some embodiments of the present disclosure.

[0038] Figure 8 This is a block diagram illustrating an evaluation apparatus according to some embodiments of the present disclosure.

[0039] Figure 9 This is a block diagram illustrating a vehicle for evaluation according to some embodiments of the present disclosure. Detailed Implementation

[0040] Some exemplary embodiments of this disclosure will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the flowcharts shown in the drawings are merely illustrative and do not necessarily include all contents and steps, nor do they necessarily have to be performed in the described order or in the order of step numbers. For example, some steps may be decomposed, while others may be combined or partially combined, and multiple steps may be interchanged or performed simultaneously; therefore, the actual order of execution may change depending on the actual situation. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0041] The embodiments described below, which are some of the embodiments of this disclosure, do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0042] It should be noted that all information, data (including but not limited to data used for inference results, truth data, evaluation reports, vehicle sensor data, etc.) and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data comply with the relevant laws, regulations, and standards of the relevant countries and regions. In particular, the vehicle sensor data, truth data, etc. involved in this application were obtained and used with the user's full knowledge and explicit consent.

[0043] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0044] Figure 1 This is a flowchart illustrating an evaluation method according to some embodiments of the present disclosure, such as... Figure 1 As shown, the evaluation method can be applied to electronic devices, including but not limited to in-vehicle terminals, vehicles, smartphones, smart tablets, wearable devices, desktop computers, laptops, smart speakers, and other terminal devices. It can also include server-side components such as local servers and cloud servers, which can be deployed in a computer cluster or a combination of multiple computers. The evaluation method may include the following steps.

[0045] In step S110, the reasoning result of the functional entity under test based on environmental data and the truth data corresponding to the environmental data are obtained.

[0046] In this embodiment of the disclosure, the reasoning result output by the functional entity under test under given environmental data input can be obtained as the actual performance of the functional entity under test; the truth data corresponding to the environmental data can also be obtained as the standard answer or standard output of the evaluation benchmark.

[0047] For example, in scenarios where functional entities for target detection are evaluated, environmental data can include vehicle environmental data collected by vehicle sensors, inference results can be category labels of target objects (such as vehicles, pedestrians, motor vehicles, bicycles, etc.) output by vehicle-assisted driving related functional models, and ground truth data can be the accurate category of target objects in the vehicle environmental data labeled by humans.

[0048] For example, environmental data could be 10 seconds of video data collected by a vehicle camera from a section of road. The inference result could be the lane line recognition type identified by the vehicle's lane line detection model based on the video data. The ground truth data could be the actual lane line type that has been pre-labeled for this section of road.

[0049] In some embodiments of this disclosure, the environmental data includes vehicle sensor data, and the functional entity under test includes models, software, chips, and / or systems for driver assistance.

[0050] In this embodiment of the disclosure, the evaluation scenario to which this method is applicable can be an in-vehicle environment, and the environmental data can be raw or pre-processed data collected by various sensors mounted on the vehicle. For example, vehicle sensor data may include 3D point cloud data collected by radar, image data collected by camera, target speed and distance data collected by radar, near-range obstacle data collected by ultrasonic sensor, and vehicle attitude and acceleration data collected by inertial measurement unit, etc.

[0051] The functional entity under test can be one or more technical components serving the driver assistance function. The functional entity under test can take the form of an algorithm model, software program, hardware chip, or a complete system composed of the above components. For example, the functional entity under test can be a lane detection model based on deep learning, path planning software running on an on-board computing platform, driver assistance chip, or a complete driver assistance system including the above models, software, and / or chips.

[0052] The functional entity under test in this embodiment can be constructed based on artificial intelligence (AI) technology. AI refers to the technology of using computer systems to simulate human intelligent behavior. Through learning and training on large amounts of data, it enables computer systems to possess capabilities such as perception, reasoning, and decision-making. AI technology can be applied in the vehicle field. AI models, represented by deep learning, such as convolutional neural networks and recurrent neural networks, can be deployed in tasks such as environmental perception, behavioral decision-making, and trajectory prediction in vehicles. By learning and reasoning from environmental data collected by vehicle sensors, these models can output inference results such as target detection results, lane detection results, traffic light status recognition results, and drivable area segmentation results, thereby supporting the realization of functions such as assisted driving.

[0053] In an exemplary embodiment, functional entities of any form can be evaluated individually, or multiple functional entities can be evaluated in combination.

[0054] Through the embodiments of this disclosure, it can be clearly defined that the environmental data is vehicle sensor data, and that the functional entities to be tested are various types of functional entities related to assisted driving, so that the evaluation method can be applied to the field of assisted driving, effectively improving the pertinence and practicality of the evaluation method.

[0055] In step S120, the inference result and the truth data are paired to obtain paired data.

[0056] In this embodiment, the inference results and truth data can be correlated one-to-one according to preset rules to form structured paired data for subsequent comparison. Pairing can eliminate problems such as mismatch in data quantity, order, or granularity, making the inference results and truth data comparable in semantics or format. The preset rules can be, for example, time matching or spatial scene matching.

[0057] For example, the target category label (e.g., truck) corresponding to the time period t1 in the model output can be paired with the target category label (e.g., truck) in the ground truth data labeled in the time period t1 to form a pair of data. The content of the pair of data can be "time period t1 - inference result is truck - ground truth data is truck".

[0058] For example, if the model outputs a reasoning result of pedestrian for time period t2, and the ground truth data does not label the target type for time period t2, then a pair of data can be formed. The content of the pair of data can be "time period t2 - reasoning result is pedestrian - ground truth data is empty".

[0059] In step S130, the paired data is evaluated based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results.

[0060] In this embodiment, based on a pre-defined framework of multiple evaluation dimensions, paired data can be calculated and statistically analyzed to obtain quantitative indicators for each dimension, ultimately outputting multi-dimensional evaluation indicator results. These evaluation dimensions can reflect different evaluation focuses, such as specific target type dimensions (e.g., obstacle detection, traffic light detection), specific accuracy dimensions (e.g., detection range accuracy within 2 meters, 5 meters, etc.), and safety dimensions (e.g., environmental data without traffic accidents, environmental data with traffic accidents). Each evaluation dimension can be independently calculated and its indicator value output.

[0061] In this embodiment of the disclosure, during the evaluation of paired data based on multiple preset evaluation dimensions, a trained artificial intelligence model can be used to analyze and evaluate the paired data and / or the multi-dimensional evaluation results to obtain evaluation results.

[0062] In an exemplary embodiment, an evaluator built on artificial intelligence technology can be invoked to evaluate the paired data to generate the multi-dimensional evaluation results. The evaluator may include a trained neural network model to extract deep feature representations of the inference results and ground truth data in terms of spatial location, shape, and semantic attributes from the paired data, and automatically execute the corresponding evaluation logic based on the extracted deep feature representations to output the data comparison results.

[0063] For example, paired data can be input into a deep neural network evaluation model corresponding to an evaluation dimension. The model can then predict the accuracy level and error type of the inference results, or directly output the evaluation index results, thereby achieving intelligent and fine-grained evaluation of the performance of functional entities.

[0064] In an exemplary embodiment, paired data or a portion of paired data can generate evaluation results under one or more evaluation dimensions, and each paired data can be classified as a true positive, false positive, false negative, or other types.

[0065] In step S140, a functional entity evaluation report is displayed based on the multi-dimensional evaluation results.

[0066] In this embodiment of the disclosure, multi-dimensional evaluation results can be summarized or categorized and presented visually to generate a functional entity evaluation report for users or the system.

[0067] In an exemplary embodiment, the evaluation report can be presented according to cases containing paired data. Each case can have a label for an evaluation dimension (e.g., a case can be used to evaluate the target type dimension for obstacles, the accuracy dimension for detection within a 2-meter range, and the safety dimension). The case can include a piece of raw environmental data, the corresponding inference result and ground truth data, and the evaluation result.

[0068] As can be seen from the above steps, the evaluation method provided in this disclosure can pair the obtained inference results of the functional entity under test with the true data, and conduct systematic evaluation based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results and display evaluation reports. This achieves comprehensive, objective, and quantitative automated evaluation of the functional entity under test, improves the completeness and credibility of the evaluation results, provides a standardized evaluation framework for the performance evaluation of functional entities, and provides quantitative basis for the optimization of functional entities.

[0069] In an exemplary embodiment, obtaining the reasoning result of the functional entity under test based on environmental data may include: obtaining the reasoning result output by the functional entity under test (such as a perception model, planning model, or control model) after processing the environmental data (such as camera images, LiDAR point clouds, etc.) collected by vehicle sensors. The reasoning result may be, for example, target detection result (such as the position, size, and category of obstacles), lane line detection result, traffic light status recognition result, trajectory prediction result, driving decision result, etc.

[0070] The functional entity under test can be run through Model-in-the-Loop (MIL) or Software-in-the-Loop (SIL) testing, taking environmental data as input and obtaining corresponding inference outputs. The inference results can include one or more inference cases, each corresponding to the inference output of environmental data over a continuous time range. Each inference case can contain one or more timestamps and the inference results at each timestamp, facilitating evaluation processing on a case-by-case basis.

[0071] In an exemplary embodiment, the ground truth data corresponding to the environmental data can be determined through manual annotation. Specifically, annotators can precisely annotate the environmental data collected by vehicle sensors, using the annotated data as ground truth data. Annotation content may include, for example, the bounding box of target objects in the image, object category labels, lane line positions, traffic light status, etc. The format of the ground truth data can correspond to the format of the inference result to facilitate subsequent comparison and evaluation.

[0072] The truth data may include one or more truth cases, each of which may be associated with a corresponding reasoning case of the reasoning result, for example, by case identifier or timestamp range, thereby forming a set of paired cases for evaluation in subsequent steps.

[0073] In an exemplary embodiment, the inference results and / or truth data may be preprocessed before being aggregated into a preset storage system (such as a data warehouse). For example, the inference results and / or truth data may undergo quality control and / or filtering processes, and then be aggregated in a preset data warehouse after real-time and / or offline synchronization.

[0074] For the inference results, data parsing (such as parsing how many inferred target types, objects, etc. are inferred in each frame) can be performed, followed by quality inspection and filtering to remove abnormal inference result data with format errors, missing fields, or exceeding the preset confidence range. Then, the data can be written to the storage system through a real-time synchronization channel or a batch offline synchronization channel.

[0075] For ground truth data, the manually labeled results can be parsed (e.g., parsing how many labeled target types, objects, etc. are in each frame) and quality checked to remove invalid data that is inconsistent with the labeling or does not conform to the labeling specifications, and then synchronized to the storage system.

[0076] In an exemplary embodiment, the inference results and truth data can be stored in layers in the storage system, for example, in the original inference data layer and the original truth data layer, respectively, to facilitate data traceability and subsequent independent management.

[0077] Figure 2 This is a schematic diagram illustrating the processing of inference results and truth data in an evaluation method according to some embodiments of the present disclosure.

[0078] like Figure 2 As shown, it includes inference case data 201, inference data workflow 202, truth case data 203, truth data workflow 204, and data warehouse 205. Among them, inference data workflow 202 and truth data workflow 204 can be two parallel processing links.

[0079] refer to Figure 2 The reasoning case data 201 may include multiple reasoning cases. The processing of the reasoning case data 201 may include: the system reads multiple reasoning cases, and for each reasoning case, it may be processed by the corresponding data parsing module, the first quality inspection filtering module and the first real-time / offline synchronization module in the reasoning data workflow 202, and the processed reasoning cases are summarized and written into the original reasoning data layer of the data warehouse 205.

[0080] The data parsing module can transform the original inference output into structured inference data; the first quality inspection and filtering module can perform quality checks on the parsed inference data and remove abnormal data with format errors or missing fields; the first real-time / offline synchronization module can synchronize the quality-inspected and filtered inference data to the original inference data layer of the data warehouse in real time or in batches.

[0081] The truth case data 203 may include multiple truth cases. The processing of the truth case data 203 may include: the system reads multiple truth cases, and for each truth case, it may be processed by the annotation and parsing module, the second quality inspection and filtering module and the second real-time / offline synchronization module in the truth data workflow 204, and the cleaned truth cases are summarized and written into the original truth data layer of the data warehouse 205.

[0082] The annotation parsing module can convert manually annotated raw annotation files into structured truth data; the second quality inspection and filtering module can perform quality checks on the parsed truth data and remove invalid data that is inconsistent with the annotations or does not meet the specifications; the second real-time / offline synchronization module can synchronize the truth data after quality inspection and filtering to the original truth data layer of the data warehouse in real time or in batches.

[0083] Through the embodiments disclosed herein, it can be ensured that both the inference results and the true data have undergone standardized cleaning and verification before entering the data warehouse 205.

[0084] Figure 2 Other aspects of the embodiments can be found in the other embodiments described above.

[0085] In some embodiments of this disclosure, the step of pairing the inference result and the truth data to obtain paired data includes: aligning the inference result with the truth data based on the timestamp of the environmental data to obtain the inference result and the truth data with matching timestamps, which are then used as the paired data.

[0086] In this embodiment of the disclosure, the timestamp information recorded during data acquisition can be used as a synchronization benchmark to establish a correlation between the inference results and the true data at the same time or within a preset tolerance window, thereby eliminating time misalignment caused by different acquisition frequencies, processing delays or transmission jitter, and ensuring that each pair of data reflects the system output and the true state at the same physical time.

[0087] For example, if a camera exposes an image at time t1 and generates a frame, the perception model outputs an inference result (such as an object detection result) based on this. The annotation result in the ground truth data, where the difference between the timestamp and t1 is within a preset tolerance window (e.g., 500ms or 1000ms), can be paired with the inference result corresponding to that frame, and the two can be associated with the same timestamp to complete the alignment.

[0088] After being timestamped and aligned, the inference results and truth data can be used as paired data. Paired data can be viewed as (inference result, truth data) pairs matched in the time dimension, where the inference result and truth data in the pair correspond to the same time reference. Paired data can serve as input for subsequent multi-dimensional evaluations, avoiding evaluation distortion caused by time misalignment.

[0089] Through the embodiments of this disclosure, timestamps can be used to align inference results with truth data, ensuring that the two are matched in time sequence, avoiding evaluation errors caused by time mismatch, providing accurate matching data for subsequent multi-dimensional evaluation, and effectively improving the accuracy and reliability of evaluation results.

[0090] Figure 3 This is a flowchart illustrating a method for generating multi-dimensional evaluation results according to some embodiments of the present disclosure.

[0091] Figure 3 As shown, in some embodiments of this disclosure, the process of evaluating the paired data based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results may include the following steps.

[0092] Step S310: Filter the pairing data based on preset evaluation dimensions to obtain filtered pairing data corresponding to each evaluation dimension.

[0093] In this embodiment, the established paired dataset can be divided according to the data filtering conditions defined for each preset evaluation dimension, retaining only the paired items that satisfy the constraints of that dimension, thereby constructing a dedicated subset for each dimension. The filtering operation ensures that subsequent evaluations of each dimension are performed within their dedicated, dimension-related data range, avoiding interference from irrelevant data in the evaluation results.

[0094] Step S320: Match the inference results in the filtered paired data corresponding to each evaluation dimension with the true data to generate data comparison results for each evaluation dimension.

[0095] In this embodiment of the disclosure, within the range of filtered paired data specific to each dimension, the inference results can be matched one by one with the corresponding truth results to determine the differences or consistency between the two in terms of spatial location, shape and / or attribute features, and a structured comparison result reflecting the performance under the evaluation dimension can be generated based on the matching results.

[0096] In an exemplary embodiment, the matching process may involve spatial alignment or threshold determination. For example, in the dimension of "detection accuracy for vehicles", the predicted vehicle bounding box of each frame after filtering can be matched with the ground truth vehicle bounding box based on the intersection-union ratio to generate comparison results such as the number of correct detections, the number of false alarms and / or the number of missed detections for each frame in this dimension.

[0097] Step S330: Determine the multi-dimensional evaluation result based on the data comparison results of the multiple evaluation dimensions.

[0098] In this embodiment, the data comparison results generated by each evaluation dimension can be comprehensively statistically analyzed and aggregated to form a multi-dimensional evaluation result covering all preset dimensions. The multi-dimensional evaluation result can be a set of independent indicator values ​​for each dimension, or it can be a comprehensive score synthesized according to business requirements.

[0099] For example, the results of comparisons of vehicle detection accuracy, pedestrian detection accuracy, and traffic light recognition accuracy can be presented in parallel. Alternatively, a comprehensive perception score can be calculated based on preset weights as a multi-dimensional evaluation result.

[0100] Through the embodiments of this disclosure, the paired data can be filtered according to the evaluation dimensions, then the filtered data of each dimension can be matched and compared, and finally the evaluation results can be determined by combining the comparison results of each dimension. This achieves refined evaluation by dimension, ensures the independence and pertinence of each evaluation dimension, effectively improves the granularity and accuracy of the evaluation, and makes the evaluation results more systematic and interpretable.

[0101] In some embodiments of this disclosure, the step of matching the inference results in the filtered paired data corresponding to each evaluation dimension with the ground truth data to generate data comparison results for each evaluation dimension includes: calling the evaluation logic corresponding to each evaluation dimension to compare the spatial position, shape, and / or attribute features between the inference results in the filtered paired data and the ground truth data to generate data comparison results for each evaluation dimension.

[0102] In this embodiment of the disclosure, within a subset of data that has been filtered by dimension, a dedicated evaluation algorithm for each dimension can be enabled to quantitatively compare the inference results with the spatial location, geometric shape, and / or semantic attributes of the true data.

[0103] Among these, spatial location comparison can calculate coordinate difference, distance error, or grid overlap rate; shape comparison can calculate bounding box intersection-union ratio, contour similarity, or polygon area difference; attribute feature comparison can verify category consistency, color matching, or motion state accuracy. The comparison results can record the matching status and / or deviation value of each paired data item, constituting the data comparison results for that dimension.

[0104] For example, regarding the accuracy of lane line detection, the evaluation logic can calculate the positional error between the predicted lane line sampling points and the true sampling points along the normal direction, and can also statistically analyze information such as average error and maximum error to form a comparison result for this dimension.

[0105] Through the embodiments of this disclosure, the evaluation logic specific to each evaluation dimension can be invoked to perform specific fine-grained evaluations on the corresponding filtered paired data, effectively improving the accuracy and depth of the data comparison results for each evaluation dimension.

[0106] In some embodiments of this disclosure, the evaluation dimensions include at least one of the following: target type dimension, accuracy dimension, safety dimension, comfort dimension, efficiency dimension, compliance dimension, and interaction dimension.

[0107] In this embodiment of the disclosure, a dimension that can reflect the performance of the functional entity under test can be selected based on the task positioning and evaluation objectives of the entity under test, so as to construct a targeted evaluation framework.

[0108] Among them, the target type dimension can be an evaluation dimension based on the semantic category to which the detected target belongs. It can measure the ability of functional entities to perceive or respond to different types of objects. For example, the target type dimension can include vehicles, pedestrians, motor vehicles, bicycles, traffic signs, traffic lights, etc.

[0109] Accuracy dimension can be an evaluation dimension based on the spatial distance range between the detected target and the vehicle. It can be used to evaluate the perception accuracy and reliability of functional entities within different distance ranges.

[0110] Safety can be an evaluation dimension centered on risk avoidance or collision prevention. Comfort can be an evaluation dimension centered on the user's riding experience. Efficiency can be an evaluation dimension centered on the efficiency of functional entities in completing tasks. Compliance can be an evaluation dimension centered on whether it complies with traffic regulations. Interaction can be an evaluation dimension centered on collaborative interaction with other traffic participants (such as other vehicles).

[0111] For example, when evaluating the assisted following driving function, a safety dimension can be selected to measure the ability to control the following distance, and a comfort dimension can be selected to quantify the rate of speed change during the acceleration process of following the vehicle, so as to describe the effect of the assisted following driving function from multiple perspectives.

[0112] Through the embodiments disclosed herein, the evaluation dimensions can comprehensively cover the key performance of the assisted driving function from multiple perspectives, achieving comprehensive coverage of the assisted driving function entity's capabilities in perception, decision-making, execution, and other aspects, thereby effectively improving the comprehensiveness and business relevance of the evaluation and meeting the evaluation needs in different scenarios.

[0113] In some embodiments of this disclosure, the evaluation is implemented by multiple evaluators that are executed in parallel, with at least one evaluator corresponding to each evaluation dimension.

[0114] In this embodiment, multiple evaluators can be configured, with at least one evaluator corresponding to each evaluation dimension. Multiple evaluators can be started simultaneously and process paired data independently. Multi-dimensional evaluation of the paired data can be performed by parallel execution of each evaluator. This decomposes the overall evaluation task into multiple independent evaluation units that can run concurrently. Each evaluator is responsible for calculating the metrics of its corresponding evaluation dimension, thereby reducing the total time required for full-dimensional evaluation.

[0115] The evaluator can be a computational module or process that can independently execute specific evaluation logic. It can receive paired data and output the data comparison results or indicators of the evaluation dimension it is responsible for. Each evaluator can encapsulate the filtering conditions, matching algorithms and / or statistical rules required for that dimension.

[0116] In a parallel architecture, each pre-defined evaluation dimension can be bound to one or more evaluators, ensuring that each dimension has a dedicated evaluator responsible for executing its evaluation logic. When there are many evaluation dimensions or a large amount of paired data, parallel execution can significantly reduce the total evaluation time.

[0117] In an exemplary embodiment, multiple evaluators that execute in parallel can be deployed within the framework. Each evaluator may include a filter, a matcher, and a metric calculator. The filter can be used to filter paired data according to the evaluation dimension corresponding to that evaluator. The matcher can compare the filtered inference results with the true results based on preset evaluation logic. The metric calculator can be used to calculate the evaluation metrics for that evaluation dimension based on the comparison results.

[0118] Through the embodiments of this disclosure, multiple parallel evaluators can be used to implement evaluation, realizing independent parallel computation of each evaluation dimension, effectively improving the execution efficiency of multi-dimensional evaluation, ensuring the independence and accuracy of evaluation results of each dimension, facilitating the expansion and maintenance of the evaluation system, and adapting to the performance requirements of large-scale evaluation scenarios.

[0119] In some embodiments of this disclosure, the multi-dimensional evaluation results include at least one of the following: number of true positives, number of false positives, number of false negatives, precision, and recall.

[0120] In this embodiment of the disclosure, the number of true positives, false positives, and false negatives can be counted based on the matching status between the inference results and the true values ​​in the paired data, and precision and recall can be calculated based on this, with at least one of them being used as the content of the multi-dimensional evaluation index result.

[0121] For example, the single-frame pairing data of the object detection task can be traversed. If the predicted box matches the ground truth box, it is counted as a true positive. If the ground truth box does not have a corresponding predicted box, it is counted as a false negative. If the predicted box does not have a corresponding ground truth box, it is counted as a false positive. Then, the precision (number of true positives divided by the total number of positive predictions) and recall (number of true positives divided by the total number of ground truth instances) can be calculated separately as evaluation indicators for the perceptual accuracy dimension.

[0122] The embodiments disclosed herein provide standardized quantitative evaluation indicators, making the evaluation results comparable and statistically significant. This facilitates horizontal and vertical comparative analysis of the performance of different functional entities or the same entity under different conditions, effectively improving the objectivity and engineering applicability of the evaluation results.

[0123] Figure 4 This is a schematic diagram of a concurrent evaluation framework in an evaluation method according to some embodiments of the present disclosure.

[0124] like Figure 4 As shown, the system includes a data preparation module 401, a data processing module 402, and a result aggregation module 403. The data preparation module 401 includes a truth data loader 4011 and an inference result loader 4012. The data processing module 402 includes parallelizable evaluators 4021 and 4022, which can be used for different evaluation dimensions. Each evaluator 4021 and 4022 can contain filters, matchers, and indicator calculators. The result aggregation module 403 includes an intermediate database 4031.

[0125] like Figure 4 As shown, in the data preparation module 401, the system can load two types of source data: truth data containing annotation and calibration information is loaded via the truth data loader 4011, and inference results are loaded via the inference result loader 4012. The truth data loader 4011 and the inference result loader 4012 can send the loaded data to the data processing module 402.

[0126] In the data processing module 402, evaluators 4021 and 4022 can process data in parallel. Each evaluator first filters the received data according to its corresponding evaluation dimension, obtaining filtered data specific to that dimension. Then, a matcher uses the evaluation logic preset for the corresponding evaluation dimension to compare the inference results in the filtered data with the true results to determine the matching relationships such as correct detection, false detection, and missed detection. Finally, an indicator calculator calculates the evaluation indicators for that evaluation dimension based on the matching results.

[0127] The data processing module 402 can send the results output by the indicator calculators in different evaluators to the result aggregation module 403.

[0128] In the results aggregation module 403, the data received from the indicator calculator can be stored in the intermediate database 4031. In the intermediate database 4031, the data can be integrated into three types of result files for output: indicator results, matching difference results, and mixed matrix statistics (which may include the number of true positives, the number of false positives, and / or the number of false negatives) to complete the transformation from raw data to evaluation indicators.

[0129] Figure 4 Other aspects of the embodiments can be found in the other embodiments described above.

[0130] Figure 5 This is a flowchart illustrating an evaluation report in an evaluation method according to some embodiments of the present disclosure.

[0131] like Figure 5 As shown in some embodiments of this disclosure, the process of displaying a functional entity evaluation report based on the multi-dimensional evaluation results may include the following steps.

[0132] Step S510: Determine the description label of the multi-dimensional evaluation result; the description label includes at least one of the following: evaluation dimension label determined according to the evaluation dimension corresponding to the multi-dimensional evaluation result, context label determined according to the environmental data corresponding to the paired data, and functional entity label determined according to the entity type of the functional entity to which the inference result belongs.

[0133] In this embodiment of the disclosure, the evaluation dimension label of the multi-dimensional evaluation result can be determined according to the evaluation dimension corresponding to the multi-dimensional evaluation result.

[0134] This involves binding the generated multi-dimensional evaluation metrics results to their source evaluation dimensions, and using the name or identifier of the corresponding evaluation dimension as the evaluation dimension label for that set of metrics results. Evaluation dimension labels allow users of the evaluation report to quickly identify the evaluation angle corresponding to that set of metrics, thus providing a clear and concise classification index in the multi-dimensional parallel output results.

[0135] For example, if a set of indicator results (which may include information such as the number of true positives, false positives, and / or false negatives) comes from the data comparison results under the "safety dimension", then the evaluation dimension label of the "safety dimension" can be added to the set of indicators; if a set of indicator results (which may include information such as the number of true positives, false positives, and / or false negatives) comes from the "comfort dimension", then the evaluation dimension label of the "comfort dimension" can be added to the set of indicators.

[0136] In this embodiment of the disclosure, the context label of the corresponding multi-dimensional evaluation result can be determined based on the environmental data corresponding to the pairing data. The context label may include at least one of the following: scene type and time period.

[0137] Specifically, scenario description information can be extracted from the environmental data that generated the paired data and converted into contextual labels to be attached to the relevant evaluation metric results. Scenario types can include environmental scenarios such as road structure, weather, and traffic density, as well as driving scenarios such as left turns and swerving. Time periods can be divided according to the time of data collection, such as "daytime," "nighttime," and "dawn / dusk," thus enabling the metric results to carry semantic identifiers of their operational context.

[0138] In this embodiment of the disclosure, the functional entity label of the corresponding multi-dimensional evaluation result can be determined according to the entity type of the functional entity to which the reasoning result belongs.

[0139] Specifically, the entity type or version identifier corresponding to the functional entity under evaluation can be converted into a functional entity label, and associated with the multi-dimensional evaluation results produced by that functional entity. Functional entity labels can be used to distinguish inference results and their metrics from different sources when evaluating multiple functional entities or multiple versions simultaneously. For example, if the functional entity under evaluation is the first perception model "V1", then the functional entity label "V1" can be attached to all its metric results; if the evaluation object is the same algorithm A on different chips, then the labels "Algorithm A - Chip 1" and "Algorithm A - Chip 2" can be attached respectively.

[0140] Step S520: Based on the description tags, display the functional entity evaluation report categorized by tags.

[0141] In this embodiment, the generated evaluation dimension tags, context tags, and / or functional entity tags can be used as classification keys to structure and group the results of multi-dimensional evaluation indicators, forming an evaluation report indexed by tags. The evaluation report can be displayed by tag classification, allowing users to quickly locate the evaluation indicator results under the corresponding description tag by selecting a specific description tag (including evaluation dimension tags, context tags, and / or functional entity tags), enabling multi-dimensional and multi-condition cross-query and comparative analysis.

[0142] Through the embodiments disclosed herein, evaluation results can be categorized and displayed using evaluation dimension tags, context tags, and functional entity tags, thereby achieving a structured and tagged presentation of the evaluation report. This effectively improves the readability and searchability of the evaluation report, making it easier for users with different roles to view evaluation results in specific dimensions or scenarios as needed.

[0143] In some embodiments of this disclosure, the method further includes: in response to receiving a selection operation of a description tag in the functional entity evaluation report, displaying detailed evaluation data corresponding to the selected tag.

[0144] In this embodiment of the disclosure, when a user selects one or more evaluation dimension labels (such as "security dimension"), context labels (such as "rainy day"), and / or functional entity labels (such as "model V1") on the evaluation report interface by clicking, checking, or hovering, the system can filter the underlying paired data, intermediate statistics, or original indicator sequences according to the combination conditions of the selected labels, and can present the fine-grained information obtained after filtering in the form of an expanded list, a single-frame snapshot, or an error distribution map.

[0145] The detailed evaluation data may include specific cases, timestamp-level error details, and other content.

[0146] For example, if a user clicks the "Nighttime" context label and the "Pedestrian Type Dimension" label in the evaluation report, the system can display the detection and matching details of pedestrian objects in each frame of the image in the nighttime scene, the visual annotations of missed and false detection targets, and the corresponding precision and recall curves so that users can analyze the reasons for failure.

[0147] Through the embodiments of this disclosure, detailed evaluation data can be displayed in response to the selection operation of tags in the evaluation report, realizing the drill-down analysis capability of the evaluation report, which makes it easier for users to locate the root cause of the problem of functional entities and conduct in-depth analysis, effectively improving the interactivity and practicality of the evaluation report.

[0148] Figure 6 This is a schematic diagram illustrating a functional entity evaluation report in an evaluation method according to some embodiments of the present disclosure.

[0149] like Figure 6 As shown, it can include two branches: functional entity dimension and scenario dimension, which are used to subdivide and classify the functional entity evaluation report from multiple perspectives.

[0150] At the functional entity level, multi-dimensional evaluation results can be categorized based on the type of functional entity, such as "Model V1," "Model V2," and "Model V3." Further classification can then be performed according to the detected target type, such as Vulnerable Road Users (VRUs), Vehicles, and Cars (CARs). For specific vehicle types, more refined stratification based on accuracy (e.g., detection distance 0-20m, detection distance 20-50m) can be added.

[0151] Vehicles and cars can be different obstacles that the model can recognize. (Reference) Figure 6 Model V1 can be used to detect obstacles such as vulnerable road users, Model V2 can be used to detect obstacles such as vehicles, and Model V3 can be used to detect obstacles such as cars. It should be noted that Model V2 and Model V3 differ in their specific recognition methods. For example, Vehicle is a broad category of "vehicles," and Model V2 only needs to determine "this is a vehicle target"; Car is a subcategory of "cars," and Model V3 needs to further distinguish whether it is a truck, bus, or motorcycle.

[0152] At the scene level, multi-dimensional evaluation results can be categorized into "daytime" and "nighttime" based on time period or lighting conditions. Within the "daytime" scene, classification can also be based on the type of the detected target, such as Vulnerable Road Users (VRUs) and Vehicles. For vehicle targets, a functional entity type dimension (e.g., Model V1, Model V2) can be introduced, combined with an accuracy dimension (e.g., detection distance 0-20m, detection distance 20-50m) for deeper analysis.

[0153] This multi-dimensional tree structure supports comprehensive, fine-grained analysis of the performance of functional entities.

[0154] Figure 6 Other aspects of the embodiments can be found in the other embodiments described above.

[0155] Figure 7 This is a flowchart illustrating an evaluation method according to some embodiments of the present disclosure.

[0156] like Figure 7 As shown, it includes a multi-source heterogeneous data module 701, a data parsing and data storage module 702, a calculation module 703, and a report generation module 704.

[0157] like Figure 7 As shown, the multi-source heterogeneous data module 701 can aggregate inference inputs (such as calibration information, sensor acquisition data, and software packages) and truth / evaluator inputs (such as labeled truth values ​​and evaluation metrics) to form the data foundation for evaluation. The multi-source heterogeneous data module 701 can send the aggregated data to the data parsing and loading module 702.

[0158] In the data parsing and loading module 702, data processing can be performed using computing engine interfaces (such as streaming computing engine interfaces and batch processing computing engine interfaces). Parsing techniques are used to transform the received data into structured inference results and truth data. The data parsing and loading module 702 can then send the inference results and truth data to the computing module 703.

[0159] In the calculation module 703, the evaluator can be used to execute the evaluation logic in parallel to generate intermediate results (i.e., multi-dimensional evaluation results), and the intermediate results can be integrated into indicator results, matching difference results, etc. and output to the report generation module 704.

[0160] In the report generation module 704, the intermediate results generated by each evaluator can be summarized, the summarized evaluation results can be processed, and scenario reports, case reports, tag reports or other custom reports can be automatically generated to complete the automated evaluation process from data input to evaluation report output.

[0161] Figure 7 Other aspects of the embodiments can be found in the other embodiments described above.

[0162] It should be noted that the above figures are merely illustrative representations of the processes included in methods according to some embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0163] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0164] Figure 8 This is a block diagram illustrating an evaluation apparatus according to some embodiments of the present disclosure. (Refer to...) Figure 8 The device includes: an acquisition unit 801, a pairing unit 802, an evaluation unit 803, and a display unit 804.

[0165] The acquisition unit 801 is used to acquire the reasoning result of the functional entity under test based on environmental data, and the truth data corresponding to the environmental data; the pairing unit 802 is used to pair the reasoning result and the truth data to obtain paired data; the evaluation unit 803 is used to evaluate the paired data based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results; and the display unit 804 is used to display the functional entity evaluation report based on the multi-dimensional evaluation results.

[0166] In some embodiments of this disclosure, the pairing unit 802 performs pairing processing on the inference result and the truth data to obtain paired data, including: aligning the inference result and the truth data based on the timestamp of the environmental data to obtain the inference result and the truth data with matching timestamps, which are used as the paired data.

[0167] In some embodiments of this disclosure, the evaluation unit 803 evaluates the paired data based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results, including: filtering the paired data based on preset evaluation dimensions to obtain filtered paired data corresponding to each evaluation dimension; matching the inference results in the filtered paired data corresponding to each evaluation dimension with the ground truth data to generate data comparison results for each evaluation dimension; and determining the multi-dimensional evaluation results based on the data comparison results of the multiple evaluation dimensions.

[0168] In some embodiments of this disclosure, the evaluation unit 803 matches the inference results in the filtered paired data corresponding to each evaluation dimension with the true data to generate data comparison results for each evaluation dimension, including: calling the evaluation logic corresponding to each evaluation dimension to compare the spatial position, shape and / or attribute features between the inference results in the filtered paired data and the true data to generate data comparison results for each evaluation dimension.

[0169] In some embodiments of this disclosure, the evaluation dimensions include at least one of the following: target type dimension, accuracy dimension, safety dimension, comfort dimension, efficiency dimension, compliance dimension, and interaction dimension.

[0170] In some embodiments of this disclosure, the display unit 804 displays a functional entity evaluation report based on the multi-dimensional evaluation results, including: determining the evaluation dimension label of the multi-dimensional evaluation results based on the evaluation dimensions corresponding to the multi-dimensional evaluation results; determining the context label of the corresponding multi-dimensional evaluation results based on the environmental data corresponding to the paired data, wherein the context label includes at least one of the following: scene type and time period; and displaying the functional entity evaluation report categorized by label based on the description label.

[0171] In some embodiments of this disclosure, the display unit 804 is further configured to: in response to receiving a selection operation of a description tag in the functional entity evaluation report, display detailed evaluation data corresponding to the selected tag.

[0172] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0173] Figure 9 This is a block diagram illustrating a vehicle 900 for evaluation according to some embodiments of the present disclosure. For example, vehicle 900 can be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicle. Vehicle 900 can be a driver-assisted vehicle, a semi-driver-assisted vehicle, or a driver-free vehicle.

[0174] Reference Figure 9 The vehicle 900 may include various subsystems, such as an infotainment system 910, a perception system 920, a decision control system 930, a drive system 940, and a computing platform 950. The vehicle 900 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the vehicle 900 can be interconnected via wired or wireless means.

[0175] In some embodiments, the infotainment system 910 may include a communication system, an entertainment system, and a navigation system, etc.

[0176] The perception system 920 may include several sensors for sensing information about the environment surrounding the vehicle 900. For example, the perception system 920 may include a global positioning system (which may be GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU), lidar, millimeter-wave radar, ultrasonic radar, and a camera device.

[0177] The decision control system 930 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0178] The drive system 940 may include components that provide powered motion to the vehicle 900. In one embodiment, the drive system 940 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0179] Some or all of the functions of the vehicle 900 are controlled by a computing platform 950. The computing platform 950 may include at least one processor 951 and a memory 952, the processor 951 being able to execute instructions 953 stored in the memory 952.

[0180] Processor 951 can be any conventional processor. Processors may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0181] The memory 952 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0182] In addition to instruction 953, memory 952 can also store data, such as road maps, route information, vehicle position, direction, speed, and other data. The data stored in memory 952 can be used by computing platform 950.

[0183] In this embodiment of the disclosure, the processor 951 may execute instructions 953 to complete all or part of the steps of the above-described evaluation method.

[0184] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described evaluation method. The electronic device includes, but is not limited to, terminal devices such as in-vehicle terminals, vehicles, smartphones, smart tablets, wearable devices, desktop computers, laptops, and smart speakers, and may also include server-side components such as local servers and cloud servers, which may be deployed in a computer cluster consisting of one or more computers.

[0185] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein instructions in the storage medium are loaded by a processor and execute all or part of the steps of the evaluation method described above.

[0186] According to a fifth aspect of the present disclosure, a computer program product includes a computer program that, when executed by a processor, implements all or part of the steps of the evaluation method described above.

[0187] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0188] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An evaluation method, characterized in that, include: Obtain the reasoning results of the functional entity under test based on environmental data, as well as the truth data corresponding to the environmental data; The inference results and the truth data are paired to obtain paired data; The paired data is evaluated based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results; The functional entity evaluation report is presented based on the multi-dimensional evaluation results.

2. The method according to claim 1, characterized in that, The process of pairing the inference result and the truth data to obtain paired data includes: The inference result is aligned with the truth data based on the timestamp of the environmental data to obtain the inference result and the truth data with matching timestamps, which are used as the pairing data.

3. The method according to claim 1, characterized in that, The process of evaluating the paired data based on multiple preset evaluation dimensions to generate multi-dimensional evaluation results includes: The paired data is filtered based on preset evaluation dimensions to obtain filtered paired data corresponding to each evaluation dimension. The inference results in the filtered paired data corresponding to each evaluation dimension are matched with the true data to generate data comparison results for each evaluation dimension. The multi-dimensional evaluation results are determined based on the data comparison results of the multiple evaluation dimensions.

4. The method according to claim 3, characterized in that, The step of matching the inference results in the filtered paired data corresponding to each evaluation dimension with the ground truth data to generate data comparison results for each evaluation dimension includes: The evaluation logic corresponding to each evaluation dimension is invoked to compare the spatial location, shape, and / or attribute features between the inference results in the filtered paired data and the true data, thereby generating data comparison results for each evaluation dimension.

5. The method according to claim 1, 3, or 4, characterized in that, The evaluation dimensions include at least one of the following: target type dimension, accuracy dimension, safety dimension, comfort dimension, efficiency dimension, compliance dimension, and interaction dimension.

6. The method according to claim 1, characterized in that, The presentation of the functional entity evaluation report based on the multi-dimensional evaluation results includes: Determine the description tags for the multi-dimensional evaluation results; the description tags include at least one of the following: evaluation dimension tags determined based on the evaluation dimensions corresponding to the multi-dimensional evaluation results, context tags determined based on the environmental data corresponding to the paired data, and functional entity tags determined based on the entity type of the functional entity to which the inference result belongs. Based on the description tags, a functional entity evaluation report categorized by tag is displayed.

7. The method according to claim 6, characterized in that, The method further includes: In response to receiving a selection operation for the description tag in the evaluation report of the functional entity, the detailed evaluation data corresponding to the selected tag is displayed.

8. An evaluation device, characterized in that, include: The acquisition unit is used to acquire the reasoning result of the functional entity under test based on environmental data, as well as the truth data corresponding to the environmental data; A pairing unit is used to pair the inference result and the truth data to obtain paired data; The evaluation unit is used to evaluate the paired data based on multiple preset evaluation dimensions and generate multi-dimensional evaluation results. The display unit is used to display the functional entity evaluation report based on the multi-dimensional evaluation results.

9. The apparatus according to claim 8, characterized in that, The evaluation unit evaluates the paired data based on multiple preset evaluation dimensions, generating multi-dimensional evaluation results, including: The paired data is filtered based on preset evaluation dimensions to obtain filtered paired data corresponding to each evaluation dimension. The inference results in the filtered paired data corresponding to each evaluation dimension are matched with the true data to generate data comparison results for each evaluation dimension. The multi-dimensional evaluation results are determined based on the data comparison results of the multiple evaluation dimensions.

10. The apparatus according to claim 8, characterized in that, The display unit displays a functional entity evaluation report based on the multi-dimensional evaluation results, including: Determine the description tags for the multi-dimensional evaluation results; the description tags include at least one of the following: evaluation dimension tags determined based on the evaluation dimensions corresponding to the multi-dimensional evaluation results, context tags determined based on the environmental data corresponding to the paired data, and functional entity tags determined based on the entity type of the functional entity to which the inference result belongs. Based on the description tags, a functional entity evaluation report categorized by tag is displayed.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the method according to any one of claims 1-7.

12. A non-transitory computer-readable storage medium, wherein instructions in the storage medium are loaded by a processor and execute the method of any one of claims 1-7.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images