Intelligent network connection sensing system evaluation method, device and equipment and storage medium

Abstract

The invention discloses an intelligent network connection sensing system evaluation method and device, equipment and a storage medium. The method comprises the following steps: acquiring performance indexes of a sensing system in multiple dimensions; according to the type of the performance index, performing performance scoring by adopting a corresponding index evaluation method; according to the weight distributed by each index and the performance score, obtaining a sensing system reliability comprehensive score of the single scene; and evaluating the overall reliability of the intelligent networking sensing system based on the comprehensive score. According to the method, the function reliability of the sensing system can be comprehensively and objectively evaluated, and a scientific basis is provided for system design, test and optimization, so that the overall safety and reliability of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent connected vehicle evaluation technology, and in particular to an evaluation method, device, equipment and storage medium for intelligent connected perception systems. Background Technology

[0002] In the field of intelligent connected vehicles, the reliability of the perception system directly affects the vehicle's driving safety and functional implementation. Perception systems typically consist of multiple sensors, including cameras, millimeter-wave radar, ultrasonic radar, lidar, and satellite positioning units. These sensors are responsible for detecting targets, locating the vehicle's position, and providing environmental information to support the implementation of Advanced Driver Assistance Systems (ADAS) and autonomous driving functions. However, current methods for evaluating the functionality and performance of perception systems suffer from the following problems: 1. Fragmented evaluation methods: Existing evaluation methods typically score individual sensors or single functions independently, lacking a comprehensive evaluation of the collaborative work of multiple sensors. For example, the evaluation of performance indicators such as target detection, positioning accuracy, and speed estimation is conducted separately, making it difficult for the evaluation results to fully reflect the overall reliability of the perception system. 2. High subjectivity: Traditional evaluation methods rely on manual scoring or subjective judgment, which is easily influenced by the evaluator's subjective preferences or uneven weighting of different scenarios. This subjectivity may lead to deviations between the evaluation results and actual performance, affecting the objectivity and consistency of the evaluation. 3. Lack of adaptability to dynamic scenarios: Intelligent connected vehicles face complex and ever-changing dynamic scenarios in actual driving, while existing evaluation methods are mostly based on static or single-scenario designs, making it difficult to adapt to the perception performance evaluation of dynamic targets. For example, there is a lack of effective dynamic evaluation methods for the perception error of target motion speed or the detection range of lateral displacement.

[0003] Therefore, how to comprehensively evaluate the reliability of perception systems and provide a reliable basis for the design, testing and optimization of intelligent connected vehicles is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] The main objective of this invention is to provide an evaluation method, device, equipment, and storage medium for intelligent connected sensing systems, which can comprehensively and objectively evaluate the functional reliability of sensing systems, provide a scientific basis for system design, testing, and optimization, and thus improve the overall security and reliability of the system.

[0005] Firstly, this application provides an evaluation method for an intelligent connected sensing system, wherein the method includes the following steps: Obtain performance metrics from multiple dimensions of the perception system; Performance scores are calculated using the corresponding evaluation method based on the type of the performance indicator. Based on the weight assigned to each indicator and the performance score, a comprehensive reliability score for the perception system in a single scenario is obtained. The overall reliability of the intelligent connected sensing system is evaluated based on the comprehensive score.

[0006] In conjunction with the first aspect mentioned above, as an optional implementation method, the performance indicators are classified into: functional data, deterministic data, non-periodic time series data, and random data. If the data is determined to be functional data, the 0-1 method is used for performance scoring. The 0-1 method is as follows: if the target or vehicle location information is identified, a value of 1 is assigned, otherwise a value of 0 is assigned. If the data is determined to be deterministic, the relative error method is used to assess its reliability. If the data is determined to be non-periodic time series, its reliability is assessed using the root mean square error method. If the data is determined to be random, the JS divergence method is used to assess its reliability.

[0007] In conjunction with the first aspect mentioned above, as an optional implementation method, according to the formula: Calculate the reliability of the detection distance data in the deterministic data, where, The target distance prediction value. This represents the true distance to the target. According to the formula: Calculate the reliability of positioning accuracy data in deterministic data, where, For the positioning error at each moment, To determine the minimum positioning error in the test, This represents the maximum positioning error tested. According to the formula: To calculate the reliability of aperiodic time series data, where, For the first The true value of the velocity of the target. No. Predicted velocity values ​​of individual targets; According to the formula: To calculate the reliability of random data, where, , for , for , This refers to the camera's nominal sensing range. This refers to the actual sensing range of the camera. For and This refers to intermediate process quantities.

[0008] In conjunction with the first aspect mentioned above, as an optional implementation, the functional data includes: target recognition capability and vehicle positioning data; The deterministic data includes: displacement and positioning accuracy data; The non-periodic time series data includes: velocity and acceleration data; The randomness data includes: image data captured by the camera.

[0009] In conjunction with the first aspect mentioned above, as an optional implementation method, the analytic hierarchy process (AHP) is used to determine the weights among the various indicators. ; Based on computational performance scoring, a set of reliability evaluations for a single scenario is obtained. ,That The total number of indicators; Using the linear weighting method, we obtain the first... Comprehensive reliability score of the perception system in various scenarios ,in, ; right The reliability of the perception system in each scenario was evaluated multiple times, and the average value was taken as the comprehensive score.

[0010] In conjunction with the first aspect mentioned above, as an optional implementation method, a comprehensive reliability score for the perception system in a single scenario is obtained, resulting in a set of reliability scores for multiple scenarios. ; The mean of the distribution is obtained by statistically distributing the individual scenarios in the reliability score set using a normal distribution. and standard deviation ; Based on the mean and standard deviation To formulate a comprehensive evaluation and grading system for the reliability of sensing systems, so as to evaluate the overall reliability of intelligent connected sensing systems.

[0011] In conjunction with the first aspect mentioned above, as an optional implementation method, performance indicators in multiple dimensions are determined based on the components included in the perception system. The sensing system includes components such as millimeter-wave radar, ultrasonic radar, camera, lidar, and satellite positioning. The performance metrics across these multiple dimensions include: target recognition capability, accuracy, and motion state; The target recognition capability includes: detection rate, false detection rate, and target classification accuracy. The accuracy includes: target recognition error, relative distance, self-calibration size, and self-positioning error; The target motion state includes: velocity accuracy, acceleration accuracy, and trajectory prediction accuracy.

[0012] Secondly, this application provides an evaluation device for an intelligent connected sensing system, the device comprising: The acquisition module is used to acquire performance metrics of the perception system across multiple dimensions. An execution module is used to perform performance scoring based on the type of the performance indicator and the corresponding indicator evaluation method. The processing module is used to obtain a comprehensive reliability score of the perception system for a single scenario based on the weight assigned to each indicator and the performance score. The evaluation module is used to evaluate the overall reliability of the intelligent connected sensing system based on the comprehensive score.

[0013] Thirdly, this application also provides an electronic device, the electronic device comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described in any one of the first aspects.

[0014] Fourthly, this application also provides a computer-readable storage medium storing computer program instructions that, when executed by a computer, cause the computer to perform the method described in any of the first aspects.

[0015] This application provides an evaluation method, apparatus, device, and storage medium for an intelligent connected sensing system. The method includes the following steps: acquiring performance indicators of the sensing system across multiple dimensions; performing performance scoring using corresponding indicator evaluation methods based on the type of the performance indicators; obtaining a comprehensive reliability score for the sensing system in a single scenario based on the weights assigned to each indicator and the performance scores; and evaluating the overall reliability of the intelligent connected sensing system based on the comprehensive score. This application can comprehensively and objectively evaluate the functional reliability of the sensing system, providing a scientific basis for system design, testing, and optimization, thereby improving the overall security and reliability of the system.

[0016] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] Figure 1 This is a flowchart of an evaluation method for an intelligent connected sensing system provided in the embodiments of this application; Figure 2 This is a schematic diagram of an evaluation device for an intelligent connected sensing system provided in an embodiment of this application; Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of this application; Figure 4This is a schematic diagram of a computer-readable program medium provided in an embodiment of this application. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0020] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings represent functional entities and do not necessarily correspond to physically or logically independent entities.

[0021] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0022] Reference Figure 1 , Figure 1 The diagram shown is a flowchart of an evaluation method for an intelligent connected sensing system provided by the present invention. Figure 1 As shown, the method includes the following steps: Step S101: Obtain performance indicators of the perception system in multiple dimensions.

[0023] Specifically, performance indicators in multiple dimensions are determined based on the components included in the perception system; The sensing system includes components such as millimeter-wave radar, ultrasonic radar, camera, lidar, and satellite positioning. The performance metrics across these multiple dimensions include: target recognition capability, accuracy, and motion state; The target recognition capability includes: detection rate, false detection rate, and target classification accuracy. The accuracy includes: target recognition error, relative distance, self-calibration size, and self-positioning error; The target motion state includes: velocity accuracy, acceleration accuracy, and trajectory prediction accuracy.

[0024] To illustrate this with examples, a perception system includes: millimeter-wave radar, ultrasonic radar, camera, lidar, and satellite positioning. Each component has corresponding metrics. For example, the target recognition capability (detection rate, false detection rate, target classification accuracy, etc.) is used to evaluate whether the system can accurately identify targets and classify them. These are the corresponding metrics for millimeter-wave radar, camera, and lidar. The accuracy of target detection includes: target recognition error, relative distance, self-calibrated size, and self-positioning error. These are the corresponding indicators for millimeter-wave radar, ultrasonic radar, camera, lidar, and satellite positioning.

[0025] For the target's motion state, the accuracy includes: velocity accuracy, acceleration accuracy, and trajectory prediction accuracy. These are the metrics for millimeter-wave radar, cameras, and lidar.

[0026] In summary, it can be understood that obtaining multiple dimensional indicators means obtaining multiple indicators for each component.

[0027] In one embodiment, the reliability index is evaluated by assigning a value of 0 or 1 to the system's functionality. If the result is 1, the reliability of the sensing system is further quantified. Actual values ​​for each index are calculated through experimental data collection. For example, the target detection rate is calculated as the ratio of the number of detected targets to the actual number of targets; the false detection rate is calculated as the ratio of the number of falsely detected targets to the total number of detected targets; the target 2D positioning accuracy is calculated as the overlap between the system-estimated target bounding box and the actual bounding box; and the target velocity estimation accuracy is calculated as the deviation between the system-estimated velocity and the actual velocity. Since different indices have different dimensions and value ranges, they need to be normalized to unify their value range between [0,1] for easier subsequent comprehensive calculations. Normalization typically uses a linear transformation method to map the actual values ​​to the [0,1] interval.

[0028] Step S102: Based on the type of the performance indicator, use the corresponding indicator evaluation method to score the performance.

[0029] Specifically, the performance indicators are classified as follows: functional data, deterministic data, non-periodic time series data, and stochastic data; If the data is determined to be functional data, the 0-1 method is used for performance scoring. The 0-1 method is as follows: if the target or vehicle location information is identified, a value of 1 is assigned, otherwise a value of 0 is assigned. If the data is determined to be deterministic, the relative error method is used to assess its reliability. If the data is determined to be non-periodic time series, its reliability is assessed using the root mean square error method. If the data is determined to be random, the JS divergence method is used to assess its reliability.

[0030] For ease of understanding and illustration, performance indicators are categorized as follows: presence or absence of function (functional data), deterministic data, non-periodic time series data, and stochastic data.

[0031] The functional data is scored using a 0-1 method, such as whether a target is identified or located: millimeter-wave radar (1 for identification, 0 for non-identification); ultrasonic radar (1 for identification, 0 for non-identification); lidar (1 for identification, 0 for non-identification); satellite positioning unit (1 for positioning, 0 for non-positioning). Deterministic data: Using the relative error method, such as millimeter-wave radar, ultrasonic radar, and lidar, to detect the relative distance to the target. The sensing system outputs a predicted target distance. The true distance to the target is obtained through a true value measurement system. Reliability is calculated by combining the principle of relative error method. , .

[0032] According to the formula: Calculate the reliability of positioning accuracy data in deterministic data, where, For the positioning error at each moment, To determine the minimum positioning error in the test, This represents the maximum positioning error tested.

[0033] Non-periodic time series data: Root mean square error method: For example, in the perception assessment of target motion velocity, the predicted sequence data of target motion velocity is defined as follows: The true data of the motion sequence is , Represented as a time step, This represents the target's velocity. Using the root mean square error method, the reliability can be obtained. , To calculate the reliability of aperiodic time series data, where, For the first The true value of the velocity of the target. No. Predicted velocity values ​​for each target.

[0034] Random Data: JS Divergence Method: For example, in a scenario where there is lateral relative displacement between the target and the test vehicle, the effective horizontal sensing range of the camera is evaluated. The effective sensing range of the camera is defined as follows: during the process of the target moving from outside the sensing range into the sensing range, the corresponding position in the first frame where the target information can be correctly perceived is taken as one of the boundaries of the effective sensing range. During the process of the target moving from inside the sensing range into outside the sensing range, the corresponding position in the last frame where the target information can be correctly perceived is taken as another boundary of the effective sensing range. Assume the nominal horizontal sensing range of the camera is... The actual effective sensing range of the camera in the sensing system is For ease of calculation, let's assume... ,make , The reliability can then be calculated using the formula. The formula for calculating JS divergence is as follows: , for , for , This refers to the camera's nominal sensing range. This refers to the actual sensing range of the camera. For and For intermediate process quantities, .

[0035] It's worth noting that the effective sensing range of a perception system includes different aspects such as horizontal and vertical directions. Different test scenarios may involve calculating the sensing range in different directions. If the scenario only involves calculating the effective sensing range in one direction, then the JS divergence test value in that direction is used as the sole evaluation metric. If it involves calculating the effective sensing range in multiple directions, then a linear weighted method is further employed, combining the test results from all directions to calculate the evaluation information for the effective sensing range.

[0036] It should also be noted that the functional data includes: target recognition capability and vehicle positioning data; The deterministic data includes: displacement and positioning accuracy data; The non-periodic time series data includes: velocity and acceleration data; The randomness data includes: image data captured by the camera.

[0037] Step S103: Based on the weight assigned to each indicator and the performance score, obtain the comprehensive reliability score of the perception system for a single scenario.

[0038] Specifically, the weights among the various indicators are determined using the analytic hierarchy process (AHP). ; Based on computational performance scoring, a set of reliability evaluations for a single scenario is obtained. ,That The total number of indicators; Using the linear weighting method, we obtain the first... Comprehensive reliability score of the perception system in various scenarios ,in, ; right The reliability of the perception system in each scenario was evaluated multiple times, and the average value was taken as the comprehensive score.

[0039] For example, given N scenarios, each with multiple metrics, we evaluate these metrics and assign corresponding weights based on their importance in practical applications and their impact on the overall system performance (e.g., both millimeter-wave radar and cameras measure target distance, but millimeter-wave radar excels at ranging while cameras excel at target recognition; therefore, millimeter-wave radar has a higher weight for ranging, indicating higher reliability, while cameras have a higher weight for target recognition). This ensures the scientific validity and rationality of the evaluation results.

[0040] After obtaining a single-scenario reliability evaluation set ,That The total number of indicators and the corresponding set of weights. Using the linear weighting method, we obtain the first... Comprehensive reliability score of the perception system in various scenarios Six trials were conducted in the same scenario, and the final reliability score of the perception system in that scenario was the average of the scores from the six trials. This score reflects the overall perception performance level of the system in that scenario (i.e., its ability to perceive objects in that environment). The reliability of the perception system in each scenario is evaluated multiple times, and the average value is taken as the final comprehensive score for that single scenario.

[0041] Step S104: Evaluate the overall reliability of the intelligent connected sensing system based on the comprehensive score.

[0042] Specifically, the overall reliability score of the perception system in a single scenario is used to obtain a set of reliability scores for multiple scenarios. ; The mean of the distribution is obtained by statistically distributing the individual scenarios in the reliability score set using a normal distribution. and standard deviation ; Based on the mean and standard deviation To formulate a comprehensive evaluation and grading system for the reliability of sensing systems, so as to evaluate the overall reliability of intelligent connected sensing systems.

[0043] To facilitate understanding, examples are provided to illustrate the reliability of calculations for a single scenario. Perform a statistical distribution, assuming a normal distribution, and calculate the mean of the distribution. and standard deviation By comprehensively considering the mean and standard deviation To formulate a comprehensive evaluation and grading system for perceived reliability (this is equivalent to weakening the weight of each scenario, or considering them to be equally weighted).

[0044] In the functional reliability evaluation of the AEB system, the reliability calculated for a single scenario is... This can be comprehensively analyzed using statistical distribution methods. Assume... Follows a normal distribution ,in The mean value represents the central tendency of reliability across all scenarios, reflecting the average level of overall reliability. The standard deviation represents the dispersion of reliability, reflecting the differences between different scenarios. By comprehensively considering the mean... and standard deviation This approach allows for the development of a comprehensive evaluation grading system for perceived credibility. Essentially, this method weakens the weighting differences between various scenarios, or can be considered as employing an equal-weighting approach, thus avoiding the bias caused by subjective weighting in traditional methods. Compared to traditional weighted single-dimensional indicators, the comprehensive evaluation method based on statistical distribution has significant advantages: 1) Objectivity: It directly reflects the distribution characteristics of the data through the mean and standard deviation, reducing the subjective influence of human weighting. 2) Comprehensiveness: It not only considers the average level of reliability but also reflects the differences between scenarios, resulting in a more comprehensive evaluation. 3) Flexibility: The grading rules can be adjusted according to actual needs (such as dividing into more levels) to adapt to different application scenarios.

[0045] The grading rules are as follows: High reliability level A: uA ≤ u ≤ 1 and 0 ≤ s ≤ u - uA; Higher reliability level B: uB ≤ u < 1 and u - uA

[0046] Reference Figure 2 , Figure 2 The diagram shown is a schematic of an evaluation device for an intelligent connected sensing system provided by the present invention. Figure 2 As shown, the device includes: Acquisition module 201: It is used to acquire performance indicators of the perception system in multiple dimensions.

[0047] Execution module 202: It is used to perform performance scoring according to the type of the performance index and the corresponding index evaluation method.

[0048] Processing module 203: It is used to obtain a comprehensive reliability score of the perception system for a single scene based on the weight assigned to each indicator and the performance score.

[0049] Evaluation module 204: It is used to evaluate the overall reliability of the intelligent connected sensing system based on the comprehensive score.

[0050] ​Furthermore, in one possible implementation, the execution module is also used to classify the performance indicators, which include: functional data, deterministic data, non-periodic time series data, and random data; If the data is determined to be functional data, the 0-1 method is used for performance scoring. The 0-1 method is as follows: if the target or vehicle location information is identified, a value of 1 is assigned, otherwise a value of 0 is assigned. If the data is determined to be deterministic, the relative error method is used to assess its reliability. If the data is determined to be non-periodic time series, its reliability is assessed using the root mean square error method. If the data is determined to be random, the JS divergence method is used to assess its reliability.

[0051] Furthermore, in one possible implementation, the processing module is also configured to process according to the formula: Calculate the reliability of the detection distance data in the deterministic data, where, The target distance prediction value. This represents the true distance to the target. According to the formula: Calculate the reliability of positioning accuracy data in deterministic data, where, For the positioning error at each moment, To determine the minimum positioning error in the test, This represents the maximum positioning error tested. According to the formula: To calculate the reliability of aperiodic time series data, where, For the first The true value of the velocity of the target. No. Predicted velocity values ​​of individual targets; According to the formula: To calculate the reliability of random data, where, , for , for , This refers to the camera's nominal sensing range. This refers to the actual sensing range of the camera. For and This refers to intermediate process quantities.

[0052] Furthermore, in one possible implementation, the processing module is also used to acquire the functional data, including: target recognition capability and vehicle positioning data; The deterministic data includes: displacement and positioning accuracy data; The non-periodic time series data includes: velocity and acceleration data; The randomness data includes: image data captured by the camera.

[0053] Furthermore, in one possible implementation, the processing module is also used to determine the weights among the various indicators using the analytic hierarchy process (AHP). ; Based on computational performance scoring, a set of reliability evaluations for a single scenario is obtained. ,That The total number of indicators; Using the linear weighting method, we obtain the first... Comprehensive reliability score of the perception system in various scenarios ,in, ; right The reliability of the perception system in each scenario was evaluated multiple times, and the average value was taken as the comprehensive score.

[0054] Furthermore, in one possible implementation, the evaluation module is also used to comprehensively score the reliability of the perception system in a single scenario, thereby obtaining a set of reliability scores for multiple scenarios. ; The mean of the distribution is obtained by statistically distributing the individual scenarios in the reliability score set using a normal distribution. and standard deviation ; Based on the mean and standard deviation To formulate a comprehensive evaluation and grading system for the reliability of sensing systems, so as to evaluate the overall reliability of intelligent connected sensing systems.

[0055] Furthermore, in one possible implementation, the acquisition module is also used to determine performance indicators in multiple dimensions based on the components included in the sensing system. The sensing system includes components such as millimeter-wave radar, ultrasonic radar, camera, lidar, and satellite positioning. The performance metrics across these multiple dimensions include: target recognition capability, accuracy, and motion state; The target recognition capability includes: detection rate, false detection rate, and target classification accuracy. The accuracy includes: target recognition error, relative distance, self-calibration size, and self-positioning error; The target motion state includes: velocity accuracy, acceleration accuracy, and trajectory prediction accuracy.

[0056] The following reference Figure 3 To describe an electronic device 300 according to this embodiment of the present invention. Figure 3The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0057] like Figure 3 As shown, the electronic device 300 is presented in the form of a general-purpose computing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, and a bus 330 connecting different system components (including storage unit 320 and processing unit 310).

[0058] The storage unit stores program code that can be executed by the processing unit 310, causing the processing unit 310 to perform the steps described in the "Embodiment Methods" section of this specification according to various exemplary embodiments of the present invention.

[0059] Storage unit 320 may include readable media in the form of volatile storage units, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0060] Storage unit 320 may also include a program / utility 324 having a set (at least one) of program modules 325, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0061] Bus 330 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0062] Electronic device 300 can also communicate with one or more external devices (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 300, and / or any device that enables electronic device 300 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 350. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 360. As shown, network adapter 360 communicates with other modules of electronic device 300 via bus 330. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0063] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0064] According to the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention can also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0065] refer to Figure 4 As shown, a program product 400 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0066] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0067] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0068] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0069] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0070] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0071] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

[0072] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

Claims

1. An evaluation method for an intelligent connected sensing system, characterized in that, include: Obtain performance metrics from multiple dimensions of the perception system; Performance scores are calculated using the corresponding evaluation method based on the type of the performance indicator. Based on the weight assigned to each indicator and the performance score, a comprehensive reliability score for the perception system in a single scenario is obtained. The overall reliability of the intelligent connected sensing system is evaluated based on the comprehensive score.

2. The method according to claim 1, characterized in that, The step of scoring performance based on the type of performance indicator using the corresponding indicator evaluation method includes: The performance metrics are categorized as follows: functional data, deterministic data, non-periodic time series data, and stochastic data. If the data is determined to be functional data, the 0-1 method is used for performance scoring. The 0-1 method is as follows: if the target or vehicle location information is identified, a value of 1 is assigned, otherwise a value of 0 is assigned. If the data is determined to be deterministic, the relative error method is used to assess its reliability. If the data is determined to be non-periodic time series, its reliability is assessed using the root mean square error method. If the data is determined to be random, the JS divergence method is used to assess its reliability.

3. The method according to claim 2, characterized in that, include: According to the formula: Calculate the reliability of the detection distance data in the deterministic data, where, The target distance prediction value. This represents the true distance to the target. According to the formula: Calculate the reliability of positioning accuracy data in deterministic data, where, For the positioning error at each moment, To determine the minimum positioning error in the test, This represents the maximum positioning error tested. According to the formula: To calculate the reliability of aperiodic time series data, where, For the first The true value of the velocity of the moving target. No. Predicted velocity values ​​of individual targets; According to the formula: To calculate the reliability of random data, where, , for , for , This refers to the camera's nominal sensing range. This refers to the actual sensing range of the camera. For and This refers to intermediate process quantities.

4. The method according to claim 2, characterized in that, include: The functional data includes: target recognition capability and vehicle positioning data; The deterministic data includes: displacement and positioning accuracy data; The non-periodic time series data includes: velocity and acceleration data; The randomness data includes: image data captured by the camera.

5. The method according to claim 1, characterized in that, The comprehensive reliability score of the perception system for a single scenario is obtained by combining the weights assigned to each indicator with the performance score, including: Use the analytic hierarchy process (AHP) to determine the weights among the various indicators. ; Based on computational performance scoring, a set of reliability evaluations for a single scenario is obtained. ,That The total number of indicators; Using the linear weighting method, we obtain the first... Comprehensive reliability score of the perception system in various scenarios ,in, ; right The reliability of the perception system in each scenario was evaluated multiple times, and the average value was taken as the comprehensive score.

6. The method according to claim 1, characterized in that, The evaluation of the overall reliability of the intelligent connected sensing system based on the comprehensive score includes: A comprehensive reliability score for the perception system in a single scenario is used to obtain a set of reliability scores for multiple scenarios. ; The mean of the distribution is obtained by statistically distributing the individual scenarios in the reliability score set using a normal distribution. and standard deviation ; Based on the mean and standard deviation To formulate a comprehensive evaluation and grading system for the reliability of sensing systems, so as to evaluate the overall reliability of intelligent connected sensing systems.

7. The method according to claim 1, characterized in that, The performance metrics of the acquisition perception system across multiple dimensions include: Based on the components included in the perception system, determine performance indicators in multiple dimensions; The sensing system includes components such as millimeter-wave radar, ultrasonic radar, camera, lidar, and satellite positioning. The performance metrics across these multiple dimensions include: target recognition capability, accuracy, and motion state; The target recognition capability includes: detection rate, false detection rate, and target classification accuracy. The accuracy includes: target recognition error, relative distance, self-calibration size, and self-positioning error; The target motion state includes: velocity accuracy, acceleration accuracy, and trajectory prediction accuracy.

8. An evaluation device for an intelligent connected sensing system, characterized in that, include: The acquisition module is used to acquire performance metrics of the perception system across multiple dimensions. An execution module is used to perform performance scoring based on the type of the performance indicator and the corresponding indicator evaluation method. The processing module is used to obtain a comprehensive reliability score of the perception system for a single scenario based on the weight assigned to each indicator and the performance score. The evaluation module is used to evaluate the overall reliability of the intelligent connected sensing system based on the comprehensive score.

9. An electronic device, characterized in that, The electronic device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores computer program instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7.

Images