Automatic driving perception system redundancy checking method, system and device

Abstract

The application discloses an automatic driving perception system redundancy checking method, system and device. The method comprises the following steps: acquiring multi-modal data from multiple sensor data sources; aligning the multi-modal data based on a pre-established vehicle coordinate system; performing front fusion and back fusion on the aligned multi-modal data in parallel, wherein the front fusion comprises performing front data fusion processing on the aligned multi-modal data to obtain sensor data after data fusion; the back fusion comprises performing back target-level fusion on target objects detected from the aligned multi-modal data to obtain target object fused sensor data; and performing data checking and merging processing based on the sensor data after data fusion and the target object fused sensor data to obtain sensor perception data. The front fusion and the back fusion can be performed in parallel, redundancy checking of multi-modal data fusion is realized, and the accuracy of environment perception and the safety of automatic driving are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a method, system, and device for redundancy verification of an autonomous driving perception system. Background Technology

[0002] Intelligent vehicles have become a strategic direction for the global automotive industry, and autonomous driving technology is the core technology for their development. Autonomous driving technology includes functional modules such as perception algorithms, decision-making and planning algorithms, control algorithms, and localization algorithms. Among these, environmental perception is the primary element of autonomous driving, serving as the link between the vehicle and its environment and the foundation for other algorithm modules. The overall performance of an autonomous driving system largely depends on the quality of its perception system. The functions of the perception system include, but are not limited to, target recognition, 2D / 3D target detection, semantic segmentation, target tracking, and free space segmentation, all of which rely on raw data collected from the environment by sensors installed on the vehicle.

[0003] Current autonomous driving technologies utilize both LiDAR and camera-captured point cloud and image data for perception tasks, enabling foreground extraction. However, this single-modal data perception approach has limitations. For instance, camera-captured data is primarily located at a low position in front of the viewpoint, and objects may be occluded in complex scenes, posing significant challenges to target detection and semantic segmentation. Furthermore, LiDAR's resolution varies at different distances due to mechanical limitations, and it is susceptible to extreme weather conditions such as heavy fog and rain. These single-modal data perception deficiencies reduce the accuracy of environmental perception during autonomous driving, making it difficult to accurately perceive environmental information, target objects, and predict hazards, ultimately impacting the safety of autonomous driving. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a method, system and apparatus for redundancy verification of an autonomous driving perception system that overcomes or at least partially solves the above problems.

[0005] This invention provides a method for redundancy verification in an autonomous driving perception system, comprising:

[0006] Acquire multimodal data from multiple sensor data sources;

[0007] The multimodal data is aligned based on a pre-established vehicle coordinate system.

[0008] The aligned multimodal data is subjected to parallel pre-fusion and post-fusion. The pre-fusion includes performing preliminary data fusion processing on the aligned multimodal data to obtain fused sensor data. The post-fusion includes performing post-target-level fusion on the target objects detected from the aligned multimodal data to obtain fused sensor data of the target objects.

[0009] The sensor data is obtained by performing data verification and merging processing on the sensor data fused from the data fused with the target object.

[0010] In some optional embodiments, aligning the multimodal data based on a pre-established vehicle coordinate system includes:

[0011] A vehicle coordinate system is established with the midpoint of the axis between the two rear wheels of the vehicle as the origin, and the axis and the line perpendicular to the axis as the horizontal and vertical coordinate axes.

[0012] The data from different sensor data sources are transformed into the vehicle coordinate system to achieve the alignment of the multimodal data.

[0013] In some optional embodiments, the aligned multimodal data undergoes preliminary data fusion processing to obtain fused sensor data, including:

[0014] The image data acquired by the image sensor and the point cloud data acquired by the point cloud device are processed for temporal and spatial synchronization.

[0015] Identify two-dimensional target objects from the image data and determine the boundaries of the two-dimensional target objects;

[0016] Based on the boundary of the two-dimensional target object, the image data of the identified two-dimensional target object is combined into the point cloud data to obtain the sensor data after data fusion.

[0017] In some optional embodiments, target objects detected from the aligned multimodal data are subjected to post-target-level fusion to obtain sensor data of the fused target objects, including:

[0018] The image data acquired by the image sensor and the point cloud data acquired by the point cloud device are processed for temporal and spatial synchronization.

[0019] Identify two-dimensional target objects from the image data and determine the boundaries of the two-dimensional target objects;

[0020] The three-dimensional target object is identified from the point cloud data, and the boundary of the three-dimensional target object is determined;

[0021] Based on the boundaries of the two-dimensional target objects and the three-dimensional target objects, the two-dimensional target objects identified in the image data and the three-dimensional target objects identified in the point cloud data are merged to obtain the sensor data of target object fusion.

[0022] In some optional embodiments, data verification and merging processing are performed based on the fused sensor data and the fused sensor data of the target object to obtain sensor perception data, including:

[0023] Perform integrity verification on the sensor data after data fusion and the sensor data fused with the target object;

[0024] After the test is passed, the sensor data after data fusion and the sensor data fused with the target object are fused together, and the detected target objects are merged to obtain the total fused sensor perception data.

[0025] In some optional embodiments, at least one of the following security monitoring mechanisms is also included:

[0026] Monitor data for anomalies and issue early warnings upon detecting abnormal data;

[0027] Monitor the time consumed by each step of data alignment, pre-fusion, post-fusion, and data merging, and display the time consumption information.

[0028] Monitor processor resource consumption and issue an early warning when resource consumption exceeds a preset resource threshold;

[0029] Monitor the operation status of pre-fusion and post-fusion, and issue an early warning when one of them is abnormal;

[0030] The accuracy of the fusion result is determined by the overlap between the fused sensor data and the fused sensor data of the target object. If the result is inaccurate, an early warning is issued.

[0031] In some optional embodiments, anomaly monitoring of the monitoring data includes:

[0032] The system monitors image data acquired by image sensors and determines whether there are any anomalies in the image data based on the pixel values ​​of the pixels in the image. If the proportion of pixels with the same neighboring pixel value in a frame of an image is greater than a set pixel value threshold, or the number of frames in which some pixels with the same pixel value persists in the image is greater than a set frame number threshold, or the duration of the persistence of some pixels with the same pixel value in the image is greater than a set duration threshold, the image data is considered to be abnormal. Image anomalies include at least one of the following: green screen, black screen, light spots, dropped frames, delay, distortion, occlusion, and dirt in the image.

[0033] CRC verification is performed on the point cloud data acquired by the point cloud device. Based on the verification result, it is determined whether there are any anomalies in the point cloud data. Point cloud data anomalies include at least one of the following: empty data, missing fields, missing strength information, and missing dimension information.

[0034] In some optional embodiments, the time consumed by the data alignment, pre-fusion, post-fusion, and data merging steps is monitored, including:

[0035] In each functional module that performs data alignment, front fusion, back fusion, and data merging steps, an input timestamp is added to the input data at the input end, and an output timestamp is added to the output data at the output end.

[0036] The processing time of each functional module is calculated based on the input timestamp and the output timestamp to obtain the time consumed by each step.

[0037] In some optional embodiments, the accuracy of the fusion result is determined based on the overlap between the fused sensor data and the fused sensor data of the target object, including:

[0038] The intersection and union of the target objects in the sensor data after data fusion and the sensor data after target object fusion are processed. The intersection-union ratio is obtained based on the degree of overlap and matching of the target object bounding boxes in the data after intersection processing and the data after union processing.

[0039] The accuracy of the fusion result of the sensor data after data fusion and the sensor data fused with the target object is determined based on the relationship between the cross-union ratio and the set overlap rate threshold.

[0040] This invention provides a redundancy verification device for an autonomous driving perception system, comprising:

[0041] The data acquisition module is used to acquire multimodal data from multiple sensor data sources;

[0042] The data alignment module is used to align the multimodal data based on a pre-established vehicle coordinate system.

[0043] The system includes a pre-fusion module and a post-fusion module that operate in parallel. The pre-fusion module is used to perform preliminary data fusion processing on the aligned multimodal data to obtain fused sensor data. The post-fusion module is used to perform post-target-level fusion on the target objects detected from the aligned multimodal data to obtain fused sensor data of the target objects.

[0044] The overall fusion module is used to perform data verification and merging processing on the sensor data after data fusion and the sensor data fused from the target object to obtain sensor perception data.

[0045] In some optional embodiments, the above-described apparatus further includes at least one of the following monitors:

[0046] The data monitor is used to detect anomalies in the monitoring data and issue early warnings when abnormal data is detected.

[0047] The time monitor is used to monitor the time consumed by the data alignment, pre-fusion, post-fusion, and data merging steps respectively, and display the time consumption information.

[0048] The resource monitor is used to monitor the processor's resource consumption and issue an early warning when the resource consumption exceeds a preset resource threshold.

[0049] The results monitor is used to monitor the operation status of pre-fusion and post-fusion, and issue an early warning when one of them is abnormal; it also determines whether the fusion result is accurate based on the overlap between the sensor data after data fusion and the sensor data fused with the target object, and issues an early warning if it is inaccurate.

[0050] This invention provides an autonomous driving perception system redundancy verification system, comprising: at least two different types of sensors and the aforementioned autonomous driving perception system redundancy verification device.

[0051] This invention provides a computer storage medium storing computer-executable instructions, which, when executed by a processor, implement the aforementioned redundancy verification method for an autonomous driving perception system.

[0052] This invention provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described method for redundancy verification of an autonomous driving perception system.

[0053] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0054] The redundancy verification method for an autonomous driving perception system provided in this invention aligns multimodal data from multiple sensor data sources and then performs pre-fusion and post-fusion in parallel. In the pre-fusion stage, preliminary data fusion processing is performed on the multimodal data to obtain fused sensor data. In the post-fusion stage, target objects detected from the multimodal data undergo post-target-level fusion to obtain fused sensor data of the target object. By executing pre-fusion and post-fusion in parallel, even if a problem occurs in one stage, multimodal data fusion can still be effectively performed, achieving redundancy verification of multimodal data, overcoming the deficiencies of single-modal data, improving the accuracy and reliability of environmental perception during autonomous driving, thereby accurately predicting hazardous factors in the environment and improving the safety of autonomous driving. Further data verification and merging processing is performed on the fused sensor data and the fused sensor data of the target object. The mutual verification and merging of fused data from the pre-fusion and post-fusion stages can better avoid perception errors, accurately identify target objects, and further improve the accuracy of environmental perception and the safety of autonomous driving.

[0055] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0056] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0057] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0058] Figure 1 This is a flowchart of the redundancy verification method for the autonomous driving perception system in Embodiment 1 of the present invention;

[0059] Figure 2 This is a block diagram illustrating the redundancy verification principle of the autonomous driving perception system in Embodiment 1 of the present invention.

[0060] Figure 3 This is an example diagram of the vehicle coordinate system in Embodiment 1 of the present invention;

[0061] Figure 4 This is a flowchart of the pre-fusion process in Embodiment 1 of the present invention;

[0062] Figure 5 This is a flowchart of the post-fusion process in Embodiment 1 of the present invention;

[0063] Figure 6 This is a schematic diagram of the redundancy verification system of the autonomous driving perception system in an embodiment of the present invention;

[0064] Figure 7 This is a schematic diagram of the redundancy verification device for the autonomous driving perception system in an embodiment of the present invention. Detailed Implementation

[0065] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0066] In existing technologies, environmental perception for autonomous driving adopts a single-modal perception approach. Although single-modal perception data has certain defects, leading to a decrease in perception accuracy, data from each mode can perform well in different domains when used separately. When data from different modes are fused, better performance can be achieved. Taking the fusion perception of LiDAR and cameras as an example, the two complement each other and can produce better performance in perception tasks.

[0067] Currently, perception tasks used in autonomous driving are rapidly evolving, ranging from more advanced cross-modal feature representations and more reliable sensors across different modalities to more complex and robust deep learning models and multimodal fusion techniques. Multimodality refers to multiple data input sources, which can be information from various sensors such as cameras, LiDAR, and radar. Achieving good performance in multimodal fusion for autonomous driving systems is not easy due to noise in the raw data, underutilization of information, and misalignment of multimodal sensors.

[0068] To address the problems existing in the prior art, this invention provides a method and system for redundancy verification of an autonomous driving perception system. Facing complex traffic conditions, changing weather conditions, and various corner cases, it implements a redundancy design scheme for the autonomous driving system, which runs through all technical aspects of the entire autonomous driving process. The multimodal fusion perception module adopts a fusion of multiple sensors to cope with various possible scenarios, thereby ensuring system redundancy and safety.

[0069] Redundancy design refers to adding more than one system path to perform the same function during the system design process. This ensures that the system or equipment can still function normally when one part fails, reducing the probability of system or equipment failure and improving system reliability. In this application, the redundancy design for pre-fusion and post-fusion can adopt the form of a pipeline. A pipeline is a channel form that includes the processing flow of the basic components of the model.

[0070] A cockpit-driver integration system refers to a system that integrates and optimizes the vehicle's cockpit and powertrain. In some application scenarios, the method provided in this invention can be applied to cockpit-driver integration systems and has significant meaning and advantages within these systems. Specifically:

[0071] (1) Improved accuracy of target perception: Specifically, in this embodiment of the invention, redundancy verification of sensor data through pre-fusion and post-fusion can reduce the possibility of false alarms and missed alarms, and improve the accuracy of judging parameters such as target position, speed, and size. Furthermore, it enhances system reliability; when one sensor or sensing module in the system malfunctions or becomes abnormal, other independent sensors or modules can provide backup sensing data, ensuring that the system can still operate normally and provide accurate target perception results.

[0072] (2) The method provided in this embodiment can also provide decision support for autonomous driving systems: Specifically, accurate target perception is a key input for driving decisions and autonomous driving systems. Through the redundancy verification provided in this embodiment, more reliable target perception results can be provided, offering more accurate decision support for the driver or autonomous driving system, such as collision avoidance, lane changing, and parking. Furthermore, it enhances the safety of the frame fusion system. By accurately perceiving and verifying the target's position, speed, and behavior, potential dangerous situations can be better predicted, and corresponding safety measures, such as emergency braking or evasive maneuvers, can be taken, thereby reducing the occurrence of accidents.

[0073] (3) Increased user trust: Redundancy verification improves the accuracy of perceived targets and the reliability of the system, thereby increasing user trust in the cockpit-driver fusion system. Users will use driver assistance or autonomous driving functions with greater confidence and peace of mind, thus promoting the widespread application and acceptance of the technology.

[0074] In summary, the method provided in this embodiment has important purposes, significance and benefits in cockpit-driver fusion systems, improving perception accuracy, system reliability and driving safety, while enhancing system stability and user trust.

[0075] Example 1

[0076] Embodiment 1 of this invention provides a redundancy verification method for an autonomous driving perception system. At the software algorithm level, it proposes a redundancy verification method for perceived targets that involves parallel pre-fusion and post-fusion. This method employs a two-layer guarantee mechanism where pre-fusion and post-fusion algorithms coexist and run in parallel. Multiple algorithm models operate independently and in parallel; if any algorithm module fails, the remaining modules continue to operate normally, ensuring timely target perception. This achieves redundancy verification of perceived targets, ensuring the reliability and safety of the system at the perception level. The process is as follows: Figure 1 As shown, its implementation principle is as follows: Figure 2 As shown, it includes the following steps:

[0077] Step S101: Acquire multimodal data from multiple sensor data sources.

[0078] See Figure 2 As shown, data is acquired from different sensors, such as image sensors and point cloud devices, to obtain multimodal data from different sensor data sources.

[0079] Step S102: Align the multimodal data based on the pre-established vehicle coordinate system.

[0080] The acquired multimodal data undergoes data preprocessing. Because different sensors use different coordinate systems, coordinate system transformation can be performed to facilitate subsequent fusion, for example, all data can be transformed to the vehicle coordinate system to achieve multimodal data alignment. The alignment process includes: establishing a vehicle coordinate system with the midpoint of the axis between the two rear wheels of the vehicle as the origin, and the axis and lines perpendicular to the axis as the horizontal and vertical coordinate axes; and performing coordinate transformations on the data from different sensor data sources to transform them into the vehicle coordinate system to achieve multimodal data alignment.

[0081] In this application, cameras and LiDARs are primarily used as the main sensors in the data fusion scheme. Whether it's pre-fusion or post-fusion, spatial alignment between the camera and LiDAR is crucial. To implement the fusion algorithm, a locally unified spatial coordinate system must first be established. All sensor data uniformly adopt the vehicle coordinate system. An example of the vehicle coordinate system is shown below. Figure 3 As shown, the origin is the center point of the line connecting the axes of the two rear wheels of the vehicle. The longitudinal direction is the X-axis, the forward direction is the positive direction of the X-axis, the lateral direction is the Y-axis, the left direction is the positive direction of the Y-axis, and the direction perpendicular to the X-axis and Y-axis is the Z-axis. The spatial coordinate transformation is performed using calibration methods such as Perspective-n-Point (PNP).

[0082] Depending on the application scenario, various combinations of multiple sensors, such as ultrasonic sensors, millimeter-wave radar, cameras, and lidar, are used in the environmental perception process to achieve sensor redundancy. Since different sensors have different working principles and technical characteristics, their applicable application scenarios vary. Therefore, a multi-sensor fusion approach is adopted to address various possible scenario situations and ensure system redundancy. Multimodal data fusion from sensors can be divided into two main categories: pre-fusion and post-fusion. If only one algorithm is used at the algorithm level, a fusion failure will lead to inaccurate perception. This embodiment employs a parallel approach of pre-fusion and post-fusion, performing pre-fusion (step S103) and post-fusion (step S104) in parallel on the aligned multimodal data to ensure high stability and reliability of the perception algorithm output.

[0083] Step S103: Perform preliminary data fusion processing on the aligned multimodal data to obtain fused sensor data.

[0084] The pre-fusion process includes: performing time and space synchronization processing on the image data acquired by the image sensor and the point cloud data acquired by the point cloud device; identifying two-dimensional target objects from the image data and determining the boundaries of the two-dimensional target objects; and combining the image data of the identified two-dimensional target objects into the point cloud data based on the boundaries of the two-dimensional target objects to obtain the sensor data after data fusion.

[0085] The pre-fusion stage involves fusing raw data from multiple sensors. Data of different formats from different data sources is directly fused at the raw data level through spatial alignment, temporal synchronization, and projection, achieving data-level fusion. For example, point cloud data from a point cloud device and pixel-level image data from an image sensor can be fused. While object detection algorithms are used in this process, the actual work involves projecting the 3D point cloud onto the image and then associating it with the pixels.

[0086] See Figure 4The pre-fusion flowchart shown, taking a camera and LiDAR as sensor data sources as an example, performs time and spatial synchronization processing on the image data acquired by the camera and the point cloud data acquired by the LiDAR in a synchronizer. This detects 2D obstacles and their bounding boxes in the image data. The 3D LiDAR point cloud data is converted into a 2D image, and the feature-level representation from the camera image is fused using a 2D detection algorithm to achieve better performance. Finally, the image data is combined with the original point cloud generated by the LiDAR branch to complete the target detection task.

[0087] Step S104: Perform post-target-level fusion on the target objects detected from the aligned multimodal data to obtain the sensor data of the target object fusion.

[0088] The image data acquired by the image sensor and the point cloud data acquired by the point cloud device are processed in a time and space synchronization manner; two-dimensional target objects are identified from the image data and their boundaries are determined; three-dimensional target objects are identified from the point cloud data and their boundaries are determined; based on the boundaries of the two-dimensional and three-dimensional target objects, the two-dimensional target objects identified from the image data and the three-dimensional target objects identified from the point cloud data are merged to obtain sensor data fused with the target objects.

[0089] The post-fusion stage involves fusing target objects detected independently by multiple sensors, combining the prediction results of each modality model to produce a final result. If a camera detects an obstacle and radar also detects the same obstacle, these results are fused together to form the best estimate of the obstacle's position, category, and velocity. This can be achieved by fusing 3D bounding boxes from LiDAR with 2D bounding boxes from visual detection algorithms. The results from 3D LiDAR can be mapped to 2D and fused within the 2D imagery, but this can also be reversed. Simultaneously, trajectory information can be further fused using prediction and tracking algorithms, adding temporal information to accurately estimate the target object's orientation, velocity, position, or category in real time.

[0090] Post-fusion is a late-stage, target-level fusion method that refers to fusing the results of each type of pipeline. See also Figure 5The post-fusion flowchart shown illustrates a process using a camera and a LiDAR as sensor data sources. It utilizes the outputs from the LiDAR point cloud and camera images to synchronize the image data acquired by the camera and the point cloud data acquired by the LiDAR in a synchronizer, performing time- and spatial synchronization. The camera image data is processed using a Bird's Eye View (BEV) model, converting it into 3D image data and detecting 3D obstacles and their 3D bounding boxes. Similarly, 3D obstacles and their 3D bounding boxes are detected from the LiDAR point cloud data. Finally, the image data and the target objects detected by the LiDAR branch are fused based on these bounding boxes to complete the target detection task. This process combines the BEV algorithm from the image branch and the 3D algorithm from the LiDAR branch, further refining the results of each 3D region algorithm. Furthermore, for each overlapping region, statistical features such as confidence level, distance, and intersection-union ratio (IOU) are used to integrate the 3D bounding box outputs from both modalities. Based on the results from both modalities, a final prediction is made, outputting information such as obstacle orientation, velocity, position, or category. This post-fusion can be viewed as an ensemble method that utilizes multimodal information to optimize the final solution.

[0091] Whether it's the data-level fusion in step S103 or the target-level fusion in step S104, deep learning models can be used, focusing on the feature fusion stage. This application adopts a parallel approach of pre-fusion and post-fusion, employing a dual guarantee mechanism of pre-fusion and post-fusion to effectively ensure high redundancy at the perception algorithm level and ensure robustness. This provides higher reliability, redundancy, and ultimately, higher safety at the vehicle level.

[0092] Step S105: Perform data verification and merging processing on the sensor data after data fusion and the sensor data fused with the target object to obtain sensor perception data.

[0093] After pre-fusion and post-fusion processing, the data can be subjected to final-fusion processing. Integrity verification is performed on the fused sensor data and the fused sensor data of the target object. If the verification passes, the fused sensor data and the fused sensor data of the target object are further fused, and the detected target objects are merged to obtain the final fused sensor perception data.

[0094] When combining multiple sensors based on their different characteristics and fusing multimodal data, the pre-fusion scheme faces challenges. For example, sensors like cameras and radar have significantly different characteristics, requiring data from both methods to be reorganized in a new coordinate system. Deep fusion methods typically use an external calibration matrix to directly project all LiDAR points onto their corresponding pixels, and vice versa. However, due to noise and other interference, this pixel-by-pixel alignment is not precise enough.

[0095] In post-fusion schemes, there are other information losses during the transformation between the input space and feature space. Typically, projection dimensionality reduction inevitably leads to a significant loss of information, such as mapping a 3D LiDAR point cloud to a 2D BEV image. Therefore, both pre-fusion and post-fusion inevitably cause a certain degree of information inaccuracy and loss, resulting in missed detections, false detections, and ultimately, vehicle-level functional failures.

[0096] Since existing simple redundancy methods such as sensor combinations or single-sensor fusion are insufficient to ensure system safety, the multimodal fusion method for autonomous driving perception tasks provided in this application, through higher-level feature representations and more complex deep learning models, fuses information from LiDAR and camera sensor data. This allows for more accurate perception of the surrounding environment and provides additional redundancy in environments where one sensor malfunctions. The fused target data has higher accuracy and reliability than target information output from a single sensor, guiding the function to make better and safer decisions, thereby reducing erroneous decisions such as false triggering and missed triggering.

[0097] The redundancy verification method for autonomous driving perception systems provided in this invention is a novel redundancy verification method at the perception level, which can effectively improve the stability and reliability of the perception algorithm. This method aligns multimodal data from multiple sensor data sources and then performs pre-fusion and post-fusion in parallel. In the pre-fusion stage, preliminary data fusion processing is performed on the multimodal data to obtain fused sensor data. In the post-fusion stage, target objects detected from the multimodal data are fused at the target level to obtain fused sensor data of the target object. By executing pre-fusion and post-fusion in parallel, even if a problem occurs in one stage, multimodal data fusion can still be effectively performed, achieving redundancy verification of multimodal data, overcoming the shortcomings of single-modal data, improving the accuracy and reliability of environmental perception during autonomous driving, thereby accurately predicting hazardous factors in the environment and improving the safety of autonomous driving. Further data verification and merging processing is performed on the fused sensor data and the fused sensor data of the target object. The mutual verification and merging processing of the fused data in the pre-fusion and post-fusion stages can better avoid perception errors, accurately identify target objects, and further improve the accuracy of environmental perception and the safety of autonomous driving.

[0098] Example 2

[0099] Embodiment 2 of the present invention provides a redundancy verification method for an autonomous driving perception system. Building upon the redundancy design of Embodiment 1, it further provides functions such as sensor data anomaly monitoring, time consumption monitoring, resource consumption monitoring, and perception result integrity monitoring, providing a redundancy verification scheme at the perception level from the perspective of expected functional safety. The method includes executing at least one of the following safety monitoring mechanisms:

[0100] (1) Monitor the data for anomalies and issue early warnings when abnormal data is detected.

[0101] See Figure 2 As shown, a data monitor can be added before the data preprocessing stage to monitor abnormal data. It monitors the image data acquired by the image sensor and determines whether there are any anomalies based on the pixel values ​​of the pixels in the image. If the proportion of pixels with the same neighboring pixel value in a frame exceeds a set pixel threshold, or if the number of frames in which some pixels with the same pixel value persists exceeds a set frame retention threshold, or if the duration of the persistence of some pixels with the same pixel value exceeds a set retention time threshold, then the image data is considered abnormal. Image anomalies include at least one of the following: green screen, black screen, light spots, dropped frames, delay, distortion, occlusion, and dirt.

[0102] CRC verification is performed on the point cloud data acquired by the point cloud device. Based on the verification result, it is determined whether there are any anomalies in the point cloud data. Point cloud data anomalies include at least one of the following: empty data, missing partial fields, missing strength information, and missing dimension information.

[0103] The sensor data anomaly monitoring module primarily addresses issues such as green screens, black screens, light spots, frame drops, delays, distortions, occlusions, and dirt in camera image data, and empty data, missing fields, intensity information, and dimension information in LiDAR data. It provides a real-time monitoring module for raw data to prevent sensor algorithm malfunctions due to data problems. For camera image data anomalies, it effectively filters data based on pixel values, the proportion of neighboring pixels in the entire image, and the persistence time of certain pixels, applying thresholds to pinpoint the cause of the anomaly and providing appropriate warnings to the driver via the HMI. For LiDAR point cloud data anomalies, it performs CRC checks to determine if the point cloud data is empty or abnormal, locating the abnormal fields and providing appropriate warnings to the driver via the HMI.

[0104] In this context, a neighborhood refers to the set of pixels that are adjacent to (surrounding) a pixel p(x, y). The four neighboring pixels of pixel p(x, y) are: (x+1, y), (x-1, y), (x, y+1), (x, y-1).

[0105] (2) Monitor the time consumed by the data alignment, front fusion, back fusion and data merging steps respectively, and display the time consumption information.

[0106] A TimeMonitor can be added to each stage of data alignment, pre-fusion, post-fusion, and data merging to monitor processing time. Input timestamps are added to the input data of each functional module performing these steps, and output timestamps are added to the output data. The processing time of each functional module is calculated based on the input and output timestamps to obtain the total processing time for each step.

[0107] See Figure 2 As shown, the entire perception algorithm model is roughly divided into sub-modules such as data preprocessing, pre-fusion, post-fusion, and total fusion (data merging). In the time monitoring safety mechanism, timestamps are added to the input and output ends of each module to count the time consumption, avoid the algorithm delay from affecting the overall vehicle performance, and thus quickly locate the time loss of the corresponding sub-module.

[0108] (3) Monitor the processor's resource consumption and issue an early warning when the resource consumption exceeds the preset resource threshold.

[0109] A resource consumption monitoring module can be added to the perception model to statistically analyze its resource consumption, such as the consumption of CPU and GPU resources. Optionally, the CPU and GPU resource consumption of each functional module can be statistically analyzed separately. If resource consumption is too high, a warning message will be displayed via the HMI. This method allows for real-time monitoring of resource consumption and close monitoring of the operation of the vehicle's Advanced Driving Assistance System (ADAS). If problems occur, they can be quickly located and resolved to prevent related ADAS function failures.

[0110] (4) Monitor the operation status of pre-fusion and post-fusion, and issue an early warning when one of them is abnormal.

[0111] It can monitor the operational status of pre-fusion and post-fusion stages. When pre-fusion or post-fusion fails, although the redundancy mechanism will not have a significant impact on the accuracy of the data, it is still necessary to issue timely warnings and handle abnormal issues.

[0112] (5) Determine whether the fusion result is accurate based on the overlap between the sensor data after data fusion and the sensor data fused with the target object. If it is inaccurate, issue an early warning.

[0113] See Figure 2 As shown, a Results Monitor can be added to the overall fusion stage to monitor the completeness of the sensing results. The intersection and union of the target objects in the fused sensor data and the fused target object sensor data are processed. The crossover-union ratio (CUI) is obtained based on the degree of overlap between the target object bounding boxes in the intersection and union data. The accuracy of the fusion result is determined by the relationship between the CUI and a set overlap rate threshold.

[0114] To avoid missed or false detections in the perception algorithm, a result monitor is added to monitor the integrity of the perception results. This is mainly achieved by performing IOU matching and threshold comparison between the final fused perception output obstacle information (label, bounding box, position, etc.) and the previous and subsequent fused perception results. When there are missed obstacles in the final fused result, the union of the various fused results is used to supplement each other, improving the coverage of the perception results and thus enhancing the robustness of the perception algorithm. Here, IOU refers to the Intersection Over Union, which is the overlap ratio between the predicted bounding box and the ground truth bounding box, i.e., the ratio of their intersection to their union. The ideal situation is complete overlap, with a ratio of 1.

[0115] While single-fusion algorithm schemes incorporate redundant design, they are prone to information inaccuracies and loss, leading to missed detections and false detections at the perception layer. The coexistence and parallel operation of front-end and back-end fusion mechanisms, through the parallel use of multiple algorithm models, ensures the reliability and security of the system at the perception layer, mitigating missed detections and false detections to a certain extent and guaranteeing the normal and orderly operation of the system. In this embodiment, a software-level monitoring and security mechanism is further added to the perception algorithm scheme. Sensor data anomaly monitoring, time monitoring, resource monitoring, and perception result integrity monitoring modules are added to different sub-modules. These mechanisms proactively mitigate risks such as missed detections, false detections, and delays caused by data anomalies, time delays, resource overload, and incomplete perception results. At the software algorithm level, corresponding software security mechanisms are added to ensure high security and strong stability at the perception layer.

[0116] Based on the same inventive concept, embodiments of the present invention also provide an autonomous driving perception system redundancy verification system, which can be installed in an autonomous driving vehicle, and the structure of the system is as follows. Figure 6 As shown, the system includes: an autonomous driving perception system redundancy verification device 1 and at least two different types of sensors 2, which are used to acquire environmental data. The sensors 2 may include at least one of an image sensor and a point cloud device, and may also include other sensors. Multiple types of image sensors and point cloud devices can be configured as needed. For example, the image sensor can be a camera, webcam, etc., used to acquire image data of the vehicle's environment, and the point cloud device can be of different types, used to acquire point cloud data of the vehicle's environment.

[0117] Based on the same inventive concept, embodiments of the present invention also provide an autonomous driving perception system redundancy verification device, which can be installed in the control equipment of an autonomous vehicle, and the structure of the device is as follows. Figure 7 As shown, it includes:

[0118] Data acquisition module 101 is used to acquire multimodal data from multiple sensor data sources;

[0119] The data alignment module 102 is used to align multimodal data based on a pre-established vehicle coordinate system;

[0120] The pre-fusion module 103 and the post-fusion module 104 operate in parallel. The pre-fusion module is used to perform preliminary data fusion processing on the aligned multimodal data to obtain fused sensor data. The post-fusion module is used to perform post-target-level fusion on the target objects detected from the aligned multimodal data to obtain fused sensor data of the target objects.

[0121] The overall fusion module 105 is used to perform data verification and merging processing on the sensor data after data fusion and the sensor data fused with the target object to obtain sensor perception data.

[0122] In some optional embodiments, the above-described apparatus further includes at least one of the following monitors:

[0123] Data monitor 106 is used to monitor data for anomalies and issue early warnings when abnormal data is detected.

[0124] The time monitor 107 is used to monitor the time consumed by the data alignment, pre-fusion, post-fusion and data merging steps respectively, and display the time consumption information.

[0125] Resource monitor 108 is used to monitor the processor's resource consumption and issue an early warning when the resource consumption exceeds a preset resource threshold.

[0126] The result monitor 109 is used to monitor the operation status of pre-fusion and post-fusion, and to issue an early warning when one of them is abnormal; and to determine whether the fusion result is accurate based on the overlap between the sensor data after data fusion and the sensor data fused with the target object, and to issue an early warning if it is inaccurate.

[0127] This invention also provides a computer storage medium storing computer-executable instructions, which, when executed by a processor, implement the aforementioned redundancy verification method for an autonomous driving perception system.

[0128] This invention also provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method for redundancy verification of an autonomous driving perception system.

[0129] Regarding the systems and apparatus in the above embodiments, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0130] The methods and apparatus described in this invention effectively avoid information inaccuracies and loss in a single fusion scheme by combining pre-fusion and post-fusion, thereby improving the detection accuracy and other performance of the perception layer algorithm. Through a dual-layer protection mechanism of pre-fusion and post-fusion, algorithm failures at the perception layer are effectively avoided, preventing vehicle-level risks and improving robustness and safety. A monitoring module security mechanism is provided at the software algorithm level to monitor the operation of the perception submodule in real time, and feedback through the Human-Machine Interface (HMI) allows for rapid identification of problems, preventing vehicle-wide malfunctions and hazards.

[0131] Unless otherwise specifically stated, terms such as processing, calculation, operation, determination, display, etc., may refer to the actions and / or processes of one or more processing or computing systems or similar devices that represent the manipulation and conversion of data representing physical (e.g., electronic) quantities within the registers or memory of the processing system into other data similarly representing physical quantities within the memory, registers, or other such information storage, transmission, or display devices of the processing system. Information and signals can be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips mentioned throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or particles, light fields or particles, or any combination thereof.

[0132] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0133] In the detailed description above, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features in a single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, with each claim representing a separate preferred embodiment of the invention.

[0134] Those skilled in the art will also understand that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments herein can be implemented as electronic hardware, computer software, or a combination thereof. To clearly illustrate the interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps described above are generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in alternative ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this disclosure.

[0135] The steps of the methods or algorithms described in conjunction with the embodiments herein can be directly embodied in hardware, software modules executed by a processor, or a combination thereof. The software modules can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in a user terminal. Alternatively, the processor and storage medium can exist as discrete components in the user terminal.

[0136] For software implementation, the techniques described in this application can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described in this application. This software code can be stored in memory units and executed by a processor. The memory units can be implemented within the processor or outside the processor; in the latter case, they are communicatively coupled to the processor via various means, as is well known in the art.

[0137] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

Claims

1. A method for redundancy verification in an autonomous driving perception system, characterized in that, include: Acquire multimodal data from multiple sensor data sources; The multimodal data is aligned based on a pre-established vehicle coordinate system. The aligned multimodal data is subjected to parallel pre-fusion and post-fusion. The pre-fusion includes performing preliminary data fusion processing on the aligned multimodal data to obtain fused sensor data. The preliminary data fusion processing includes combining image data into point cloud data based on the boundary of two-dimensional target objects in the image data. The post-fusion includes performing post-target-level fusion on the target objects detected from the aligned multimodal data to obtain sensor data of the target objects fused. Post-target-level fusion involves merging the two-dimensional target objects identified in the image data and the three-dimensional target objects identified in the point cloud data based on the boundaries of the two-dimensional target objects in the image data and the boundaries of the three-dimensional target objects in the point cloud data. Data verification and merging are performed on the sensor data after data fusion and the sensor data fused with the target object to obtain sensor perception data; The intersection and union of the target objects in the sensor data after data fusion and the sensor data after target object fusion are processed. The intersection-union ratio is obtained based on the degree of overlap and matching of the target object bounding boxes in the data after intersection processing and the data after union processing. The accuracy of the fusion result of the sensor data after data fusion and the sensor data fused with the target object is determined based on the relationship between the cross-union ratio and the set overlap rate threshold. If the result is inaccurate, an early warning is issued.

2. The method as described in claim 1, characterized in that, The alignment of the multimodal data based on a pre-established vehicle coordinate system includes: A vehicle coordinate system is established with the midpoint of the axis between the two rear wheels of the vehicle as the origin, and the axis and the line perpendicular to the axis as the horizontal and vertical coordinate axes. The data from different sensor data sources are transformed into the vehicle coordinate system to achieve the alignment of the multimodal data.

3. The method as described in claim 1, characterized in that, The aligned multimodal data undergoes preliminary data fusion processing to obtain fused sensor data, including: The image data acquired by the image sensor and the point cloud data acquired by the point cloud device are processed for temporal and spatial synchronization. Identify two-dimensional target objects from the image data and determine the boundaries of the two-dimensional target objects; Based on the boundaries of the two-dimensional target object, the image data identifying the two-dimensional target object is combined into the point cloud data to obtain the fused sensor data; and / or, The target objects detected from the aligned multimodal data are subjected to post-target-level fusion to obtain sensor data of the fused target objects, including: The image data acquired by the image sensor and the point cloud data acquired by the point cloud device are processed for temporal and spatial synchronization. Identify two-dimensional target objects from the image data and determine the boundaries of the two-dimensional target objects; The three-dimensional target object is identified from the point cloud data, and the boundary of the three-dimensional target object is determined; Based on the boundaries of the two-dimensional target objects and the three-dimensional target objects, the two-dimensional target objects identified in the image data and the three-dimensional target objects identified in the point cloud data are merged to obtain the sensor data of target object fusion.

4. The method as described in claim 1, characterized in that, Based on the fused sensor data and the fused sensor data of the target object, data verification and merging processes are performed to obtain sensor perception data, including: Perform integrity verification on the sensor data after data fusion and the sensor data fused with the target object; After the test is passed, the sensor data after data fusion and the sensor data fused with the target object are fused together, and the detected target objects are merged to obtain the total fused sensor perception data.

5. The method according to any one of claims 1-4, characterized in that, It also includes implementing at least one of the following security monitoring mechanisms: Monitor data for anomalies and issue early warnings upon detecting abnormal data; Monitor the time consumed by each step of data alignment, pre-fusion, post-fusion, and data merging, and display the time consumption information. Monitor processor resource consumption and issue an early warning when resource consumption exceeds a preset resource threshold; Monitor the operation status of pre-fusion and post-fusion, and issue an early warning when one of them is abnormal.

6. The method as described in claim 5, characterized in that, Anomaly monitoring of the monitoring data includes: The system monitors image data acquired by image sensors and determines whether there are any anomalies in the image data based on the pixel values ​​of the pixels in the image. If the proportion of pixels with the same neighboring pixel value in a frame of an image is greater than a set pixel value threshold, or the number of frames in which some pixels with the same pixel value persists in the image is greater than a set frame number threshold, or the duration of the persistence of some pixels with the same pixel value in the image is greater than a set duration threshold, the image data is considered to be abnormal. Image anomalies include at least one of the following: green screen, black screen, light spots, dropped frames, delay, distortion, occlusion, and dirt in the image. Perform CRC verification on the point cloud data acquired by the point cloud device, and determine whether there are any anomalies in the point cloud data based on the verification result. Point cloud data anomalies include at least one of the following: empty data, missing fields, missing strength information, and missing dimension information; and / or, Monitor the time consumed by each step: data alignment, pre-fusion, post-fusion, and data merging. In each functional module that performs data alignment, front fusion, back fusion, and data merging steps, an input timestamp is added to the input data at the input end, and an output timestamp is added to the output data at the output end. The processing time of each functional module is calculated based on the input timestamp and the output timestamp to obtain the time consumed by each step.

7. A redundancy verification device for an autonomous driving perception system, characterized in that, include: The data acquisition module is used to acquire multimodal data from multiple sensor data sources; The data alignment module is used to align the multimodal data based on a pre-established vehicle coordinate system. The system comprises a pre-fusion module and a post-fusion module that operate in parallel. The pre-fusion module performs preliminary data fusion processing on the aligned multimodal data to obtain fused sensor data. This preliminary data fusion processing includes combining image data into point cloud data based on the boundaries of two-dimensional target objects in the image data. The post-fusion module performs post-target-level fusion on the target objects detected from the aligned multimodal data to obtain fused sensor data of the target objects. This post-target-level fusion includes merging the two-dimensional target objects identified in the image data and the three-dimensional target objects identified in the point cloud data based on the boundaries of the two-dimensional target objects in the image data and the boundaries of the three-dimensional target objects in the point cloud data. The overall fusion module is used to perform data verification and merging processing on the sensor data after data fusion and the sensor data fused from the target object to obtain sensor perception data; The intersection and union of the target objects in the sensor data after data fusion and the sensor data after target object fusion are processed. The intersection-union ratio is obtained based on the degree of overlap and matching of the target object bounding boxes in the data after intersection processing and the data after union processing. The accuracy of the fusion result of the sensor data after data fusion and the sensor data fused with the target object is determined based on the relationship between the cross-union ratio and the set overlap rate threshold. If the result is inaccurate, an early warning is issued.

8. A redundancy verification system for an autonomous driving perception system, characterized in that, include: At least two different types of sensors and the redundancy verification device for the autonomous driving perception system as described in claim 7.

9. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which, when executed by a processor, implement the redundancy verification method for the autonomous driving perception system according to any one of claims 1-6.

10. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the redundancy verification method for an autonomous driving perception system according to any one of claims 1-6.

Images