Perception enhancement method and device based on dual-system heterogeneity and electronic equipment

Abstract

The application provides a perception enhancement method and device based on a dual-system heterogeneous system and electronic equipment, which can be applied to the field of unmanned driving technology. The method comprises the following steps: acquiring first multi-modal sensor data collected by a first sensing system and second multi-modal sensor data collected by a second sensing system; inputting the first multi-modal sensor data into a first model to output a local planning track of a vehicle and first obstacle perception information around the vehicle; inputting the second multi-modal sensor data into a second model to output second obstacle perception information of the vehicle and a three-dimensional space occupation grid of the surrounding environment of the vehicle, wherein the network structure of the second model is different from that of the first model; checking the first obstacle perception information based on the second obstacle perception information, and checking the local planning track based on the three-dimensional space occupation grid to obtain a checking result; and guiding the vehicle to move based on the local planning track when the checking result is passed.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, specifically to a perception enhancement method, device, and electronic device based on dual-system heterogeneity. Background Technology

[0002] Autonomous driving systems typically rely on perception modules to understand the environment and perform trajectory planning and vehicle control based on the perception results. Existing solutions mainly fall into two categories: one is a cascaded perception solution based on a single sensor system, where the perception results serve as the sole basis for planning. If the perception system misses or falsely detects anything, it will directly lead to an unsafe trajectory. The other is an end-to-end autonomous driving solution, which directly inputs sensor data into a single neural network to output trajectory or control commands. The internal reasoning process of this type of solution is difficult to interpret and verify, and it lacks independent verification and fallback mechanisms, thus facing safety risks in long-tail scenarios as well.

[0003] Furthermore, all of the above solutions are highly dependent on the proper functioning of individual sensor suites. LiDAR, cameras, and millimeter-wave radar may all experience performance degradation or failure under conditions such as rain, fog, strong light overexposure, or obstruction. Failure of any type of sensor will result in the system losing the corresponding mode of perception capability, leading to a decline in overall perception performance and consequently increasing the risk of collision. Summary of the Invention

[0004] In view of the above problems, this application provides a perception enhancement method, apparatus and electronic device based on dual-system heterogeneity to improve perception reliability and driving safety.

[0005] According to a first aspect of this application, a perception enhancement method based on dual-system heterogeneity is provided, comprising: acquiring first multimodal sensor data collected by a first sensing system and second multimodal sensor data collected by a second sensing system; inputting the first multimodal sensor data into a first model deployed on a first computing chip, and outputting a local planned trajectory of the vehicle and first obstacle perception information around the vehicle; inputting the second multimodal sensor data into a second model deployed on a second computing chip, and outputting second obstacle perception information of the vehicle and a three-dimensional spatial occupancy grid of the vehicle's surrounding environment, wherein the network structure of the second model is different from that of the first model; verifying the first obstacle perception information based on the second obstacle perception information, and verifying the local planned trajectory based on the three-dimensional spatial occupancy grid to obtain a verification result; and guiding the vehicle to move based on the local planned trajectory if the verification result is successful.

[0006] According to an embodiment of this application, the step of inputting the first multimodal sensor data into a first model deployed on a first computing chip and outputting the local planning trajectory of the vehicle and the first obstacle perception information around the vehicle includes: extracting features from the image data, lidar point cloud data, and millimeter-wave radar detection point data in the first multimodal sensor data to obtain first multi-scale two-dimensional image features, first three-dimensional voxel features of the vehicle's surrounding environment, and first radar features; performing spatiotemporal alignment and fusion of the first multi-scale two-dimensional image features, the first three-dimensional voxel features, and the first radar features in a bird's-eye view to obtain fused bird's-eye view features; fusing the fused bird's-eye view features of the current frame with the fused bird's-eye view features of K historical frames through a temporal self-attention mechanism to obtain temporal fusion features, where K is a positive integer; and inputting the temporal fusion features into multiple different task decoding branches to output the local planning trajectory and the first obstacle perception information in parallel.

[0007] According to an embodiment of this application, the step of inputting the second multimodal sensor data into a second model deployed on a second computing chip and outputting the second obstacle perception information of the vehicle and the three-dimensional spatial occupancy grid of the vehicle's surrounding environment includes: extracting features from the image data, lidar point cloud data, and millimeter-wave radar detection point data in the second multimodal sensor data based on an algorithm different from the first large model, thereby obtaining second multi-scale two-dimensional image features, second three-dimensional sparse features of the vehicle's surrounding environment, and second radar features; projecting the second multi-scale two-dimensional image features, the second three-dimensional sparse features of the vehicle's surrounding environment, and the second radar features into a three-dimensional voxel space to construct a three-dimensional voxel feature field; inputting the three-dimensional voxel feature field into an occupancy prediction branch and outputting the three-dimensional spatial occupancy grid; and outputting the second obstacle perception information through a target detection branch based on the three-dimensional voxel feature field or the three-dimensional spatial occupancy grid.

[0008] According to an embodiment of this application, the step of verifying the first obstacle perception information based on the second obstacle perception information and verifying the local planning trajectory based on the three-dimensional space occupancy grid to obtain a verification result includes: performing target association matching between each target object in the second obstacle perception information and each target object in the first obstacle perception information to obtain a matching target object pair; verifying the relative distance and collision time of each trajectory point on the local planning trajectory relative to the predicted position of each target object in the target object pair to obtain a first verification sub-result; verifying whether each trajectory point on the local planning trajectory falls within the voxel unit marked as occupied in the three-dimensional space occupancy grid to obtain a second verification sub-result; and determining the verification result if both the first verification sub-result and the second verification sub-result pass the verification.

[0009] According to an embodiment of this application, before verifying the first obstacle perception information based on the second obstacle perception information, verifying the local planning trajectory based on the three-dimensional space occupancy grid, and obtaining the verification result, the method further includes: spatiotemporally aligning each target object in the second obstacle perception information with each target object in the first obstacle perception information; and spatiotemporally aligning the three-dimensional space occupancy grid with the local planning trajectory.

[0010] According to an embodiment of this application, the step of verifying the relative distance and collision time of each trajectory point on the local planning trajectory with respect to the predicted positions of each target object in the target object pair to obtain a first verification result includes: calculating the minimum distance between the vehicle envelope at each trajectory point on the local planning trajectory and the predicted position envelope of each target object in the target object pair; calculating the collision time at each trajectory point based on the minimum distance and the movement speed of each target object; and determining that the first verification result is verified as passed if the minimum distance at each trajectory point is greater than or equal to a preset safety distance threshold and the collision time at each trajectory point is greater than or equal to a preset collision time threshold.

[0011] According to an embodiment of this application, the step of verifying whether each trajectory point on the local planning trajectory falls within the voxel unit marked as occupied in the three-dimensional space occupation grid to obtain the second verification result includes: obtaining the occupation probability of the voxel unit at the corresponding position of each trajectory point on the local planning trajectory in the three-dimensional space occupation grid; if the occupation probability at the corresponding position of each trajectory point is less than a preset occupation probability threshold, then the second verification result is determined to be a successful verification.

[0012] According to an embodiment of this application, if the verification result is a failure, the method further includes: constructing a passable area based on the passable space in the three-dimensional spatial occupancy grid and the predicted positions of each target object in the second obstacle perception information; performing path search within the passable area to generate a backup planning trajectory; re-verifying the backup planning trajectory; and, if the backup planning trajectory passes the verification, using the backup planning trajectory to guide the vehicle's movement.

[0013] A second aspect of this application provides a perception enhancement device based on a dual-system heterogeneity, the device comprising: a data acquisition module for acquiring first multimodal sensor data acquired by a first sensing system and second multimodal sensor data acquired by a second sensing system; a first processing module for inputting the first multimodal sensor data into a first model deployed on a first computing chip, and outputting a local planned trajectory of the vehicle and first obstacle perception information around the vehicle; a second processing module for inputting the second multimodal sensor data into a second model deployed on a second computing chip, and outputting second obstacle perception information of the vehicle and a three-dimensional spatial occupancy grid of the vehicle's surrounding environment, wherein the network structure of the second model is different from that of the first model; a verification module for verifying the first obstacle perception information based on the second obstacle perception information, and verifying the local planned trajectory based on the three-dimensional spatial occupancy grid to obtain a verification result; and a control module for guiding the vehicle to move based on the local planned trajectory if the verification result is a successful verification.

[0014] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above. Attached Figure Description

[0015] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0016] Figure 1 A flowchart illustrating a perception enhancement method based on dual-system heterogeneity according to an embodiment of this application is shown schematically;

[0017] Figure 2 The illustration shows a schematic diagram of dual-model collaborative cooperation in a perception enhancement method based on dual-system heterogeneity according to an embodiment of this application;

[0018] Figure 3 A schematic diagram illustrates a structural block diagram of a dual-system heterogeneous sensing enhancement device according to an embodiment of this application; and

[0019] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a dual-system heterogeneous perception enhancement method according to an embodiment of this application. Detailed Implementation

[0020] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0022] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0023] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0024] Figure 1 A flowchart illustrating a perception enhancement method based on dual-system heterogeneity according to an embodiment of this application is shown.

[0025] like Figure 1 As shown, the perception enhancement method based on dual-system heterogeneity in this embodiment includes operations S110 to S150.

[0026] In operation S110, first multimodal sensor data collected by the first sensing system and second multimodal sensor data collected by the second sensing system are acquired.

[0027] Multimodal sensor data refers to raw data collected by various types of sensors (such as cameras, lidar, millimeter-wave radar, etc.) that reflects different dimensions of information about the vehicle's surrounding environment. The first sensing system and the second sensing system are two independently set sensor suites, each including at least one or more combinations of cameras, lidar, and millimeter-wave radar.

[0028] In operation S120, the first multimodal sensor data is input to the first model deployed on the first computing chip, and the local planned trajectory of the vehicle and the first obstacle perception information around the vehicle are output.

[0029] The first model is an end-to-end network based on bird's-eye view feature fusion and multi-task parallel decoding. It achieves integrated reasoning for perception and planning through a unified representation of multimodal features in the bird's-eye view space. The local planned trajectory includes the position, heading, velocity, and curvature information of a series of trajectory points within the next few seconds. The first obstacle perception information describes the position, size, type, and motion state of obstacles in the form of 3D bounding boxes. In operation S130, the second multimodal sensor data is input to the second model deployed on the second computing chip, outputting the vehicle's second obstacle perception information and a 3D spatial occupancy grid of the vehicle's surrounding environment. The network structure of the second model differs from that of the first model.

[0030] The second model employs a perception network with a different structure than the first model. It performs inference based on three-dimensional voxel space construction and occupancy prediction, providing a set of perception results independent of the main system and avoiding common-cause failures with the main system. The three-dimensional occupancy grid is a 3D voxelized grid of the vehicle's surrounding environment, used to characterize whether each voxel unit is occupied by obstacles, reflecting 3D spatial information that cannot be expressed in a two-dimensional bird's-eye view, such as suspended obstacles.

[0031] In operation S140, the first obstacle perception information is verified based on the second obstacle perception information, and the local planning trajectory is verified based on the three-dimensional space occupancy grid to obtain the verification result.

[0032] This step achieves end-to-end security verification from perception to trajectory through two independent checks: the first check compares the obstacle detection results of the two heterogeneous systems to verify the credibility of the main system's perception; the second check detects spatial conflicts between the trajectory planned by the main system and the three-dimensional space grid to verify the trajectories themselves are passable in physical space.

[0033] When operating S150, if the verification result is successful, the vehicle is guided to move based on the local planning trajectory.

[0034] If the verification passes, the vehicle's control module receives the verified local planned trajectory and, based on the position, heading, speed, and curvature information contained in the trajectory, controls the vehicle's steering, acceleration, and braking actuators to ensure the vehicle travels along the trajectory. If the verification fails, the system switches to a safety degradation processing path, such as by replanning the path, to maintain the vehicle in a safe state. Thus, through the above steps, a complete closed-loop process is completed, from multi-source data acquisition, dual-system heterogeneous sensing, and dual safety verification to trajectory execution or safety degradation.

[0035] The following will provide a detailed description of the perception enhancement method based on dual-system heterogeneity provided in this application, in conjunction with steps S110 to S150.

[0036] In step S110, first multimodal sensor data collected by the first sensing system and second multimodal sensor data collected by the second sensing system are acquired.

[0037] The first and second sensing systems are two independently configured sensor suites, each including at least one or more combinations of a camera, LiDAR, and millimeter-wave radar. The first sensing system provides the primary sensing data source, while the second sensing system provides an independent sensing data source for the secondary system. The two sensing systems are completely decoupled at the hardware level, each with its own independent power supply, communication interface, and data processing link, and do not interfere with each other. Multimodal sensor data refers to raw data collected by various types of sensors, reflecting different dimensions of information about the vehicle's surrounding environment, such as image data collected by a camera, 3D point cloud data collected by LiDAR, and detection point data collected by millimeter-wave radar (which may include information such as distance, azimuth, and radial velocity). This data is then provided to the first and second models respectively for further processing.

[0038] By setting up two independent sensing systems, this embodiment of the application avoids the risk of loss of overall sensing capability due to the failure of a single sensor kit from the source of data acquisition, and provides basic data protection for subsequent dual-system heterogeneous sensing and security verification.

[0039] In step S120, the first multimodal sensor data is input to the first model deployed on the first computing chip, and the local planning trajectory of the vehicle and the first obstacle perception information around the vehicle are output. S120 includes S121 to S124.

[0040] In step S121, feature extraction is performed on the image data, lidar point cloud data and millimeter-wave radar detection point data in the first multimodal sensor data, respectively, to obtain the first multi-scale two-dimensional image features, the first three-dimensional voxel features of the vehicle's surrounding environment and the first radar features.

[0041] In this embodiment, image data is processed using a first image backbone network (e.g., ResNet or Swin-Transformer) to extract multi-scale two-dimensional image features, which preserve texture, edge, and semantic information in the image. LiDAR point cloud data is processed using a first point cloud backbone network (e.g., VoxelNet or PointPillars) to extract three-dimensional voxel features, which preserve the geometric structure information of objects in three-dimensional space. Millimeter-wave radar detection point data is processed using a first multilayer perceptron for feature embedding to obtain first radar features, which include information such as the target's distance, azimuth, and radial velocity. These three features are extracted from sensor data of different modalities, each reflecting different environmental attributes, providing complementary information sources for subsequent multimodal fusion.

[0042] In step S122, the first multi-scale two-dimensional image features, the first three-dimensional voxel features, and the first radar features are spatiotemporally aligned and fused under the bird's-eye view to obtain fused bird's-eye view features.

[0043] Since the three features originate from different sensors and have different data formats and coordinate systems, spatiotemporal alignment is necessary. Specifically, image features are projected onto the Bird's-Eye View (BEV) space through viewpoint transformation. Point cloud features and radar features are directly mapped to the BEV grid according to their three-dimensional spatial coordinates. Simultaneously, data collected at the same time are timestamped to ensure spatiotemporal consistency of features during fusion. The aligned three features are then fused within the same BEV grid to obtain a unified fused BEV feature. This feature maintains the global perception range of the bird's-eye view spatially and integrates the complementary advantages of multiple modal sensors in terms of information.

[0044] In step S123, the fused bird's-eye view features of the current frame and the fused bird's-eye view features of K historical frames are fused with contextual information through a temporal self-attention mechanism to obtain temporal fused features, where K is a positive integer.

[0045] Single-frame BEV features only reflect the current environmental state and cannot provide motion information. This step caches the temporal fusion features (BEV features) of historical K-frames, using the current frame features as the query matrix and the historical K-frame features as the key and value matrices. A temporal self-attention mechanism is used to calculate the spatiotemporal similarity between spatial locations in the current frame and historical frames, obtaining an attention weight matrix. Weighted summation then yields a temporal fusion feature that integrates motion trends and historical context information. This feature provides subsequent decoding branches with dynamic information such as obstacle speed and heading changes.

[0046] In step S124, the temporal fusion features are input into multiple different task decoding branches, and the local planning trajectory and the first obstacle perception information are output in parallel.

[0047] Single-frame BEV features only reflect the current environmental state and cannot provide motion information. This step caches the fused BEV features of historical K-frames, using the current frame features as the query matrix and the historical K-frame features as the key and value matrices. A temporal self-attention mechanism is used to calculate the spatiotemporal similarity between spatial locations in the current frame and those in historical frames, obtaining an attention weight matrix. Weighted summation then yields a temporal fused feature that integrates motion trends and historical context information. This feature provides subsequent decoding branches with dynamic information such as obstacle speed and heading changes.

[0048] In step S130, the second multimodal sensor data is input to the second model deployed on the second computing chip, and the second obstacle perception information of the vehicle and the three-dimensional spatial occupancy grid of the vehicle's surrounding environment are output. S130 includes S131 to S134.

[0049] In step S131, based on an algorithm different from the first large model, feature extraction is performed on the image data, lidar point cloud data and millimeter-wave radar detection point data in the second multimodal sensor data, respectively, to obtain the second multi-scale two-dimensional image features, the second three-dimensional sparse features of the vehicle's surrounding environment and the second radar features.

[0050] In this embodiment, image data is processed using a second image backbone network (e.g., DINOv2 or ConvNeXt) to extract multi-scale two-dimensional image features. The network structure of this second image backbone network differs from that of the first image backbone network used in the first model. LiDAR point cloud data is processed using a sparse convolutional network (SparseConv) to extract three-dimensional sparse features. This network performs feature calculations only at the spatial locations where the point cloud exists, resulting in higher computational efficiency compared to the regular voxel grid method. Millimeter-wave radar detection point data is processed using a second multilayer perceptron for feature embedding to obtain second radar features. By employing a backbone network and feature extraction algorithm completely different from the first model, this embodiment ensures heterogeneity between the two models at the algorithm architecture level, thereby preventing both systems from failing simultaneously due to the same algorithmic defects.

[0051] In step S132, the second multi-scale two-dimensional image features, the second three-dimensional sparse features of the vehicle's surrounding environment, and the second radar features are projected into a three-dimensional voxel space to construct a three-dimensional voxel feature field.

[0052] Image features are projected onto corresponding positions in 3D space through viewpoint transformation. The 3D sparse features output by the sparse convolutional network are mapped to corresponding voxel units in a 3D voxel grid according to their spatial coordinates. Radar features are embedded into corresponding voxel units based on the 3D spatial position of their detection points. Through the above projection operation, features from different modalities are uniformly mapped to the same 3D voxel space, forming a structured 3D voxel feature field, providing a unified feature representation for subsequent occupancy prediction and target detection. This 3D voxel feature field retains information in the height direction and can characterize three-dimensional structures such as suspended obstacles that cannot be expressed in 2D BEV space.

[0053] In step S133, the three-dimensional voxel feature field is input into the occupancy prediction branch, and the three-dimensional spatial occupancy grid is output.

[0054] The 3D voxel feature field contains structured feature information of the vehicle's surrounding environment in 3D space. The occupancy prediction branch performs binary or soft classification prediction on each voxel unit in this feature field to determine whether each voxel unit is occupied by an obstacle, outputting a 3D spatial occupancy grid. This occupancy grid, in the form of a 3D voxel mesh, represents the three-dimensional occupancy state of the vehicle's surrounding environment, comprehensively reflecting various occupancy information including ground obstacles, suspended obstacles, and other obstacles in 3D space, providing a complete spatial constraint basis for subsequent trajectory safety verification.

[0055] In step S134, based on the three-dimensional voxel feature field or the three-dimensional spatial occupancy grid, the second obstacle perception information is output through the target detection branch.

[0056] The target detection branch can perform target detection in one of two ways: one is sparse target detection directly based on the 3D voxel feature field, and the other is target detection based on a 3D spatial grid. This branch outputs second obstacle perception information through a target query mechanism or a grid-based clustering segmentation method, describing the obstacle's position, velocity, heading, size, category, and confidence level in the form of a 3D bounding box. This perception information is completely independent of the first obstacle perception information output by the first model in terms of source and algorithm, and can serve as an independent basis for consistency comparison with the main system's perception results in subsequent verification steps.

[0057] In step S140, the first obstacle perception information is verified based on the second obstacle perception information, and the local planning trajectory is verified based on the three-dimensional space occupation grid to obtain the verification result. S140 includes S141 to S144.

[0058] In step S141, each target object in the second obstacle perception information is matched with each target object in the first obstacle perception information to obtain a matching target object pair.

[0059] Before verification, each target in the second obstacle perception information is spatiotemporally aligned with each target in the first obstacle perception information. Specifically, each target in the second obstacle perception information is projected according to its motion state (e.g., velocity, heading) to the spatiotemporal position corresponding to each trajectory point of the local planned trajectory, aligning it with the first obstacle perception information under the same spatiotemporal reference. Simultaneously, the 3D spatial occupancy grid is uniformly mapped to the world coordinate system of the local planned trajectory, ensuring that subsequent spatial conflict detection is performed in the same coordinate system.

[0060] After spatiotemporal alignment, each target in the aligned second obstacle perception information is matched with each target in the first obstacle perception information. For example, by calculating the bounding box intersection-union ratio or center point distance, the same physical entity detected by both systems is identified, and matching target pairs are obtained. This matching result is a prerequisite for subsequent distance and collision time verification. Only matched targets are included in the calculation to avoid misjudgments caused by inconsistencies in the number of detection results from the two systems.

[0061] In step S142, the relative distance and collision time of each trajectory point on the local planning trajectory relative to the predicted position of each target object in the target object pair are verified to obtain the first verification result.

[0062] For each trajectory point on the locally planned trajectory, calculate the minimum distance between the vehicle's envelope (i.e., the smallest circumscribed convex polygon considering vehicle size) at that trajectory point and the predicted position envelopes of the matched targets. Based on this minimum distance and the motion velocity of each target, calculate the minimum distance using the formula... Calculate the collision time at each trajectory point, where d i v is the minimum distance to the i-th trajectory point. i Let ϵ be the relative velocity. v To prevent division by zero, the minimum relative velocity is used. If the minimum distance between each trajectory point on the locally planned trajectory is greater than or equal to the preset safety distance threshold, and the collision time of each trajectory point is greater than or equal to the preset collision time threshold, it indicates that the trajectory maintains a sufficient safety margin with the target object jointly confirmed by the two systems, and the first verification result is determined to be a successful verification; otherwise, it indicates that the trajectory has a collision risk, and the first verification result is a failed verification.

[0063] In step S143, it is verified whether each trajectory point on the local planning trajectory falls within the voxel unit marked as occupied in the three-dimensional space occupied grid, and the second verification result is obtained.

[0064] In some embodiments, the occupancy probability of each trajectory point on the local planning trajectory at its corresponding position in the 3D space occupancy grid is obtained. If the occupancy probability at the corresponding position of each trajectory point is less than a preset occupancy probability threshold, it indicates that the position of the trajectory point is not occupied by obstacles in the 3D space, and the trajectory is passable in the 3D space. In this case, the second verification result is determined to be a successful verification. Conversely, if any trajectory point falls into a voxel with an occupancy probability greater than or equal to the threshold, it indicates that the trajectory will traverse the occupied space area, and the second verification result is a failed verification. This verification utilizes the 3D space occupancy grid's ability to perceive 3D spaces such as suspended obstacles, compensating for the lack of height information in the 2D BEV grid.

[0065] In step S144, if both the first and second verification results pass the verification, the verification result is determined.

[0066] When both the first and second checksum results pass, it indicates that the local planning trajectory output by the main system has passed dual verification of distance and collision time at the dynamic target level, and also passed space crossing detection at the static and dynamic occupied space levels. The trajectory meets the safety conditions in all verification dimensions, therefore the verification result is determined to be passed. Conversely, if any checksum result fails, the verification result is determined to be failed.

[0067] In S150, if the verification result is successful, the locally planned trajectory is sent to the vehicle control module. The control module drives the steering, acceleration, and braking actuators according to the position, heading, speed, and curvature information contained in the trajectory, so that the vehicle travels along the trajectory. If the verification result is unsuccessful, the system switches to a safety degradation processing path, such as by replanning the path, to maintain the vehicle in a safe state. Thus, through the above steps, a closed loop is completed from multi-source data acquisition, dual-system heterogeneous perception, dual safety verification to trajectory execution or safety degradation.

[0068] In some embodiments, if the verification result is a failure, a passable area is constructed based on the passable space in the three-dimensional spatial grid and the predicted positions of each target object in the second obstacle perception information; a path search is performed within the passable area to generate a backup planning trajectory; the backup planning trajectory is re-verified; if the backup planning trajectory passes the verification, the backup planning trajectory is used to guide the vehicle to move.

[0069] Specifically, the passable area is constructed as follows: voxel units marked as passable are extracted from the 3D spatial occupancy grid to form the initial passable space; based on the predicted position and size of each target object in the second obstacle perception information, an expansion region for each target object is generated; the expansion region is removed from the initial passable space to obtain the final passable area. The expansion region refers to the area formed by extending a certain safety distance outward from the actual space occupied by the target object, used to reserve a safety margin between the vehicle and the obstacle, and to prevent the backup planned trajectory from being too close to the obstacle.

[0070] Within the established traversable area, a coarse path search algorithm (such as Algorithm A or Hybrid Algorithm A) is used to perform a coarse path search to obtain an initial path from the vehicle's current position to the target point. Subsequently, using the traversable area as a constraint, a quadratic programming algorithm is used to smooth the initial path, generating a backup planned trajectory that satisfies the vehicle's kinematic constraints. By combining coarse programming with smoothing optimization, the smoothness and executability of the trajectory can be improved while ensuring path safety.

[0071] The aforementioned verification steps (i.e., the verification process described in steps S142 to S144) are re-executed on the generated backup planned trajectory to verify its safety. If the backup planned trajectory passes the verification, it is issued and executed as the final control command; if it still fails, it indicates that no feasible path meeting the safety conditions can be found within the currently passable area, and a stop command is output to control the vehicle to brake safely. Through the above-mentioned safety degradation processing mechanism, this embodiment of the application can attempt to generate a safe alternative path with the help of the secondary system's perception results when the main system trajectory fails the verification, and only triggers a stop when no safe path can be found, thereby maintaining the vehicle's normal traffic capacity as much as possible while ensuring safety.

[0072] In some embodiments, after re-verifying the backup planned trajectory, if the verification passes, the backup planned trajectory, carrying a backup identifier, is sent to the vehicle control module to distinguish it from the original trajectory output by the main system; if the verification still fails, a parking command is output to the vehicle control module to control the vehicle to brake and stop.

[0073] Furthermore, the verification parameters involved in the above embodiments, such as the safety distance threshold, collision time threshold, occupancy probability threshold, and prediction time domain, can all be configured to adaptively adjust based on the vehicle's current speed, road grade, and / or current weather conditions. For example, the safety distance threshold can be increased when driving at high speeds, while the collision time threshold can be appropriately relaxed in rainy or foggy weather to balance safety and traffic efficiency. These parameters need to be calibrated through scenario simulation and real-vehicle testing during the system verification phase to match the actual performance requirements under different operating conditions.

[0074] Figure 2The illustration shows a schematic diagram of dual-model collaborative cooperation in a dual-system heterogeneous perception enhancement method according to an embodiment of this application.

[0075] The first sensing system (sensor kit A) and the second sensing system (sensor kit B) are configured independently, each including at least one or more combinations of a camera, LiDAR, and millimeter-wave radar. The first sensing system is used to acquire data from the first multimodal sensor and provide it to a first model deployed on a first computing chip (computing chip A); the second sensing system is used to acquire data from the second multimodal sensor and provide it to a second model deployed on a second computing chip (computing chip B). The two sensing systems are completely independent in hardware and do not affect each other.

[0076] The first model receives data from a first multimodal sensor and performs inference based on an end-to-end network architecture of bird's-eye view feature fusion and multi-task parallel decoding, outputting a locally planned trajectory. The second model receives data from a second multimodal sensor and performs inference based on a perception network architecture of 3D voxel space construction and occupancy prediction, outputting second obstacle perception information and a 3D spatial occupancy grid. The network structure of the second model differs from that of the first model.

[0077] The perception results output by the second model are transmitted to the first computing chip via a cross-chip communication link (e.g., Ethernet). The trajectory verification and replanning module deployed on the first computing chip receives the local planned trajectory output by the first model and the perception results output by the second model, verifies the first obstacle perception information based on the second obstacle perception information, and verifies the local planned trajectory based on the three-dimensional spatial occupancy grid to obtain the verification result, and performs replanning when the verification fails.

[0078] This application eliminates the common failure path of a single system paralyzing due to hardware failure, algorithm defects, or sensor malfunction by setting up two completely independent perception systems at the sensor, computing chip, and algorithm architecture levels, significantly improving the functional safety level of the autonomous driving system. Simultaneously, addressing the inherent problem of insufficient interpretability in end-to-end models, the secondary system provides an interpretable verification channel based on physical space and kinematic models, independently ensuring the safety of the trajectory output by the main system's black box, effectively compensating for the deficiencies of end-to-end models in safety verification. This application also forms a complete closed loop from perception to planning to verification and execution, ensuring that the trajectory sent to the control module simultaneously meets the dual conditions of perception consistency and trajectory safety. Furthermore, a multi-level degradation strategy prioritizes replanning when verification fails and only triggers parking when a safe path cannot be found, ensuring safety while also considering traffic efficiency and passenger comfort. The combined effect of the independent sensing system and data augmentation during the training phase ensures the system's robustness in long-tail scenarios such as severe weather and sensor failure.

[0079] Based on the aforementioned perception enhancement method based on dual-system heterogeneity, this application also provides a perception enhancement device based on dual-system heterogeneity. The following will combine... Figure 3 The device is described in detail.

[0080] Figure 3 A schematic block diagram of a perception enhancement device based on a dual-system heterogeneity according to an embodiment of this application is shown.

[0081] like Figure 3 As shown, the dual-system heterogeneous perception enhancement device 300 of this embodiment includes a data acquisition module 310, a first processing module 320, a second processing module 330, a verification module 340, and a control module 350.

[0082] The data acquisition module 310 is used to acquire first multimodal sensor data collected by the first sensing system and second multimodal sensor data collected by the second sensing system. In one embodiment, the data acquisition module 310 can be used to perform the operation S110 described above, which will not be repeated here.

[0083] The first processing module 320 is used to input the first multimodal sensor data into the first model deployed on the first computing chip, and output the local planned trajectory of the vehicle and the first obstacle perception information around the vehicle. In one embodiment, the first processing module 320 can be used to perform the operation S120 described above, which will not be repeated here.

[0084] The second processing module 330 is used to input the second multimodal sensor data into the second model deployed on the second computing chip, and output the vehicle's second obstacle perception information and the three-dimensional spatial occupancy grid of the vehicle's surrounding environment. The network structure of the second model is different from that of the first model. In one embodiment, the second processing module 330 can be used to perform the operation S130 described above, which will not be repeated here.

[0085] The verification module 340 is used to verify the first obstacle perception information based on the second obstacle perception information, and to verify the local planning trajectory based on the three-dimensional spatial occupancy grid, thereby obtaining a verification result. In one embodiment, the verification module 340 can be used to perform the operation S140 described above, which will not be repeated here.

[0086] The control module 350 is used to guide the vehicle to move based on a locally planned trajectory if the verification result is successful. In one embodiment, the verification module control module 350 can be used to perform the operation S150 described above, which will not be repeated here.

[0087] According to embodiments of this application, any multiple modules among the data acquisition module 310, the first processing module 320, the second processing module 330, the verification module 340, and the control module 350 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the data acquisition module 310, the first processing module 320, the second processing module 330, the verification module 340, and the control module 350 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the data acquisition module 310, the first processing module 320, the second processing module 330, the verification module 340, and the control module 350 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0088] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a dual-system heterogeneous perception enhancement method according to an embodiment of this application.

[0089] like Figure 4 As shown, an electronic device 400 according to an embodiment of this application includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0090] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 402 and / or RAM 403. It should be noted that the programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0091] According to embodiments of this application, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0092] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0093] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 402 and / or RAM 403 and / or one or more memories other than ROM 402 and RAM 403 described above.

[0094] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the dual-system heterogeneous perception enhancement method provided in the embodiments of this application.

[0095] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0096] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0097] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0098] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0100] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0101] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A perception enhancement method based on dual-system heterogeneity, characterized in that, The method includes: Acquire first multimodal sensor data collected by the first sensing system and second multimodal sensor data collected by the second sensing system; The first multimodal sensor data is input into the first model deployed on the first computing chip, and the local planned trajectory of the vehicle and the first obstacle perception information around the vehicle are output. The second multimodal sensor data is input into the second model deployed on the second computing chip, and the second obstacle perception information of the vehicle and the three-dimensional spatial occupancy grid of the vehicle's surrounding environment are output. The network structure of the second model is different from that of the first model. The first obstacle perception information is verified based on the second obstacle perception information, and the local planning trajectory is verified based on the three-dimensional space occupancy grid to obtain the verification result; If the verification result is successful, the vehicle is guided to move based on the local planned trajectory.

2. The method according to claim 1, characterized in that, The step of inputting the first multimodal sensor data into a first model deployed on a first computing chip and outputting the vehicle's local planned trajectory and the first obstacle perception information around the vehicle includes: Feature extraction is performed on the image data, lidar point cloud data, and millimeter-wave radar detection point data in the first multimodal sensor data respectively to obtain the first multi-scale two-dimensional image features, the first three-dimensional voxel features of the vehicle's surrounding environment, and the first radar features. The first multi-scale two-dimensional image features, the first three-dimensional voxel features, and the first radar features are spatiotemporally aligned and fused under a bird's-eye view to obtain fused bird's-eye view features. The fused bird's-eye view features of the current frame and the fused bird's-eye view features of K historical frames are fused with contextual information through a temporal self-attention mechanism to obtain temporal fused features, where K is a positive integer; The temporal fusion features are input into multiple different task decoding branches, and the local planning trajectory and the first obstacle perception information are output in parallel.

3. The method according to claim 1, characterized in that, The step of inputting the second multimodal sensor data into the second model deployed on the second computing chip and outputting the vehicle's second obstacle perception information and the three-dimensional spatial occupancy grid of the vehicle's surrounding environment includes: Based on an algorithm different from the first model, feature extraction is performed on the image data, lidar point cloud data and millimeter-wave radar detection point data in the second multimodal sensor data, respectively, to obtain the second multi-scale two-dimensional image features, the second three-dimensional sparse features of the vehicle's surrounding environment and the second radar features. The second multi-scale two-dimensional image features, the second three-dimensional sparse features of the vehicle's surrounding environment, and the second radar features are projected into a three-dimensional voxel space to construct a three-dimensional voxel feature field. Input the three-dimensional voxel feature field into the occupancy prediction branch and output the three-dimensional spatial occupancy grid. Based on the three-dimensional voxel feature field or the three-dimensional spatial occupancy grid, the second obstacle perception information is output through the target detection branch.

4. The method according to claim 1, characterized in that, The step of verifying the first obstacle perception information based on the second obstacle perception information and verifying the local planning trajectory based on the three-dimensional spatial occupancy grid to obtain the verification result includes: The target objects in the second obstacle perception information are matched with the target objects in the first obstacle perception information to obtain matching target object pairs; The relative distance and collision time of each trajectory point on the local planning trajectory relative to the predicted positions of each target object in the target object pair are verified to obtain the first verification sub-result; Verify whether each trajectory point on the local planning trajectory falls within the voxel unit marked as occupied in the three-dimensional space occupied grid to obtain the second verification result; If both the first and second verification results pass verification, the verification result is determined.

5. The method according to claim 4, characterized in that, Before verifying the first obstacle perception information based on the second obstacle perception information, verifying the local planning trajectory based on the three-dimensional spatial occupancy grid, and obtaining the verification result, the method further includes: The targets in the second obstacle perception information are spatiotemporally aligned with the targets in the first obstacle perception information; Furthermore, the three-dimensional spatial occupancy grid is spatiotemporally aligned with the local planning trajectory.

6. The method according to claim 4, characterized in that, The first verification sub-result obtained by verifying the relative distance and collision time of each trajectory point on the local planning trajectory with respect to the predicted positions of each target object in the target object pair includes: Calculate the minimum distance between the vehicle envelope at each trajectory point on the local planning trajectory and the predicted position envelope of each target object in the target object pair; Calculate the collision time at each trajectory point based on the minimum distance and the speed of each target object; If the minimum distance at each trajectory point is greater than or equal to a preset safe distance threshold, and the collision time at each trajectory point is greater than or equal to a preset collision time threshold, then the first verification result is determined to be verified successfully.

7. The method according to claim 4, characterized in that, The step of verifying whether each trajectory point on the local planning trajectory falls within the voxel unit marked as occupied in the three-dimensional space occupied grid, and obtaining the second verification sub-result, includes: Obtain the occupancy probability of the voxel unit at the corresponding position in the three-dimensional space occupancy grid for each trajectory point on the local planning trajectory; If the occupancy probability at the corresponding position of each trajectory point is less than the preset occupancy probability threshold, then the second verification result is determined to be verified as passed.

8. The method according to claim 1, characterized in that, If the verification result is a failure, the method further includes: Based on the passable space in the three-dimensional spatial grid and the predicted positions of each target object in the second obstacle perception information, a passable area is constructed. Perform a path search within the passable area to generate alternative planned trajectories; The backup planned trajectory is re-verified; If the backup planned trajectory passes the verification, the vehicle is guided to move using the backup planned trajectory.

9. A sensing enhancement device based on dual-system heterogeneity, characterized in that, The device includes: The data acquisition module is used to acquire first multimodal sensor data collected by the first sensing system and second multimodal sensor data collected by the second sensing system. The first processing module is used to input the first multimodal sensor data into the first model deployed on the first computing chip, and output the local planning trajectory of the vehicle and the first obstacle perception information around the vehicle; The second processing module is used to input the second multimodal sensor data into the second model deployed on the second computing chip, and output the second obstacle perception information of the vehicle and the three-dimensional spatial occupancy grid of the vehicle's surrounding environment. The network structure of the second model is different from that of the first model. The verification module is used to verify the first obstacle perception information based on the second obstacle perception information, and to verify the local planning trajectory based on the three-dimensional space occupancy grid, so as to obtain the verification result; The control module is used to guide the vehicle to move based on the local planned trajectory if the verification result is that the verification passes.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

Images