Driving environment perception method, device, vehicle, medium and program product

Abstract

The application provides a driving environment perception method and device, a vehicle, a medium and a program product, and relates to the technical field of automatic driving. The method comprises: obtaining an aerial view feature in the field of view of an autonomous vehicle; the aerial view feature is obtained by fusing a plurality of groups of time-sequentially adjacent target frames; the target frames are obtained based on multi-modal feature fusion; a height dimension feature is added to the aerial view feature, and the aerial view feature is extended to a three-dimensional voxel feature based on the height dimension feature; the three-dimensional voxel feature comprises a plurality of voxels, and each voxel is used to represent a three-dimensional spatial position; the three-dimensional voxel feature is input into a pre-trained semantic prediction model to obtain a semantic category corresponding to each voxel in the three-dimensional voxel feature; and the driving environment of the autonomous vehicle within a specified time period is determined based on the semantic category. The application can obtain the driving environment of the autonomous vehicle within a specified time period based on a refined semantic description, thereby achieving scene-level perception of the environment around the vehicle.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a driving environment perception method, device, vehicle, medium, and program product. Background Technology

[0002] With the continuous development of autonomous driving technology, the perception requirements of vehicles regarding their surroundings have evolved from traditional "object detection" to "scene-level understanding." The system not only needs to identify various traffic participants on the road (such as vehicles and pedestrians), but also needs to continuously and densely model the entire spatial environment to provide more comprehensive information support for downstream path planning and decision-making modules. In practical applications, autonomous driving systems face not only structurally regular, clearly defined targets (such as vehicles and pedestrians), but also a large number of irregular, weakly structured, and even boundaryless environmental elements, such as water splashes, dust, grass, and road debris. This type of environmental information has a significant impact on driving safety, but existing perception methods struggle to effectively model it. Summary of the Invention

[0003] This application provides a driving environment perception method, device, vehicle, medium, and program product to solve one or more of the above-mentioned technical problems.

[0004] In a first aspect, embodiments of this application provide a driving environment perception method applied to autonomous vehicles. The method includes: acquiring bird's-eye view features within the field of vision of the autonomous vehicle; the bird's-eye view features are obtained by fusing multiple sets of temporally adjacent target frames; the target frames are obtained by fusing multimodal features; adding height dimension features to the bird's-eye view features, and expanding the bird's-eye view features into three-dimensional voxel features based on the height dimension features; the three-dimensional voxel features include multiple voxels, which are used to represent three-dimensional spatial positions; inputting the three-dimensional voxel features into a pre-trained semantic prediction model to obtain the semantic categories corresponding to the voxels in the three-dimensional voxel features; and determining the driving environment of the autonomous vehicle within a specified time period based on the semantic categories.

[0005] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in any of the above-mentioned embodiments.

[0006] Thirdly, embodiments of this application provide a vehicle including a camera, a lidar, a millimeter-wave radar, and the aforementioned electronic devices.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in any of the above-mentioned embodiments.

[0008] Fifthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the above-described method.

[0009] Compared with related technologies, this application has the following advantages:

[0010] According to the embodiments of this application, firstly, bird's-eye view features of each traffic participant within the field of vision of the autonomous vehicle are obtained. The bird's-eye view features are obtained by fusing multiple sets of temporally adjacent target frames, thereby preserving more complete driving environment features. The target frames are obtained by fusing multimodal features, thereby combining the advantages of various modal features, ensuring better environmental adaptability, and ensuring the detection accuracy of driving environment features. Height dimension features are added to the bird's-eye view features, thereby expanding the bird's-eye view features into three-dimensional voxel features. Then, the three-dimensional voxel features are input into a pre-trained semantic prediction model to obtain the semantic category corresponding to the voxel in the three-dimensional voxel features, so as to finely divide the environment within the vehicle's field of vision into multiple voxels and generate a semantic category for each voxel. Finally, based on the fine semantic description, the driving environment of the autonomous vehicle within a specified time period is obtained, realizing scene-level perception of the environment around the vehicle.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0013] Figure 1 A flowchart of a driving environment perception method provided in an embodiment of this application is shown;

[0014] Figure 2 This illustration shows a schematic diagram of the implementation steps of a driving environment perception method provided in an embodiment of this application;

[0015] Figure 3 This illustration shows a real-world driving environment as provided in an embodiment of this application.

[0016] Figure 4 This illustration shows a LiDAR data acquisition effect diagram of a driving environment provided in an embodiment of this application;

[0017] Figure 5 The diagram illustrates the perception effect of a driving environment provided in an embodiment of this application.

[0018] Figure 6 This paper shows a structural block diagram of a driving environment sensing device provided in an embodiment of this application;

[0019] Figure 7 A block diagram of an electronic device used to implement embodiments of this application is shown. Detailed Implementation

[0020] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0021] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0022] Compared to traditional object detection-based perception methods, semantic occupancy modeling, which focuses on spatially continuous representation, can provide richer environmental information. However, the following prominent challenges remain in the process of semantic occupancy modeling for autonomous driving:

[0023] First, unstructured targets are difficult to model, and traditional detection methods lack sufficient expressive power. In real road environments, there are many targets unsuitable for representation using 3D bounding boxes, such as: dynamic non-entities (exhaust) like water splashes, dust, and exhaust fumes; natural objects (vegetation) with blurred boundaries like grass and trees; and road debris (such as stones and other objects) with irregular shapes (onroad obstacles). These targets typically lack clear geometric boundaries or stable forms, making it difficult for traditional detection methods to accurately model them, easily leading to missed or false detections, thus affecting system safety. Specifically, in the traditional occupancy grid scheme, the occupancy grid method divides the 3D space into regular grids and performs a binary judgment of "whether it is occupied" for each grid cell to achieve spatial modeling of the environment. This method has been widely used in robot navigation and early autonomous driving systems. However, in autonomous driving scenarios facing complex road environments, traditional occupancy methods still have significant shortcomings: They lack semantic expressive power; traditional occupancy methods can only determine whether a space is occupied, but cannot distinguish between different types of occupants, such as obstacles, vegetation, or dynamic non-entities (like water splashes or dust), making it difficult to support refined environmental understanding. They also lack the ability to model dynamic environments: in rainy, dusty, or complex traffic environments, there are numerous dynamic non-entity targets that exhibit continuous changes in space, making it difficult for traditional occupancy methods to model stably. Finally, they offer limited support for planning: due to the lack of semantic information, they cannot distinguish between "passable areas" and "high-risk areas," limiting their application value in path planning and decision-making modules.

[0024] Second, sparse spatial representation makes it difficult to support planning decisions: Existing 3D object detection methods typically output discrete bounding boxes, modeling only explicit targets and lacking the ability to continuously represent the entire spatial environment. Specifically, this manifests as: the inability to describe the spatial distribution of "accessible" and "impassable" areas; difficulty in distinguishing the risk levels of different areas; and the inability to provide continuous spatial constraint information for path planning. Therefore, relying solely on detection results is insufficient to meet the environmental modeling needs of autonomous driving systems. Specifically, 3D object detection methods identify and locate vehicles, pedestrians, and other targets in the environment by outputting 3D bounding boxes, and are one of the mainstream perception solutions in current autonomous driving systems. However, these methods have the following limitations in environmental modeling: sparse spatial representation: detection methods only model explicit targets and cannot cover all areas in continuous space, resulting in incomplete environmental representation. Difficulty in modeling irregular targets: objects with irregular shapes or blurred boundaries (such as grass, scattered objects) and dynamic non-entities without clear geometric structures (such as water splashes, dust) are difficult to represent using standard 3D bounding boxes. It lacks the ability to describe non-target areas: it cannot provide semantic information about background areas or free space, which limits its application in scene-level understanding.

[0025] Third, traditional occupancy modeling lacks semantic information: To compensate for the shortcomings of detection methods, some methods use occupancy grids for environmental modeling. However, traditional occupancy methods typically only determine whether a spatial unit is occupied (occupied / free), which has the following problems: it cannot distinguish between different types of occupiers (such as obstacles, vegetation, dynamic objects, etc.); it cannot express the accessibility attributes corresponding to different categories (such as compressible, non-compressible, traversable, etc.); and it has limited support for downstream planning modules. Therefore, traditional occupancy modeling still has significant shortcomings in its expressive power.

[0026] Fourth, the utilization of multimodal information is insufficient: Autonomous driving systems are typically equipped with multiple sensors (such as cameras, LiDAR, and 4D millimeter-wave radar), and different modalities have complementary advantages in perception capabilities: cameras provide rich semantic information; LiDAR provides accurate spatial geometry; and 4D millimeter-wave radar provides stable distance and velocity information and has stronger robustness under adverse weather conditions. However, existing methods still have the following shortcomings in multimodal fusion: the fusion method is relatively simple, making it difficult to fully mine cross-modal information; there is a lack of unified spatial representation (such as unified BEV modeling); and the modeling ability for dynamic environments (such as rain, fog, dust, etc.) is insufficient.

[0027] Specifically, pure vision-based methods typically rely on multi-view images and deep learning models for semantic segmentation or semantic occupancy prediction, offering advantages in semantic representation. However, in autonomous driving scenarios, these methods suffer from the following problems: Large depth estimation errors: Vision methods depend on depth estimation to map image information to 3D space, leading to significant errors at long distances or in complex scenes. Sensitivity to environmental changes: Visual features degrade severely under conditions such as low light, backlight, rain, and fog, affecting model stability. Insufficient geometric structure representation: Difficulty in accurately depicting spatial structures results in biased semantic occupancy results. Some methods rely on single sensors (such as LiDAR or cameras) for occupancy modeling. While achieving some success under specific conditions, they still have significant limitations: LiDAR solutions: Sparse point cloud data, insufficient semantic information, and limited ability for long-distance and fine-grained semantic classification. Vision solutions: Rich in semantics but lacking precise geometric structure information, resulting in insufficient spatial positioning capabilities. Millimeter-wave radar solutions: Robust in complex environments, but with low resolution, making it difficult to independently perform fine semantic modeling. Therefore, single-modal methods struggle to meet the high-precision environmental modeling requirements of autonomous driving.

[0028] To improve perception performance, existing technologies introduce multimodal fusion methods to jointly model data from cameras, LiDAR, and millimeter-wave radar. However, in semantic occupancy modeling tasks, existing multimodal methods still suffer from the following problems: The fusion methods are relatively simple: they often employ simple concatenation or weighted fusion, making it difficult to fully exploit the complementary information between different modalities. They lack a unified spatial representation: the spatial alignment of different modal data is complex, affecting the fusion effect. They provide insufficient support for semantic occupancy modeling: most methods still focus on object detection and fail to fully utilize multimodal information for continuous spatial semantic modeling.

[0029] Fifth, insufficient utilization of temporal information makes dynamic scene modeling difficult: In autonomous driving scenarios, the environment exhibits significant temporal characteristics, such as continuous movement of dynamic targets; intermittent observations due to occlusion; and changes in dynamic non-physical entities like water splashes and dust. Relying solely on single-frame information can easily lead to: unstable detection results; discontinuous dynamic region modeling; and missing information about distant targets. Therefore, effectively utilizing multi-frame temporal information for fusion is a crucial issue for improving semantic occupancy modeling capabilities.

[0030] Sixth, high real-time performance and computational efficiency requirements: Autonomous driving systems place extremely high demands on the real-time performance of perception modules, especially in high-speed driving scenarios, where perception latency directly impacts decision-making and control safety. Semantic occupancy modeling typically requires dense predictions in three-dimensional space, incurring significant computational overhead and presenting the following challenges: computational pressure from high-resolution meshes; additional computational complexity from multimodal fusion; and the impact of temporal fusion on real-time performance. Therefore, while improving modeling accuracy, designing an efficient model structure to meet real-time requirements is a key issue.

[0031] In summary, existing technologies for autonomous driving environment modeling still suffer from insufficient expressive power, lack of semantic information, inadequate utilization of multimodal approaches, and difficulty in achieving real-time performance. To address these issues, this application provides a driving environment perception method, device, vehicle, medium, and program product. By jointly utilizing multi-source information from cameras, LiDAR, and 4D millimeter-wave radar, it performs unified modeling in the BEV space and introduces a multi-frame temporal fusion mechanism to achieve continuous and dense semantic representation of the environment. This application aims to: achieve a shift from "target-level detection" to "scene-level modeling"; effectively model unstructured and dynamic non-physical targets (such as water splashes and dust); provide rich semantic category information to support downstream path planning and decision-making; improve perception robustness in long-distance, occluded, and harsh environments; and achieve high-precision semantic occupancy prediction while ensuring real-time performance. This effectively overcomes the limitations of existing technologies in environmental perception and improves the safety and reliability of autonomous driving systems in complex scenarios.

[0032] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0033] This application provides a method for perceiving the driving environment. This method is applicable to the environment perception module in assisted driving and autonomous driving systems for passenger cars and commercial vehicles (such as trucks), and is particularly suitable for autonomous driving scenarios in long-haul logistics, highways, port transportation, and complex weather conditions. This method can be deployed and run on the vehicle-side computing platform of an autonomous vehicle (e.g., an onboard industrial computer, an onboard GPU server, or a dedicated perception processing chip). Figure 1 The diagram shown is a flowchart of a driving environment perception method according to an embodiment of this application. The method may include:

[0034] Step S101: Obtain bird's-eye view features within the field of vision of the autonomous vehicle; the bird's-eye view features are obtained by fusing multiple sets of temporally adjacent target frames; the target frames are obtained by fusing multimodal features.

[0035] In this possible implementation, bird's-eye view features refer to generating a two-dimensional planar feature map, similar to an aerial view, by spatially transforming the environmental information perceived by the autonomous vehicle. This feature comprehensively reflects the layout of the vehicle's surrounding environment, aiding in path planning and obstacle detection. Bird's-eye view features are spatial feature representations formed on a two-dimensional plane (X-axis and Y-axis) after feature extraction and fusion of raw data collected by multiple sensors (such as cameras, LiDAR, and millimeter-wave radar). Each pixel or grid unit of the BEV (Bird's Eye View) feature corresponds to a location on the ground plane and contains semantic, geometric, and other information about that location.

[0036] The generation process of bird's-eye view features is based on the fusion of multiple sets of temporally adjacent target frames. Temporally adjacent means that the consecutive frame data are close in time. There is a correlation between these frames. By fusing them, the missing information of a single frame perspective can be compensated, thereby improving the stability and accuracy of spatial structure recognition.

[0037] In real-world road environments, relying on a single sensor (such as a camera or LiDAR) can easily lead to missed or false detections. In this embodiment, the target frame is obtained based on multimodal feature fusion. Multimodal features refer to data from different sensors (such as cameras, radar, and LiDAR). Fusing these features can combine visual texture information with depth and distance information, so that the generated target frame has both high-resolution image details and accurate distance and shape parameters, and can also avoid missed or false detections.

[0038] In this possible implementation, bird's-eye view features are used to present the vehicle's surrounding environment completely and intuitively in two dimensions, enabling more efficient path planning and risk assessment. Temporally adjacent frame fusion solves the problem of insufficient information in a single frame, enhancing the continuity and stability of environmental perception in dynamic scenes. Multimodal feature fusion improves the perception accuracy of the target frame, taking into account both visual details and spatial depth information, adapting to different lighting, weather, and complex traffic conditions.

[0039] Step S102: Add a height dimension feature to the bird's-eye view feature, and based on the height dimension feature, expand the bird's-eye view feature into a three-dimensional voxel feature; the three-dimensional voxel feature includes multiple voxels, and the voxels are used to characterize three-dimensional spatial positions.

[0040] In this possible implementation, the height dimension feature is used to introduce Z-axis information on top of the original two-dimensional BEV plane, representing the structure and properties of space in the vertical direction (height direction). The height dimension feature can be generated using lidar point cloud height information, millimeter-wave radar range layer information, or visual depth estimation. It can be determined based on the height of interest in the autonomous driving scenario. For example, considering the actual driving environment of autonomous vehicles, the value range of the height dimension feature can be 0-4 meters. The bird's-eye view feature includes planar resolution data; adding the height dimension feature expands the bird's-eye view feature into a three-dimensional voxel feature. The three-dimensional voxel feature is a three-dimensional mesh representation formed by expanding the two-dimensional BEV feature according to the height dimension; each smallest unit is called a "voxel." Each voxel corresponds to a specific location in three-dimensional space, which can be described as (X, Y, Z coordinates). The tensor shape of the 3D voxel feature is (B, C, H, W, Z), where C is the number of feature channels per height layer, Z is the number of height layers, i.e., the height dimension feature, B represents the batch size, and H and W represent the BEV plane resolution.

[0041] In this embodiment, two-dimensional BEV features, i.e., bird's-eye view features, are extended into three-dimensional voxel feature representations through reshape / reconstruction. This three-dimensional representation enables a continuous and fine-grained description of space, providing a data foundation for refined environmental modeling. By introducing a height dimension into the BEV features and extending them into three-dimensional voxel features, continuous spatial expression is achieved.

[0042] Step S103: Input the three-dimensional voxel features into the pre-trained semantic prediction model to obtain the semantic category corresponding to the voxel in the three-dimensional voxel features.

[0043] In this possible implementation, the three-dimensional voxel features generated in step S102 are input into a pre-trained semantic prediction model to perform semantic category discrimination for each voxel. The network structure of the pre-trained semantic prediction model may include convolutional layers and classification prediction layers. The convolutional layers are used for feature extraction based on the three-dimensional voxel features. Specifically, the convolutional layers can employ three-dimensional convolution (3D Conv) to extract features simultaneously in the X, Y, and Z spatial dimensions; or they can employ two-dimensional convolution (2D Conv) combined with a height expansion method, i.e., performing two-dimensional convolution calculations in the BEV plane (X and Y dimensions) and combining it with the height dimension (Z) for feature expansion, thereby reducing computational load while preserving spatial structural information. The classification prediction layer predicts semantic categories based on the features output by the convolutional layer, resulting in P(x, y, z, c), where x, y, z represent the positions of voxels in 3D space, c represents the semantic category, and P(x, y, z, c) represents the probability distribution of spatial location (x, y, z) belonging to semantic category c. This probability distribution can be used to generate a 3D semantic occupancy map, which not only contains the occupancy status of each spatial unit but also rich semantic information, directly providing accessibility and safety reference information for subsequent path planning, risk assessment, and other modules.

[0044] It should be noted that the pre-trained semantic prediction model can be a deep learning model trained on a large-scale dataset with 3D spatial semantic labels, and already possessing stable semantic classification capabilities. This model can predict the semantic category (e.g., road, obstacle, plant, dynamic non-entity, etc.) of each voxel based on the input 3D voxel features. The pre-trained semantic prediction model can be based on architectures such as Convolutional Neural Networks (CNN), Transformer, 3D U-Net, and BEVFormer, and supports multimodal input features. During the training phase of this pre-trained semantic prediction model, the following loss functions can be used, but are not limited to: multi-class cross-entropy loss; Focal Loss (to alleviate class imbalance); class weighting mechanism (to enhance rare class learning); and temporal consistency constraints (to improve the stability of predictions in consecutive frames).

[0045] The output of the pre-trained semantic prediction model may include: a 3D semantic occupancy grid and a BEV semantic occupancy map. The 3D semantic occupancy grid is a three-dimensional spatial representation using voxels as basic units. Each voxel corresponds to a fixed-volume spatial location and contains two main pieces of information: occupancy status: whether the spatial location is occupied by an object or substance; semantic category: what type of object occupies the spatial location, such as traffic participants, plants, buildings, barriers, dynamic non-entities (rain, fog, water splashes), etc. The 3D semantic occupancy grid covers every voxel location within the entire perception range. The BEV semantic occupancy map is a two-dimensional planar representation of the 3D semantic occupancy grid projected or aggregated along the height direction (Z-axis). Each pixel corresponds to a fixed area location on the ground plane and includes the occupancy status and main semantic category of that location. The view is a bird's-eye view (top view), with the horizontal and vertical axes corresponding to the X and Y directions of the vehicle's coordinate system. Two-dimensional BEV maps require less computation and are suitable for use in real-time path planning and local obstacle avoidance. After height information is aggregated, the system can identify passable or impassable locations on the ground more quickly.

[0046] This output not only provides the spatial occupancy status but also includes semantic category information for each location. It can be directly applied to autonomous driving path planning modules, traversable area analysis, and risk assessment and decision-making systems. Through semantic occupancy information, different target categories can correspond to different decision-making strategies. For example: barrier (isolation facilities): determined as an impassable area; vegetation: determined as a trampable area; exhaust (dynamic non-physical objects such as water splashes / dust): determined as a traversable area, but requiring speed reduction to ensure safety; onroad obstacle (road surface obstacle): determined as a high-risk area, requiring detour planning or avoidance measures. This semantic occupancy result achieves a direct mapping from 3D spatial semantic modeling to intelligent decision-making strategies, providing autonomous driving systems with refined environmental understanding and a high-safety path planning basis.

[0047] In this possible implementation, the probability distribution P(x,y,z,c) can distinguish multiple semantic categories, including roads, obstacles, plants, and dynamic non-entities (such as rainwater and dust), overcoming the shortcomings of traditional ocupancy which only provides binary information of "occupied / unoccupied". The semantic category of each voxel can be used to map accessibility (pressable, inpressable, traversable) and risk level, directly supporting the planning module in generating safe paths. With the support of multimodal input, the pre-trained model can stably identify dynamic non-entity targets such as water splashes, dust, and exhaust fumes, making up for the deficiency of traditional detection boxes in modeling such targets. Semantic prediction of the entire 3D voxel space makes environmental modeling no longer limited to sparse target boxes, but forms a dense scene-level representation, providing continuous spatial constraint information and improving the planning usability of complex scenes. Multimodal fusion and highly extended methods reduce the recognition degradation of visual single modality in harsh environments such as low visibility, backlight, rain, and snow, improving perception stability.

[0048] This application's embodiments significantly enhance the modeling capabilities for unstructured and dynamic targets: height-dimensional features enable the system to capture the vertical diffusion and distribution of dynamic non-entities such as water splashes and dust, overcoming the missed detection and false detection problems caused by traditional two-dimensional representations. Three-dimensional voxel features provide continuous occupancy information for the entire space, not only determining "whether it is occupied" but also distinguishing occupants of different heights and types to identify passable and impassable areas, supporting path planning and risk assessment.

[0049] Step S104: Based on the semantic category, perform spatial aggregation to determine the driving environment of the autonomous vehicle within a specified time period.

[0050] In this possible implementation, the driving environment refers to a comprehensive description of the spatial environment in which the vehicle is currently and will be passing through for a certain period of time in an autonomous driving scenario. This includes information such as road structure, distribution of traffic participants, locations of obstacles and unstructured targets, passable areas, risk levels, and dynamic trends. The specified time period is a time window preset or calculated based on the requirements of the autonomous driving task, such as 1 second, 2 seconds, or 5 seconds in the future, used to predict and assess changes in the driving environment. The time period can be set by the planning module or dynamically adjusted to adapt to vehicle speed and environmental complexity.

[0051] Spatial aggregation refers to the comprehensive processing of the semantic categories of all voxels in the 3D semantic occupancy map obtained in step S103, integrating the semantic information of each location according to spatial location and correlation to form a global scene description. The spatial aggregation process may include: attribute mapping: mapping semantic categories to accessibility (crossable, compressible, impassable), risk level (low, medium, high, lethal), and dynamic state (static, moving, changing trend); region aggregation: merging adjacent or related voxels in space according to category and attribute to form higher-level region-level information (road area, obstacle area, dynamic non-entity diffusion area, etc.); structural correlation: using proximity relationships and road network structure to perform semantic structuring processing on the aggregation results, such as marking driving lanes, parking areas, and no-entry areas.

[0052] In this possible implementation, a global scene model is formed by aggregating semantic categories, enabling the environment description to cover the entire continuous space and avoiding perception blind spots caused by sparse detection results. The aggregation process considers the spatial diffusion and temporal changes of dynamic non-physical elements (rain, dust, etc.), allowing the system to generate stable and reliable driving environment descriptions even in harsh environments. Driving environment prediction within a specified time period allows vehicles to anticipate potential risks, such as impassable areas ahead or approaching dynamic obstacles, thereby improving safety.

[0053] This application provides a method for perceiving the driving environment. First, bird's-eye view features of each traffic participant within the field of vision of an autonomous vehicle are acquired. The bird's-eye view features are obtained by fusing multiple sets of temporally adjacent target frames, thereby preserving more complete driving environment features. The target frames are obtained by fusing multimodal features, thereby combining the advantages of various modal features, ensuring better environmental adaptability, and ensuring the detection accuracy of driving environment features. Height dimension features are added to the bird's-eye view features, thereby expanding the bird's-eye view features into three-dimensional voxel features. Then, the three-dimensional voxel features are input into a pre-trained semantic prediction model to obtain the semantic category corresponding to the voxel in the three-dimensional voxel features, so as to finely divide the environment within the vehicle's field of vision into multiple voxels and generate a semantic category for each voxel. Finally, based on the fine semantic description, the driving environment of the autonomous vehicle within a specified time period is obtained, realizing scene-level perception of the environment around the vehicle.

[0054] In one possible implementation, expanding the bird's-eye view features into three-dimensional voxel features based on the height dimension features can be performed according to the following steps:

[0055] Based on the height dimension feature, a three-dimensional voxel space corresponding to the bird's-eye view feature is determined; the three-dimensional voxel space includes: batch size data, number of feature channels, planar resolution data, and height dimension feature; the ratio of the number of feature channels to the height dimension feature is used as the updated number of feature channels; based on the batch size data, the updated number of feature channels, the planar resolution data, and the height dimension feature, the three-dimensional voxel feature in the three-dimensional voxel space is determined.

[0056] In this possible implementation, the batch size (B) is the number of samples processed at one time; the number of feature channels (C) represents the feature dimension carried by each spatial location; the planar resolution data (Height, H and Width, W) represents the grid resolution of the BEV plane in the horizontal direction; and the height dimension feature (Z) represents the layer resolution in the vertical direction. In one possible implementation, the bird's-eye view features can be represented as (B, C, H, W). After adding the height dimension feature, the tensor structure in the three-dimensional voxel space can be represented as (B, C, H, W, Z). In the tensor structure, the original channel dimension C is split into the number of channels C / Z on each height slice (Z layer); forming a three-dimensional feature structure of (B, C / Z, H, W, Z), thus obtaining the three-dimensional voxel features.

[0057] In this possible implementation, by introducing a height dimension and adjusting the number of feature channels, an effective transformation from two-dimensional BEV features to three-dimensional voxel features is achieved: combining planar resolution with the height dimension forms a continuous and regular three-dimensional mesh representation in space, enabling the system to acquire fine-grained information in the vertical direction. Adjusting the number of feature channels according to the height layer maintains independent and balanced semantic and geometric features for each height slice, improving the model's efficiency in three-dimensional convolution and height-related modeling. Based on the three-dimensional voxel space, it can cover the full spatial distribution of unstructured targets (such as plants and debris) and dynamic non-physical objects (rainwater, dust), meeting the environmental modeling needs of autonomous driving in complex scenarios.

[0058] In one possible implementation, spatial aggregation based on the semantic categories to determine the driving environment of the autonomous vehicle within a specified time period can be achieved by the following steps: determining multiple voxels corresponding to the same target detection result based on the semantic categories corresponding to the voxels; and determining the driving environment of the autonomous vehicle based on the target detection results within the specified time period.

[0059] In this possible implementation, multiple voxels belonging to the same target detection result are identified based on the semantic category corresponding to each voxel; see [link to relevant documentation]. Figure 2 The diagram shown illustrates the implementation steps of the driving environment perception method. Figure 2 The output of the semantic occupancy grid is shown in the figure. Figure 2 The images in the bottom row show voxels of different colors; multiple voxels of the same color can be used to characterize a target detection result. Figure 2 The target detection results include isolation facilities, vegetation, buildings, fences, road obstacles, etc. These results can be fused from multimodal sensors (cameras, LiDAR, millimeter-wave radar), containing information such as the target's spatial location, category, and motion state. The aggregation process can merge multiple voxels belonging to the same target to form a complete multi-voxel representation of the target in three-dimensional space, reducing redundancy and improving spatial consistency. Based on the target detection results within a specified time period (including the predicted positions and states of static and dynamic targets), combined with the vehicle's current position, speed, and direction of travel, the driving environment for that time period is determined. Driving environment determination may include: identification and marking of passable areas; location of high-risk or impassable areas; trajectory prediction and risk assessment of dynamic targets; and prediction of environmental condition change trends (spatial diffusion prediction of dynamic non-physical elements such as rain, snow, dust, and fog). The driving environment output can use a time-stamped three-dimensional semantic occupancy map or structured environmental data (such as road topology, risk zone coordinates, predicted trajectories, etc.) for direct use by the planning and decision-making modules. The output data has spatial continuity and temporal prediction capabilities, enabling autonomous driving systems to provide high-precision environmental models that can be directly used for path planning in complex scenarios.

[0060] Figure 3 This image depicts a real-world driving environment. Figure 4 Demonstrates the use of lidar to Figure 3 The display effect image generated after capturing real-world images. Figure 5 Showing the Figure 3 Mid-scale real-world image, a perception effect diagram generated using this application. See also... Figure 5 The diagram shown illustrates the perceived effect of the driving environment. Figure 5 The image shows the driving environment output after perceiving the real-world environment. The entire driving environment includes various color regions, representing targets with different semantic categories in the environment. For example, a green region, composed of multiple voxels displayed in green, can be used to represent road targets; a pink region, composed of multiple voxels displayed in pink, is used to represent building targets; and a blue region, composed of multiple voxels displayed in blue, is used to represent plant targets.

[0061] In one possible implementation, determining the driving environment of the autonomous vehicle based on the target detection results within a specified time period can be achieved as follows: determining the positional relationship between the target detection results, determining a passable area based on the positional relationship; determining scene information included in the driving environment based on the passable area; the scene information includes at least one or more of roads, pedestrian zones, construction zones, and risk zones.

[0062] In this possible implementation, positional relationship determination involves analyzing the spatial positional relationships of target detection results within a specified time period. Positional relationships may include horizontal distance, longitudinal distance, relative speed, direction, and relationship with the road structure between targets. Combining the vehicle's current position, driving direction, and speed, the precise positional distribution of each target in three-dimensional space is obtained. Based on these positional relationships, the area suitable for safe vehicle travel is calculated according to space occupancy and passability rules. The delineation of passable areas may consider: the location of road boundaries and driving lanes; the avoidance distance between static obstacles and dynamic targets; the passability attributes of unstructured targets (grass, scattered objects, etc.); and the traversability and visual impact of dynamic non-physical objects (dust, fog, rain, etc.). Based on the passable area and overall spatial distribution, scene information encompassed by the driving environment is determined. Scene information includes at least one or more of the following: road: an area suitable for normal vehicle travel; pedestrian zone: an area where pedestrians are concentrated; construction zone: an area where the road is temporarily closed or changed; risk zone: an area with high-risk obstacles or impassable areas. Scene information can be accompanied by time tags to mark the changing trends within a specified time period, thereby providing a predictive basis for route planning and risk avoidance. The driving environment model is output as structured data or a 3D semantic occupancy map; the planning and decision-making modules can directly use this environment model for route generation, dynamic avoidance strategy formulation, and risk warning.

[0063] In this possible implementation, passable areas are calculated based on real-time location relationships, accurately eliminating impassable and high-risk areas, providing directly usable spatial constraints for autonomous driving path planning. The driving environment includes not only roads and obstacles, but also scene information such as pedestrian zones, construction zones, and risk zones, supporting more refined planning strategies and priority management. The output already includes passable areas and scene information, reducing the derivation calculations of the planning module and improving the overall system real-time performance.

[0064] In one possible implementation, the driving environment of the autonomous vehicle is determined based on the target detection results within a specified time period, which can be carried out in the following manner: based on preset semantic and object hardness mapping information, it is determined whether the target detection result is a crushable object; based on the determination result, it is determined that the driving environment includes crushable areas or non-crushable areas.

[0065] In this possible implementation, during the system pre-setting stage, a mapping relationship between semantic categories and object hardness attributes is established. This mapping relationship describes the physical hardness and crushability corresponding to different semantic categories. For example: static obstacles (such as guardrails, walls, and trees): hard objects, cannot be crushed; dynamic traffic participants (such as pedestrians and vehicles): cannot be crushed; unstructured loose materials (such as scattered cardboard boxes and plastic bags): soft objects, crushable; dynamic non-physical objects (such as water splashes, snow, and dust): traversable and crushable; grass and low-lying plants: crushable or partially crushable (depending on rules and risk levels). Based on the semantic category information of the target detection results, the mapping relationship table is retrieved to determine whether the target is a crushable object. During the determination process, multimodal perception data (vision, lidar, millimeter-wave radar) can be combined to obtain the object's material, shape, and stability characteristics, further improving the accuracy of crushability determination. Based on the determination results, the spatial location of the target is marked as a crushable or non-crushable area. Based on the trampling markings and combined with target location predictions within a specified time period, the complete driving environment status is determined, including traversable areas, trampling areas, and areas that are difficult to traverse or cannot be trampled.

[0066] In this possible implementation, a semantic mapping relationship between object hardness is introduced, allowing drivability judgment to no longer rely solely on spatial geometric information but incorporate physical properties, making it more adaptable to complex road environments. In emergency obstacle avoidance scenarios, crushability information can be used to allow vehicles to pass over soft or traversable objects, thereby avoiding safety risks caused by excessive avoidance. Crustable markings are directly embedded in driving environment data, enabling the path planning module to generate the optimal route without additional calculations, improving system real-time performance. The driving environment is no longer limited to "drivable / impassable" categories but includes granular crushability attributes, providing richer planning reference information for the autonomous driving system.

[0067] In one possible implementation, the driving environment of the autonomous vehicle is determined based on the target detection results within a specified time period, which can be achieved as follows: based on preset semantic and physical state mapping information, it is determined whether the target detection result is a traversable object; based on the determination result, it is determined whether the driving environment includes traversable or intraversable areas.

[0068] In this possible implementation, during the system's preset phase, a mapping relationship between semantic categories and physical state attributes is established to determine the spatial traversability of different target categories. This mapping relationship may include: solid obstacles (such as guardrails, walls, trees, vehicles, and pedestrians): impassable; liquids or loose substances (such as stagnant water, mud, and grass): traversable or partially traversable (subject to risk level labeling); gases or dynamic non-physical substances (such as water splashes, rain, snow, dust, and exhaust fumes): traversable, but may affect sensor perception stability. Based on the semantic category information contained in the target detection results, the mapping relationship table is retrieved to determine whether the target is a traversable object. During the traversability determination process, multimodal perception data (vision, lidar, and millimeter-wave radar) are combined to obtain the target's material, shape, and stability information, thereby improving the accuracy of traversability determination.

[0069] The determination results are mapped to a 3D semantic occupancy map or structured spatial data: Traversable areas: Vehicles can pass through without causing physical obstruction, but may need to reduce speed or activate specific sensor modes; Inaccessible areas: Vehicles must avoid or detour. Within a specified time period, the distribution of traversable and inaccessible areas is dynamically updated by combining the target's predicted trajectory and environmental change trends. Based on the traversability marking results, a complete driving environment model is determined, including passable, traversable, inaccessible, and high-risk areas. The results are output to the planning and decision-making module, supporting real-time path adjustment and risk avoidance.

[0070] In this possible implementation, not only are passable and impassable distinguished, but also traversability attributes are further subdivided, providing a more refined reference for planning strategies. In complex scenarios, traversability assessment helps vehicles avoid truly impassable areas while allowing passage through low-risk traversable objects, reducing unnecessary detours and sudden stops.

[0071] In one possible implementation, the voxel is a cube; the side length of the cube is a preset length; the preset length is determined based on the height dimension feature and the preset number of voxels.

[0072] In this possible implementation, the voxels are regular cubes, each cube unit representing a fixed-size volume region in three-dimensional space; the cube shape ensures that the calculation of spatial distribution and the determination of proximity relationships are more accurate during three-dimensional convolution and spatial aggregation, and the data structure is unified.

[0073] The cube has a preset side length L, used to control the spatial resolution of the 3D mesh. This preset length can be determined in the following ways: based on the height dimension feature Z: dividing the vertical height range of the space into Z voxel layers; and based on the preset number of voxels N: ensuring that the volume of a unit voxel matches the sampling requirements throughout the entire 3D voxel space. Example calculation formula: L = Spatial height range / Z. The voxel side length can be dynamically adjusted according to the needs of different autonomous driving scenarios: for example, in highway scenarios, larger voxels are selected to reduce computational costs; in urban road or complex obstacle scenarios, smaller voxels are selected to improve fine-grained modeling accuracy. In one possible implementation, the preset length can be determined within the range of 0.1 meters to 0.3 meters; for example, the cube's side length is set to 0.2 meters.

[0074] In this possible implementation, the side length is controlled by a preset length, which can balance accuracy and computational efficiency, adapting to different road scenarios and task requirements. The 3D voxels are cubic in shape, ensuring consistency of the spatial coordinate system during multimodal data fusion and reducing alignment errors caused by mesh deformation. Regular cubes facilitate feature operations in 3D convolution (3D Conv), significantly improving computational efficiency in spatial aggregation and semantic prediction. The solution provided in this application adopts BEV unified modeling, 3D voxel spatial expansion, multimodal fusion, and temporal optimization, which can achieve parallel computing on onboard GPUs, FPGAs, dedicated AI chips, and other hardware to meet real-time processing requirements. The semantic prediction model and spatial aggregation module use efficient convolution, 2D + height expansion methods, or hierarchical 3D Conv, with computational complexity significantly lower than traditional voxel 3D occupancy prediction methods, reducing storage and computing power requirements. It can also run on high-resolution meshes under limited computing resources on the vehicle side. Autonomous vehicles are equipped with multimodal sensors such as cameras, LiDAR, and 4D millimeter-wave radar. The input information of this solution is collected in real time by the vehicle-side hardware, avoiding reliance on cloud data. The core model can be obtained through pre-training and fine-tuned online after deployment using a small amount of data collected from the vehicle, without needing to access a large training dataset at runtime. Therefore, it can be implemented on the vehicle side.

[0075] In one possible implementation, the semantic category includes any of the following categories: traffic participants, plants, weather conditions, barriers, buildings, and road obstacles.

[0076] In this possible implementation, traffic participants, including vehicles, pedestrians, cyclists, and animals, are active on or around the road and may affect the vehicle's path. Plants, including grasses, shrubs, and trees, are typically static and irregularly shaped targets that may appear at road boundaries, median strips, or non-traffic areas. Weather conditions, including rain, snow, fog, and dust, are dynamic, non-physical targets that can affect sensor perception, but some are traversable or run over. Isolation facilities, including guardrails, curbs, road barriers, and fencing, are used to separate different traffic flows or protect lane safety and are generally non-traversable or non-run over. Buildings, including residential, commercial, and bridge structures, are fixed structures with clearly defined geometric boundaries and are non-traversable or non-run over. Road obstacles, including stones, metal objects, construction equipment, and debris, vary in shape and hardness and may be run over or require avoidance.

[0077] In this possible implementation, during the semantic category prediction and mesh generation process, the location of each voxel in the 3D voxel space is classified and predicted. Specifically, firstly, a pre-trained semantic prediction model is used to calculate the probability distribution of each semantic category corresponding to each voxel; then, the final semantic category of the voxel is determined by the argmax method. Based on the prediction results of all voxels, a complete 3D semantic occupancy mesh covering the entire platform area is generated. This mesh not only includes the spatial occupancy status but also retains the semantic category information of each location. The semantic categories include, but are not limited to, the following types: cone; barrier; vegetation; exhaust (dynamic non-entities such as water splashes / dust); building; fence; and onroad obstacle. This 3D semantic occupancy mesh can directly provide fine-grained environmental semantic information for downstream tasks such as path planning, accessibility analysis, and risk assessment, thereby achieving comprehensive modeling and understanding of complex driving scenarios. This application's embodiment covers static targets, dynamic targets, unstructured targets, and dynamic non-entities in autonomous driving scenarios, improving the coverage and completeness of environmental perception.

[0078] In one possible implementation, the following steps may also be performed: generating vehicle control instructions based on the driving environment; the vehicle control instructions are used to cause the autonomous vehicle to perform any of the following operations: path planning, traversable area analysis, risk assessment, risk decision-making, behavior prediction, and behavior control.

[0079] In this possible implementation, the driving environment includes information such as road layout, drivable areas, risk areas, locations and predicted trajectories of traffic participants, and dynamic non-physical spatial distribution. Control command types include: Path planning: generating the optimal route within the drivable area, considering risk level and traffic priority. Drivable area analysis: dynamically adjusting the boundaries of the drivable area to ensure safety. Risk assessment: calculating indicators such as potential collision probability and environmental change risks. Risk decision-making: triggering avoidance, emergency braking, or safe deceleration strategies. Behavior prediction: predicting the future actions of other traffic participants and making proactive decisions. Behavior control: controlling specific actions such as steering, acceleration, deceleration, braking, and lane keeping. The generated control commands are executed in real time by the vehicle's execution system (including a path tracking module, power control module, braking control module, etc.) to ensure the vehicle operates safely and efficiently in the specified driving environment.

[0080] In this possible implementation, control commands are based on scene-level driving environment, not just sparse target detection results, ensuring that decisions take into account accessibility and risk level information. It can respond in real time to environmental changes such as changes in traffic participant behavior, dynamic non-physical diffusion, and movement of risk zones, improving the timeliness and accuracy of control strategies. Multimodal input and dynamic prediction ensure that control commands can be generated stably even in low visibility, high sensor noise, and complex environments.

[0081] In one possible implementation, the multimodal features include at least one or more of the following data: first point cloud data acquired by millimeter-wave radar, second point cloud data acquired by lidar, and image data acquired by camera.

[0082] In this possible implementation, the first point cloud data from the millimeter-wave radar, including point data and velocity data, can be used to characterize information such as distance, angle, altitude, and velocity. The second point cloud data from the lidar includes point data and reflectivity data. A camera acquires multi-view camera data, and after preprocessing this data such as correction and normalization, image data acquired by the camera is obtained.

[0083] In this possible implementation, the image data acquired by the camera can provide rich semantic information, the LiDAR can provide high-precision spatial structure, and the 4D millimeter-wave radar can provide stable distance and velocity information. Through multimodal fusion, a complementary fusion strategy of different sensor information in the BEV space fully leverages the complementary advantages of each sensor in ultra-long vehicle detection, effectively improving the system's performance in the following scenarios: low light, backlight, and other visually degraded environments; adverse weather conditions such as rain and fog; and long-distance and partially occluded scenarios. The target detection method based on three-modal collaborative fusion in this application embodiment can improve robustness in complex environments.

[0084] In one possible implementation, obtaining bird's-eye view features within the field of vision of an autonomous vehicle can be achieved as follows: feature extraction is performed based on the multimodal features to obtain multiple sets of feature extraction results; the multiple sets of feature extraction results are then subjected to viewpoint transformation to obtain multiple sets of data to be fused in the autonomous vehicle's own coordinate system; the weights of the multiple sets of data to be fused are determined respectively, and feature fusion is performed on the multiple sets of data to be fused according to the weights to obtain a target frame; the bird's-eye view features are obtained by fusing multiple sets of temporally adjacent target frames.

[0085] In this possible implementation, the multimodal sensors of the autonomous driving system include LiDAR, millimeter-wave radar, and cameras, whose output data have different spatial resolutions, perception dimensions, and information types. The following section combines... Figure 2 The steps for extracting features based on the multimodal features to obtain multiple sets of feature extraction results are described. See [link to documentation]. Figure 2 The diagram illustrates the implementation steps of a driving environment perception method. For LiDAR point clouds and 4D millimeter-wave radar, the original 3D point cloud data is divided into pillars (pillar voxels, which are 3D data pillars in the BEV grid used to encode point cloud information). That is, a grid is divided on the BEV plane, and the point clouds in the vertical direction are aggregated within each grid to form pillar data units. Using a PointPillars-based encoding method (a neural network method that encodes point cloud pillars into high-dimensional vectors and directly generates BEV features), the point cloud within each pillar is converted into a high-dimensional vector representation and projected onto the BEV feature map. The following feature extraction results are obtained: LiDAR BEV features: possessing high-precision distance and shape information, suitable for detecting targets with clear structures; 4D millimeter-wave radar BEV features: containing point cloud coordinates along with velocity information (Doppler velocity measurement), suitable for identifying moving targets and distant targets.

[0086] For visual feature extraction and BEV mapping, a convolutional neural network is used to extract multi-scale image features, obtaining a two-dimensional multi-scale feature map. The View Transform method based on FastBEV (a viewpoint transformation method that maps perspective camera viewpoint features to the BEV spatial coordinate system) maps image features from the perspective coordinate system to the BEV plane coordinate system, achieving spatial consistency of visual information. The resulting extracted features are visual BEV features, containing rich texture and color information.

[0087] Next, a perspective transformation is performed, that is, the multiple sets of extracted feature results are transformed to obtain multiple sets of data to be fused in the autonomous vehicle's own coordinate system. Specifically, this can be achieved as follows: Based on extrinsic parameter calibration, the two types of point cloud data can be uniformly transformed to the autonomous vehicle's own coordinate system (ego coordinate system, a fixed coordinate system with the vehicle center as the origin, the forward direction as the X-axis, and the left direction as the Y-axis), ensuring spatial alignment consistency and providing a foundation for subsequent BEV (Bird's EyeView) modeling; based on camera extrinsic and intrinsic parameters, a mapping relationship is established between the image coordinate system of the image data acquired by the camera and the autonomous vehicle's own coordinate system. Furthermore, the above three types of BEV features are respectively transformed to the autonomous vehicle's own coordinate system (a fixed coordinate system with the vehicle center as the origin), ensuring that the data of different modalities are aligned at the same spatial scale. Three sets of data to be fused are obtained: the first set: LiDAR BEV feature data; the second set: 4D millimeter-wave radar BEV feature data; the third set: visual BEV feature data. In this possible implementation, different modalities are uniformly mapped to the BEV space, avoiding the complex calculations of traditional "pixel-level alignment" or "point-level projection".

[0088] Next, multimodal BEV feature fusion is performed, which involves determining the weights of the multiple sets of data to be fused, and then fusing the features of these multiple sets of data according to the weights to obtain the target frame. Specifically, this can be achieved through the following steps: determining the weights of the multiple sets of data to be fused, which can be dynamically adjusted based on factors such as sensor signal-to-noise ratio, distance range, occlusion, weather conditions, and detection task type. In one possible implementation, a convolutional multi-scale fusion network can be used to fusing features to obtain the target frame: three types of BEV features (LiDAR / millimeter-wave radar / vision) are input into the fusion module, and channel alignment is performed on the different modal features (ensuring consistent feature dimensions); layer-by-layer feature fusion is performed, such as a combination of concat + conv or weighted fusion based on an attention-like mechanism. The resulting fused target frame is a comprehensive perception result of a frame in the time series. In this possible implementation, fusion is performed in the BEV space, resulting in higher computational efficiency and a clearer structure.

[0089] Next, multi-frame temporal fusion is performed, which can be implemented based on an attention mechanism. That is, the bird's-eye view features are obtained by fusing multiple sets of temporally adjacent target frames. Specifically, this can be achieved through the following steps: Multiple sets of adjacent target frames are obtained from the historical time series, and temporal alignment is performed based on the vehicle's pose information (position + orientation) to eliminate feature offsets caused by vehicle motion. An attention-based temporal fusion module is used to weight the features of different time frames: higher stability weights are given to distant targets; for occluded or temporarily disappeared targets, the detection results are supplemented using historical frame information. This process outputs the enhanced temporal BEV features (i.e., the final bird's-eye view features).

[0090] In this possible implementation, multimodal feature extraction utilizes the precise distance of LiDAR, the long-range speed perception of millimeter-wave radar, and the texture and color information of vision to achieve complementary perception and improve stability under different lighting and weather conditions. Unified viewpoint transformation (automotive coordinate system alignment) eliminates parallax and scale differences between different sensors, providing a consistent spatial reference for subsequent fusion. Weighted fusion dynamically adjusts the importance of different modalities according to the environment, enhancing the system's adaptability. Multi-frame temporal fusion uses historical frame information to compensate for occluded or temporarily lost targets, enhancing detection continuity and robustness. The final BEV output integrates multi-source information from a unified top-down perspective, providing strong modeling capabilities for dynamic environments (such as water splashes and dust).

[0091] This application provides a vehicle environment perception method. Addressing the problem that traditional target detection methods can only output discrete bounding boxes and struggle to represent continuous space, this application divides the 3D space into a regular voxel grid; performs semantic category prediction on each voxel, rather than just determining occupancy status; and constructs a unified modeling framework from "occupancy" to "semantic occupancy." This method enables continuous semantic representation of the entire environment, overcoming the sparse representation limitations of traditional detection methods. For dynamic non-physical targets such as water splashes, dust, and exhaust fumes, as well as irregular structural targets such as plants and debris, which are difficult to represent with traditional 3D bounding boxes, this application utilizes a semantic occupancy grid to model the spatial distribution of these targets; introduces specialized semantic categories (such as exhaust, vegetation, and onroad obstacle); and characterizes their spatial extent and dynamic change characteristics through continuous voxel representation. This method effectively solves the problem that traditional detection methods cannot model irregular targets. This application not only performs semantic classification but also couples semantic information with autonomous driving planning requirements, proposing that different semantic categories correspond to different traffic attributes (such as passable, impassable, compromising, and requiring avoidance); it directly provides constraint information for path planning based on a semantic occupancy grid; and it constructs a semantic bridge between perception results and the decision-making module. This method achieves a direct mapping from "perception output" to "decision input".

[0092] The embodiments of this application have the following technical effects: (1) Advantage of unified semantic occupancy modeling. This application models the space through semantic occupancy, realizing: the transformation from discrete target detection to continuous space expression; effective modeling of irregular and dynamic targets; and providing more complete environmental semantic information. (2) Advantage of unified multimodal BEV expression. Different modalities are uniformly mapped to the BEV space; the complexity of cross-modal alignment is reduced; and the computational efficiency and fusion effect are improved. (3) Improved dynamic non-entity modeling capability. For dynamic non-entities such as water splashes and dust: traditional detection methods are difficult to express; this application uses semantic occupancy for continuous modeling, which significantly improves the expression capability.

[0093] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a vehicle environment perception device. For example... Figure 6 The diagram shown is a structural block diagram of a driving environment sensing device according to an embodiment of this application. The device may include:

[0094] The acquisition module 601 is used to acquire bird's-eye view features within the field of vision of the autonomous vehicle; the bird's-eye view features are obtained by fusing multiple sets of temporally adjacent target frames; the target frames are obtained by fusing multimodal features; the feature module 602 is used to add height dimension features to the bird's-eye view features, and expand the bird's-eye view features into three-dimensional voxel features based on the height dimension features; the three-dimensional voxel features include multiple voxels, and the voxels are used to represent three-dimensional spatial positions; the semantic module 603 is used to input the three-dimensional voxel features into a pre-trained semantic prediction model to obtain the semantic categories corresponding to the voxels in the three-dimensional voxel features; the aggregation module 604 is used to perform spatial aggregation based on the semantic categories to determine the driving environment of the autonomous vehicle within a specified time period.

[0095] According to the embodiments of this application, firstly, bird's-eye view features of each traffic participant within the field of vision of the autonomous vehicle are obtained. The bird's-eye view features are obtained by fusing multiple sets of temporally adjacent target frames, thereby preserving more complete driving environment features. The target frames are obtained by fusing multimodal features, thereby combining the advantages of various modal features, ensuring better environmental adaptability, and ensuring the detection accuracy of driving environment features. Height dimension features are added to the bird's-eye view features, thereby expanding the bird's-eye view features into three-dimensional voxel features. Then, the three-dimensional voxel features are input into a pre-trained semantic prediction model to obtain the semantic category corresponding to the voxel in the three-dimensional voxel features, so as to finely divide the environment within the vehicle's field of vision into multiple voxels and generate a semantic category for each voxel. Finally, based on the refined semantic description, the driving environment of the autonomous vehicle within a specified time period is obtained, realizing scene-level perception of the environment around the vehicle.

[0096] In one possible implementation, expanding the bird's-eye view feature into a three-dimensional voxel feature based on the height dimension feature includes: determining the three-dimensional voxel space corresponding to the bird's-eye view feature based on the height dimension feature; the three-dimensional voxel space includes: batch size data, number of feature channels, planar resolution data, and height dimension feature; using the ratio of the number of feature channels to the height dimension feature as the updated number of feature channels; and determining the three-dimensional voxel feature in the three-dimensional voxel space based on the batch size data, the updated number of feature channels, the planar resolution data, and the height dimension feature.

[0097] In one possible implementation, spatial aggregation based on the semantic categories is used to determine the driving environment of the autonomous vehicle within a specified time period, including: determining multiple voxels corresponding to the same target detection result according to the semantic categories corresponding to the voxels; and determining the driving environment of the autonomous vehicle based on the target detection results within the specified time period.

[0098] In one possible implementation, determining the driving environment of the autonomous vehicle based on the target detection results within a specified time period includes: determining the positional relationship between the target detection results; determining a passable area based on the positional relationship; and determining scene information included in the driving environment based on the passable area. The scene information includes at least one or more of roads, pedestrian zones, construction zones, and risk zones.

[0099] In one possible implementation, determining the driving environment of the autonomous vehicle based on the target detection results within a specified time period includes: determining whether the target detection result is a crushable object based on preset semantic and object hardness mapping information; and determining whether the driving environment includes a crushable area or an inclineable area based on the determination result.

[0100] In one possible implementation, determining the driving environment of the autonomous vehicle based on the target detection results within a specified time period includes: determining whether the target detection result is a traversable object based on preset semantic and physical state mapping information; and determining whether the driving environment includes traversable or intraversable areas based on the determination result.

[0101] In one possible implementation, the voxel is a cube; the side length of the cube is a preset length; the preset length is determined based on the height dimension feature and the preset number of voxels.

[0102] In one possible implementation, the semantic category includes any of the following categories: traffic participants, plants, weather conditions, barriers, buildings, and road obstacles.

[0103] In one possible implementation, the device further includes a control module for: generating vehicle control commands based on the driving environment; the vehicle control commands are used to cause the autonomous vehicle to perform any of the following operations: path planning, navigable area analysis, risk assessment, risk decision-making, behavior prediction, and behavior control.

[0104] In one possible implementation, the multimodal features include at least one or more of the following data: first point cloud data acquired by millimeter-wave radar, second point cloud data acquired by lidar, and image data acquired by camera.

[0105] In one possible implementation, obtaining bird's-eye view features within the field of vision of an autonomous vehicle includes: performing feature extraction based on the multimodal features to obtain multiple sets of feature extraction results; performing viewpoint transformation on the multiple sets of feature extraction results to obtain multiple sets of data to be fused in the autonomous vehicle's own coordinate system; determining the weights of the multiple sets of data to be fused, and performing feature fusion on the multiple sets of data to be fused according to the weights to obtain a target frame; and fusing multiple sets of temporally adjacent target frames to obtain the bird's-eye view features.

[0106] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0107] This application also provides a vehicle, including a camera, a lidar, a millimeter-wave radar, and the aforementioned electronic devices.

[0108] Figure 7 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 7 As shown, the electronic device includes a memory 701 and a processor 702. The memory 701 stores a computer program that can run on the processor 702. When the processor 702 executes the computer program, it implements the method described in the above embodiments. The number of memories 701 and processors 702 can be one or more.

[0109] The electronic device also includes:

[0110] The communication interface 703 is used to communicate with external devices and perform data exchange and transmission.

[0111] If the memory 701, processor 702, and communication interface 703 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0112] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0113] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0114] This application provides a computer program product, wherein the computer program product includes a computer program, which, when executed by a processor, implements the method provided in this application embodiment.

[0115] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0116] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0117] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0118] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0119] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0120] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0121] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0122] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0123] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0124] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0125] It should be noted that the data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0126] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0127] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A driving environment perception method, applied to autonomous vehicles, characterized in that, The method includes: The system acquires bird's-eye view features within the field of vision of an autonomous vehicle; these features are obtained by fusing multiple sets of temporally adjacent target frames; the target frames are obtained by fusing multimodal features; the multimodal features include: first point cloud data acquired by millimeter-wave radar, second point cloud data acquired by lidar, and image data acquired by a camera; An elevation dimension feature is added to the bird's-eye view feature, and based on the elevation dimension feature, the bird's-eye view feature is expanded into a three-dimensional voxel feature; the three-dimensional voxel feature includes multiple voxels, and the voxels are used to characterize three-dimensional spatial positions. The three-dimensional voxel features are input into a pre-trained semantic prediction model to obtain the semantic category corresponding to the voxel in the three-dimensional voxel features; the three-dimensional voxel features are used to distinguish occupies of different heights and types. Spatial aggregation is performed based on the semantic categories to determine the driving environment of the autonomous vehicle within a specified time period; Obtaining bird's-eye view features within the field of vision of an autonomous vehicle includes: performing feature extraction based on the multimodal features to obtain multiple sets of feature extraction results; performing viewpoint transformation on the multiple sets of feature extraction results to obtain multiple sets of data to be fused in the autonomous vehicle's coordinate system; determining the weights of the multiple sets of data to be fused; performing feature fusion on the multiple sets of data to be fused according to the weights to obtain a target frame; and fusing multiple temporally adjacent target frames to obtain the bird's-eye view features. Based on the semantic categories, spatial aggregation is performed to determine the driving environment of the autonomous vehicle within a specified time period, including: determining multiple voxels corresponding to the same target detection result according to the semantic categories corresponding to the voxels; and determining the driving environment of the autonomous vehicle based on the target detection results within the specified time period. Based on the target detection results within a specified time period, the driving environment of the autonomous vehicle is determined, including: determining whether the target detection result is a traversable object based on preset semantic and physical state mapping information; and determining whether the driving environment includes traversable or intraversable areas based on the determination result.

2. The method according to claim 1, characterized in that, Based on the aforementioned height dimension feature, the bird's-eye view feature is expanded into a three-dimensional voxel feature, including: Based on the height dimension feature, the three-dimensional voxel space corresponding to the bird's-eye view feature is determined; the three-dimensional voxel space includes: batch size data, number of feature channels, planar resolution data, and height dimension feature; The ratio of the number of feature channels to the height dimension feature is used as the updated number of feature channels; Based on the batch size data, the updated number of feature channels, the planar resolution data, and the height dimension features, the three-dimensional voxel features in the three-dimensional voxel space are determined.

3. The method according to claim 1, characterized in that, Based on the target detection results within a specified time period, the driving environment of the autonomous vehicle is determined, including: Determine the positional relationship between the target detection results, and determine the passable area based on the positional relationship; Based on the passable area, the scene information of the driving environment is determined; the scene information includes at least one or more of the following: road, pedestrian area, construction area, and risk area.

4. The method according to claim 1, characterized in that, Based on the target detection results within a specified time period, the driving environment of the autonomous vehicle is determined, including: Based on the preset semantic and object hardness mapping information, determine whether the target detection result is a crushable object; Based on the judgment result, it is determined that the driving environment includes either a rollable area or an inrollable area.

5. The method according to claim 1, characterized in that, The voxel is a cube; the side length of the cube is a preset length; the preset length is determined based on the height dimension feature and the preset number of voxels.

6. The method according to claim 1, characterized in that, The semantic categories include any of the following: traffic participants, plants, weather conditions, barriers, buildings, and road obstacles.

7. The method according to claim 1, characterized in that, Also includes: Based on the driving environment, vehicle control instructions are generated; the vehicle control instructions are used to cause the autonomous vehicle to perform any of the following operations: path planning, traversable area analysis, risk assessment, risk decision-making, behavior prediction, and behavior control.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-7.

9. A vehicle, characterized in that, Includes cameras, lidar, millimeter-wave radar, and the electronic device described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images