A three-dimensional semantic occupancy perception method and device for complex environments

Abstract

The present application provides a three-dimensional semantic occupancy perception method and device for complex environments, wherein the method includes: obtaining a panoramic surround image of a vehicle at the current moment, processing the panoramic surround image, and obtaining a three-dimensional occupancy perception result; downsampling the three-dimensional occupancy perception result to obtain an occupancy query result; using a SAM model to perform semantic segmentation processing on the panoramic surround image to obtain a two-dimensional semantic feature map; using a spatial cross-attention mechanism to process the two-dimensional semantic feature map and the occupancy query result to obtain a fused three-dimensional voxel feature; using a three-dimensional decoder and an occupancy head to process the fused three-dimensional voxel feature to obtain a three-dimensional semantic occupancy perception result of the panoramic surround image. The method of the present application can achieve refined 3D semantic occupancy prediction, thereby achieving stronger representation learning and generalization capabilities for irregular objects in complex environments.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a three-dimensional semantic occupancy perception method and device for complex environments. Background Art

[0002] Currently, vision-based 3D surround perception methods use the semantic information of images to infer the geometry of 3D scenes, thereby more comprehensively reconstructing the surrounding 3D scenes to improve the perception and decision-making capabilities of autonomous driving systems for complex scenes. However, since these methods are based on multi-view images, and images from different viewpoints may have different semantic understandings of the same object or scene, they are insufficient in modeling semantic consistency. In addition, when dealing with irregularly shaped objects in complex environments, these methods do not have strong enough representation learning and generalization capabilities, resulting in insufficient accurate representation and generalization capabilities for complex scenes, and difficulties in precise positioning and high-resolution image segmentation. Summary of the invention

[0003] In view of this, the present application provides a three-dimensional semantic occupancy perception method and device for complex environments to solve the above-mentioned technical problems.

[0004] In a first aspect, an embodiment of the present application provides a three-dimensional semantic occupancy perception method for a complex environment, including:

[0005] Acquire a panoramic surround image of the vehicle at the current moment, wherein the panoramic surround image includes RGB images under multiple viewing angles;

[0006] Processing the panoramic surround image to obtain a three-dimensional occupancy perception result; downsampling the three-dimensional occupancy perception result to obtain an occupancy query result;

[0007] Using the SAM model to perform semantic segmentation on the panoramic image to obtain a two-dimensional semantic feature map;

[0008] Processing the two-dimensional semantic feature map and the occupancy query result by using a spatial cross-attention mechanism to obtain a fused three-dimensional voxel feature;

[0009] The fused three-dimensional voxel features are processed using a three-dimensional decoder and an occupancy head to obtain a three-dimensional semantic occupancy perception result of the panoramic surround image.

[0010] Furthermore, the panoramic surround image is processed to obtain a three-dimensional occupancy perception result, including:

[0011] A two-dimensional encoder is used to extract the two-dimensional multi-view features of the panoramic surround image, and the two-dimensional multi-view features are projected into the three-dimensional self-vehicle coordinate system to obtain the three-dimensional multi-view features;

[0012] Extracting three-dimensional voxel features from the three-dimensional multi-view features to obtain three-dimensional voxel features;

[0013] The 3D voxel features are processed using a 3D decoder and an occupancy head to obtain the 3D occupancy perception result.

[0014] Further, the three-dimensional occupancy sensing result is downsampled to obtain an occupancy query result, including:

[0015] Using voxel segmentation function to perceive three-dimensional occupancy results Coarse Perform downsampling to obtain the occupancy query result Q H :

[0016] Q H =T v→w (F split (V o ,δ))

[0017] Among them, F split (·,·) is the voxel segmentation function, δ is the segmentation ratio, and for the three-dimensional voxel feature V o The split index of any point coordinate (x0, y0, z0) in is i, j, k are all (0, δ-1) parameters, T v→w (·) is the conversion function from voxel coordinate system to world coordinate system.

[0018] Furthermore, the two-dimensional semantic feature map and the occupancy query result are processed using a spatial cross attention mechanism to obtain a fused three-dimensional voxel feature, including:

[0019] Projecting the occupancy query result on the two-dimensional semantic feature map to obtain a first gridded two-dimensional semantic feature map;

[0020] Using the occupancy query result to perform a cross query on the gridded two-dimensional semantic feature map to obtain a second gridded two-dimensional semantic feature map;

[0021] The second gridded two-dimensional semantic feature map is projected into three-dimensional space to obtain a fused three-dimensional voxel feature.

[0022] Furthermore, the occupancy query result is projected on the two-dimensional semantic feature map to obtain a gridded two-dimensional semantic feature map, including:

[0023] Gridded 2D semantic feature map F seg for:

[0024] F seg =F Grid (F SAM , T w→c(Q H ))

[0025] Among them, F Grid (·,·) is the grid sample function, F SAM is a two-dimensional semantic feature map, T w→ (·) is the conversion function from world coordinate system to camera coordinate system.

[0026] In a second aspect, an embodiment of the present application provides a three-dimensional semantic occupancy perception device for complex environments, including:

[0027] An acquisition unit, used to acquire a panoramic surround image of the vehicle at a current moment, wherein the panoramic surround image includes RGB images under multiple viewing angles;

[0028] A first processing unit is used to process the panoramic surround image to obtain a three-dimensional occupancy perception result; downsample the three-dimensional occupancy perception result to obtain an occupancy query result;

[0029] The second processing unit is used to perform semantic segmentation processing on the panoramic surround image by using a SAM model to obtain a two-dimensional semantic feature map;

[0030] A fusion unit, used for processing the two-dimensional semantic feature map and the occupancy query result by using a spatial cross attention mechanism to obtain a fused three-dimensional voxel feature;

[0031] The semantic occupancy perception unit is used to process the fused three-dimensional voxel features using a three-dimensional decoder and an occupancy head to obtain a three-dimensional semantic occupancy perception result of the panoramic surround image.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the embodiment of the present application when executing the computer program.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method of the embodiment of the present application is implemented.

[0034] The method of the present application can achieve refined 3D semantic occupancy prediction, thereby achieving stronger representation learning and generalization capabilities for irregular objects in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 A technical roadmap for a three-dimensional semantic occupancy perception method for complex environments provided in the embodiments of the present application;

[0037] Figure 2 A flowchart of a three-dimensional semantic occupancy perception method for complex environments provided in an embodiment of the present application;

[0038] Figure 3 A functional structure diagram of a three-dimensional semantic occupancy sensing device for complex environments provided in an embodiment of the present application;

[0039] Figure 4 A functional structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0041] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0042] First, the design concept of the embodiments of the present application is briefly introduced.

[0043] Understanding the 3D geometry of the surrounding scene is crucial for autonomous driving systems, as it can help autonomous vehicles navigate in complex traffic scenarios and avoid potential dangers, thereby improving the safety and efficiency of autonomous driving systems. By acquiring three-dimensional information about the surrounding environment, autonomous driving systems can more accurately perceive and understand the location, size, and shape of roads, surrounding obstacles, and other traffic participants, so that they can more effectively plan the vehicle's obstacle avoidance path and make adaptive decisions for safe navigation. Reconstructing an accurate 3D scene model of the surroundings can also help autonomous driving systems achieve advanced driving functions such as automatic parking and automatic overtaking, providing drivers and passengers with a more comfortable and convenient travel experience.

[0044] LiDAR is a direct solution to capture geometric information, but its high sensor cost and sparse scanning points limit its further application. Recently, vision-centric autonomous driving has made remarkable progress in various 3D perception tasks, which mainly consist of key technologies such as target-centric 3D object detection and multi-object tracking. However, traditional 3D bounding boxes cannot describe irregular vehicles in daily driving scenes in detail, such as protruding tails. To solve these problems, defining the 3D world as occupied space is more conducive to representing obstacles and thus avoiding collisions. As a general scene representation method, occupied space is suitable for autonomous driving tasks such as scene completion, 3D object detection, path planning and vehicle control. In order to improve the description ability of driving scenes and enable autonomous driving systems to perceive and understand the surrounding environment more effectively, semantic occupancy perception has received increasing attention and has become a promising and challenging research direction in autonomous driving perception.

[0045] Traditional occupancy perception methods rely on geometric inputs, but these methods have some limitations when dealing with complex driving scenarios, which mainly include expensive sensor costs, high dependence on sensor accuracy, and difficulty in adapting to dynamic scenes. In addition, since geometric inputs cannot capture the visual details of complex scenes, these methods can only provide limited scene representation. In order to overcome these problems, occupancy perception methods based on visual inputs have made significant progress in recent years. These methods can obtain more visual information through multi-view images or video data, so as to understand and infer scenes more comprehensively and accurately. Compared with the direct geometric input in traditional methods, vision-based 3D surround perception methods use the semantic information of images to infer the geometry of 3D scenes, so as to reconstruct the surrounding 3D scenes more comprehensively to improve the perception and decision-making ability of autonomous driving systems for complex scenes. However, since these methods are based on multi-view images, and images from different views may have different semantic understandings of the same object or scene, they are insufficient in modeling semantic consistency. In addition, when dealing with irregularly shaped objects in complex environments, these methods do not have strong enough representation learning and generalization capabilities, resulting in insufficient accurate representation and generalization capabilities for complex scenes, and difficulties in precise positioning and high-resolution image segmentation.

[0046] To solve the above technical problems, Figure 1 As shown, the present application provides a three-dimensional semantic occupancy perception method for complex environments, which processes the panoramic surround image to obtain a three-dimensional occupancy perception result; downsamples the three-dimensional occupancy perception result to obtain an occupancy query result; uses a SAM model to perform semantic segmentation processing on the panoramic surround image to obtain a two-dimensional semantic feature map; uses a spatial cross attention mechanism (SCA) to process the two-dimensional semantic feature map and the occupancy query result to obtain a fused three-dimensional voxel feature; uses a three-dimensional decoder and an occupancy head to process the fused three-dimensional voxel feature to obtain a three-dimensional semantic occupancy perception result of the panoramic surround image.

[0047] The advantages of this application are:

[0048] 1. The method of the present application combines a refined semantic segmentation network based on the SAM model with an occupancy network to achieve refined 3D semantic occupancy prediction, thereby achieving stronger representation learning and generalization capabilities for irregular objects in complex environments, and thus having a deeper understanding of the surrounding environment.

[0049] 2. The method of this application can obtain more visual information, thereby understanding and inferring the scene more comprehensively and accurately, so as to improve the perception and decision-making ability of the autonomous driving system for complex scenes.

[0050] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0051] like Figure 2 As shown, the embodiment of the present application provides a three-dimensional semantic occupancy perception method for a complex environment, comprising the following steps:

[0052] Step 101: Acquire a panoramic surround image of the vehicle at the current moment, wherein the panoramic surround image includes RGB images under multiple viewing angles;

[0053] Specifically, multiple annular cameras are arranged on the car, and color images under multiple viewing angles can be obtained by photographing the environment around the car through the annular cameras, that is, a surround view image around the car is obtained.

[0054] Step 102: Process the panoramic surround image to obtain a three-dimensional occupancy sensing result; downsample the three-dimensional occupancy sensing result to obtain an occupancy query result;

[0055] Step 103: using the SAM model to perform semantic segmentation processing on the panoramic surround image to obtain a two-dimensional semantic feature map;

[0056] Exemplarily, the precision of the edge of the segmentation result and the ability to understand the vehicle's surroundings can be improved through a fine semantic segmentation network based on the SAM model. The SAM model has strong zero-sample generalization and visual representation capabilities, which can capture various details and features in the image and improve the precision of the edge of the segmentation result. The core of the model is to understand image content, user intent, and the relationship between them. It is pre-trained on a large-scale visual dataset SA-1B containing more than 1 billion masks. There is no need for special training or fine-tuning for each task, and the pre-trained model can be used directly to complete the segmentation task. The SAM model is selected to provide more refined semantic information for the coarse-grained three-dimensional occupancy perception result of step 102.

[0057] Step 104: Processing the two-dimensional semantic feature map and the occupancy query result by using a spatial cross-attention mechanism to obtain a fused three-dimensional voxel feature;

[0058] Step 105: Process the fused 3D voxel features using a 3D decoder and an occupancy head to obtain a 3D semantic occupancy perception result of the panoramic surround image.

[0059] In some embodiments, the panoramic surround image is processed to obtain a three-dimensional occupancy perception result, including:

[0060] A two-dimensional encoder is used to extract the two-dimensional multi-view features of the panoramic surround image, and the two-dimensional multi-view features are projected into the three-dimensional self-vehicle coordinate system to obtain the three-dimensional multi-view features;

[0061] Extracting three-dimensional voxel features from the three-dimensional multi-view features to obtain three-dimensional voxel features;

[0062] The 3D voxel features are processed using a 3D decoder and an occupancy head to obtain the 3D occupancy perception result.

[0063] Specifically, the traditional semantic segmentation network uses ViT pre-trained based on MAE as a 2D backbone. It first segments the image of each view, divides it into image blocks, and then combines them into an input sequence in a certain order. The image blocks are then passed to the ViT model as the input sequence. Each image block is converted into a feature vector sequence through the input embedding layer, and then self-attention interaction is performed through the Transformer encoder layer to finally generate a global visual representation. The multi-view image features obtained in this way can capture the spatial relationship and structure between different viewpoints, providing a powerful feature representation for the subsequent coarse-grained occupancy prediction task.

[0064] The purpose of projecting the two-dimensional multi-view features into the three-dimensional self-vehicle coordinate system is to solve the problem of lack of depth information in the input image. The conversion is based on the LSS framework, which retains the height information unlike the BEV space conversion. The core steps include Lift, Splat and Shoot. In the Lift stage, the depth distribution of the feature points after downsampling of the image plane is explicitly estimated to obtain the cone point cloud containing the image features. In the Splat stage, the cone point clouds of all cameras are combined with the internal and external parameters of the camera to form a Voxel feature map. Finally, the Shoot stage processes the Voxel feature map through the task head and outputs the perception result. This series of steps enables the system to restore the depth information of the image in three-dimensional space, thereby achieving more accurate environmental perception.

[0065] 3D occupancy perception results Coarse The formula is:

[0066] O Coarse =OccupancyHead(Decoder(T w→v (F 3D )))

[0067] Where T w→v is the conversion from world coordinates to voxel coordinates, OccupancyHead and Decoder represent the 3D decoder and occupancy head, and the output of these network structures is a coarse occupancy grid.

[0068] In some embodiments, downsampling the three-dimensional occupancy sensing result to obtain an occupancy query result includes:

[0069] Using voxel segmentation function to perceive three-dimensional occupancy results Coarse Perform downsampling to obtain the occupancy query result Q H :

[0070] Q H =T v→w (F split (V o ,δ))

[0071] Among them, F split (·,·) is the voxel segmentation function, δ is the segmentation ratio, and for the three-dimensional voxel feature V o The split index of any point coordinate (x0, y0, z0) in is i, j, k are all (0, δ-1) parameters, T v→w (·) is the conversion function from voxel coordinate system to world coordinate system.

[0072] In some embodiments, the two-dimensional semantic feature map and the occupancy query result are processed using a spatial cross-attention mechanism to obtain a fused three-dimensional voxel feature, including:

[0073] Projecting the occupancy query result on the two-dimensional semantic feature map to obtain a first gridded two-dimensional semantic feature map;

[0074] Using the occupancy query result to perform a cross query on the gridded two-dimensional semantic feature map to obtain a second gridded two-dimensional semantic feature map;

[0075] The second gridded two-dimensional semantic feature map is projected into three-dimensional space to obtain a fused three-dimensional voxel feature.

[0076] In some embodiments, the occupancy query result is projected on the two-dimensional semantic feature map to obtain a gridded two-dimensional semantic feature map, including:

[0077] Gridded 2D semantic feature map F seg for:

[0078] F seg =F Grid (F SAM , T w→c (Q H ))

[0079] Among them, F Grid (·,·) is the grid sample function, F SAM is a two-dimensional semantic feature map, T w→c(·) is the conversion function from world coordinate system to camera coordinate system.

[0080] In some embodiments, the fused three-dimensional voxel features are processed using a three-dimensional decoder and an occupancy head to obtain a three-dimensional semantic occupancy perception result of the panoramic surround image, including:

[0081] 3D semantic occupancy perception results fine for:

[0082] O fine =OccupancyHead(Decoder(LayerNorm(FC(F fused ))))

[0083] Among them, FC represents the FC layer, and LayerNorm represents the normalization operation. fine Compared to O Coarse More refined.

[0084] Based on the above embodiments, the present application provides a method and device for three-dimensional semantic occupancy perception in a complex environment. Figure 3 As shown, the three-dimensional semantic occupancy perception method and device 200 for complex environments provided in the embodiment of the present application at least includes:

[0085] An acquisition unit 201 is used to acquire a panoramic surround image of the vehicle at a current moment, wherein the panoramic surround image includes RGB images under multiple viewing angles;

[0086] The first processing unit 202 is used to process the panoramic surround image to obtain a three-dimensional occupancy perception result; downsample the three-dimensional occupancy perception result to obtain an occupancy query result;

[0087] The second processing unit 203 is used to perform semantic segmentation processing on the panoramic surround image using a SAM model to obtain a two-dimensional semantic feature map;

[0088] A fusion unit 204 is used to process the two-dimensional semantic feature map and the occupancy query result by using a spatial cross attention mechanism to obtain a fused three-dimensional voxel feature;

[0089] The semantic occupancy perception unit 205 is used to process the fused 3D voxel features using a 3D decoder and an occupancy head to obtain a 3D semantic occupancy perception result of the panoramic surround image.

[0090] It should be noted that the principle of solving the technical problem by the three-dimensional semantic occupancy perception method device 200 for complex environments provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the three-dimensional semantic occupancy perception device 200 for complex environments provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0091] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 4 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the three-dimensional semantic occupancy perception method for complex environments provided in the embodiment of the present application is implemented.

[0092] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0093] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .

[0094] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including but not limited to: an operating subsystem, one or more applications, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0095] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 4 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0096] It should be noted that Figure 4 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0097] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by a processor, the three-dimensional semantic occupancy perception method for a complex environment provided by the embodiment of the present application is implemented. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the three-dimensional semantic occupancy perception method for a complex environment provided by the embodiment of the present application by executing the built-in or installed executable program.

[0098] The three-dimensional semantic occupancy perception method for complex environments provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on an electronic device 300, the program code is used to enable the electronic device 300 to execute the three-dimensional semantic occupancy perception method for complex environments provided in the embodiment of the present application.

[0099] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0100] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0101] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0102] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A three-dimensional semantic occupancy perception method for complex environments, characterized in that: include: Acquire a panoramic surround image of the vehicle at the current moment, wherein the panoramic surround image includes RGB images under multiple viewing angles; Processing the panoramic surround image to obtain a three-dimensional occupancy perception result; Downsampling the three-dimensional occupancy sensing result to obtain an occupancy query result; Using the SAM model to perform semantic segmentation on the panoramic image to obtain a two-dimensional semantic feature map; Processing the two-dimensional semantic feature map and the occupancy query result by using a spatial cross-attention mechanism to obtain a fused three-dimensional voxel feature; Processing the fused three-dimensional voxel features using a three-dimensional decoder and an occupancy head to obtain a three-dimensional semantic occupancy perception result of the panoramic surround image; Downsampling the three-dimensional occupancy sensing result to obtain an occupancy query result, including: Using voxel segmentation function to perceive three-dimensional occupancy results Coarse Perform downsampling to obtain the occupancy query result Q H : Q H =T v→w (F split (V o ,δ)) Among them, F split (·,·) is the voxel segmentation function, δ is the segmentation ratio, and for the three-dimensional voxel feature V o The split index of any point coordinate (x0, y0, z0) in is i, j, k are all (0, δ-1) parameters, T v→w (·) is the conversion function from voxel coordinate system to world coordinate system; The two-dimensional semantic feature map and the occupancy query result are processed using a spatial cross-attention mechanism to obtain fused three-dimensional voxel features, including: Projecting the occupancy query result on the two-dimensional semantic feature map to obtain a first gridded two-dimensional semantic feature map; The occupancy query results are used to perform cross-query on the first gridded two-dimensional semantic feature map to obtain gridding the second two-dimensional semantic feature map; The second gridded two-dimensional semantic feature map is projected into three-dimensional space to obtain a fused three-dimensional voxel feature.

2. The three-dimensional semantic occupancy perception method for complex environments according to claim 1, characterized in that: The panoramic surround image is processed to obtain a three-dimensional occupancy perception result, including: A two-dimensional encoder is used to extract the two-dimensional multi-view features of the panoramic surround image, and the two-dimensional multi-view features are projected into the three-dimensional self-vehicle coordinate system to obtain the three-dimensional multi-view features; Extracting three-dimensional voxel features from the three-dimensional multi-view features to obtain three-dimensional voxel features; The 3D decoder and occupancy head are used to process the 3D voxel features to obtain the 3D occupancy perception results.

3. The three-dimensional semantic occupancy perception method for complex environments according to claim 1, characterized in that: Project the occupancy query result on the two-dimensional semantic feature map to obtain a first gridded two-dimensional semantic feature map, including: The first gridded 2D semantic feature map F seg for: F seg =F Grid (F SAM ,T w→c (Q H )) Among them, F Grid (·,·) is the grid sample function, F SAM is a two-dimensional semantic feature map, T w→c (·) is the conversion function from world coordinate system to camera coordinate system.

4. A three-dimensional semantic occupancy perception device for complex environments, characterized in that: include: An acquisition unit, used to acquire a panoramic surround image of the vehicle at a current moment, wherein the panoramic surround image includes RGB images under multiple viewing angles; A first processing unit is used to process the panoramic surround image to obtain a three-dimensional occupancy perception result; Downsampling the three-dimensional occupancy sensing result to obtain an occupancy query result; The second processing unit is used to perform semantic segmentation processing on the panoramic surround image by using a SAM model to obtain a two-dimensional semantic feature map; A fusion unit, used for processing the two-dimensional semantic feature map and the occupancy query result by using a spatial cross attention mechanism to obtain a fused three-dimensional voxel feature; A semantic occupancy perception unit, configured to process the fused 3D voxel features using a 3D decoder and an occupancy head to obtain a 3D semantic occupancy perception result of the panoramic surround image; Downsampling the three-dimensional occupancy sensing result to obtain an occupancy query result, including: Using voxel segmentation function to perceive three-dimensional occupancy results Coarse Perform downsampling to obtain the occupancy query result Q H : Q H =T v→w (F split (V o ,δ)) Among them, F split (·,·) is the voxel segmentation function, δ is the segmentation ratio, and for the three-dimensional voxel feature V o The split index of any point coordinate (x0, y0, z0) in is i, j, k are all (0, δ-1) parameters, T v→w (·) is the conversion function from voxel coordinate system to world coordinate system; The fusion unit is specifically used for: Projecting the occupancy query result on the two-dimensional semantic feature map to obtain a first gridded two-dimensional semantic feature map; Using the occupancy query result to perform a cross query on the first gridded two-dimensional semantic feature map, to obtain a second gridded two-dimensional semantic feature map; The second gridded two-dimensional semantic feature map is projected into three-dimensional space to obtain a fused three-dimensional voxel feature.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images