BEV perception optimization method based on semantic projection compensation and edge structure enhancement

Abstract

The invention discloses a BEV perception optimization method based on semantic projection compensation and edge structure enhancement, and the method comprises the steps: constructing an image edge feature pyramid under convolution perception through an MSEII module, and enhancing the boundary significance of a weak texture region; and then image semantic embedding is extracted by using a deep semantic segmentation network, geometric projection is carried out based on camera-radar external participation internal reference, image semantic features are mapped to a point cloud space, and an SPE module is constructed to complete semantic completion and structure alignment. After the image and the point cloud features are fused in a unified BEV coordinate system, a spatial context relation is further modeled through sparse point convolution, and a three-dimensional target bounding box and a semantic tag are output through a decoupling detection head. According to the method, on the basis of guaranteeing model lightweight and deployment efficiency, the long-distance target detection capability and semantic consistency in a shielding scene are remarkably improved, and the method is suitable for multi-mode BEV perception application in other resource-constrained equipment such as a mobile robot.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and robot perception, and in particular relates to a BEV perception optimization method based on semantic projection compensation and edge structure enhancement. Background Art

[0002] In mobile robotics and autonomous navigation applications, bird's-eye view (BEV) perception technology is widely used for 3D object detection and environmental semantic understanding due to its excellent spatial alignment capabilities and clear structural representation. However, existing BEV methods suffer from multiple problems in complex scenarios. First, lidar point clouds are sparse and unevenly distributed, and when encountering images with weak textures or reflective areas, the boundary information is poorly represented, resulting in loss of structural details. Second, lighting changes and dynamic obstacles (such as pedestrians, animals, and leaves) can easily cause blind spots and short-term failures in the perception system, reducing the stability of path planning. In addition, the fusion of image and point cloud features mainly relies on geometric projection, which does not fully consider imaging distortion, extrinsic parameter errors, and nonlinear depth changes, resulting in inaccurate spatial alignment of semantic and geometric features. Existing systems lack effective context completion and interpolation strategies in feature-sparse areas, and the fusion granularity is coarse, ignoring the fine-grained contribution of multi-scale features in spatial modeling, affecting perception accuracy and stability. Summary of the Invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a BEV perception optimization method based on semantic projection compensation and edge structure enhancement. It is aimed at the needs of three-dimensional target detection and semantic understanding of quadruped robots in complex dynamic environments. This method aims to address the problems of decreased perception accuracy, large feature alignment errors and insufficient semantic understanding in existing Bird's Eye View (BEV) perception systems when processing sparse point clouds, weak texture images, and in the presence of occlusion interference. A multimodal perception strategy of structure and semantic collaborative modeling is proposed. By optimizing the feature alignment mechanism and enhancing the expression of edge information, the robustness of the environmental perception of quadruped robots in unstructured terrain and dynamic scenes is improved, and the operating efficiency and deployment feasibility of the BEV model on platforms with limited computing power are improved.

[0004] Technical solution: The present invention provides a BEV perception optimization method based on semantic projection compensation and edge structure enhancement, comprising the following steps:

[0005] Step S1: Acquire camera images and lidar point cloud data from a synchronous sensor; perform color normalization, distortion correction, and size scaling on the camera image, and denoise the lidar point cloud data to obtain preprocessed image data; and improve the stability of the spatial structure and the accuracy of modal alignment.

[0006] Step S2: The preprocessed image data is input into the MSEII (Multi-Scale Edge Enhancement Module), and an image edge feature pyramid is constructed through multi-resolution convolution and high-frequency information channels to enhance the structural boundary expression capability in weak texture or reflective areas; and provide a geometric boundary reference for the accurate placement of subsequent semantic features in three-dimensional space.

[0007] Step S3: The deep semantic feature embedding of the image is extracted through the semantic segmentation network, and the image features are projected into the point cloud space through geometric mapping according to the camera-radar calibration parameters. The SPE (Semantic Projection Compensation Enhancement Module) is constructed to obtain the semantically completed point cloud features and image structure information. This mechanism semantically fills in and structurally aligns missing or sparse areas in the point cloud through spatial projection of image semantics, thereby significantly improving the geometric restoration and semantic resolution capabilities of the point cloud in incomplete areas.

[0008] Step S4: Project the semantically completed point cloud features and image structure information into a unified bird's-eye view BEV coordinate system. For feature-sparse areas, use interpolation or mask filling strategies to repair them to ensure feature continuity and modality alignment consistency. Use voxel pooling to generate a fused BEV feature map.

[0009] Step S5: Use the sparse point convolutional network (SMPConv) to perform spatial modeling on the BEV feature map, build the BEVFusion model, and extract context information; complete the bounding box positioning and semantic classification of the three-dimensional target through the decoupled detection head, and output the final detection and segmentation results;

[0010] Step S6: Use mixed precision quantization to optimize the BEVFusion model, quantizing different network modules to INT8 and FP16 precision respectively to improve the reasoning efficiency and perception stability of the model on the quadruped robot platform.

[0011] Furthermore, in step S1, the camera image and lidar point cloud data from the synchronous acquisition system are first obtained. For the camera image, the mean-standard deviation normalization method is used to perform color standardization on each channel of the image to reduce the interference of different lighting conditions, imaging device parameters and exposure status on the overall color distribution of the image, and enhance the uniformity and perceptual stability of the input image. Subsequently, a distortion correction operation is performed on the image, and the barrel distortion and pincushion distortion caused by the wide-angle lens are corrected through the camera intrinsic parameter matrix and the distortion parameter model to ensure the consistency of each geometric structure in the image with the real physical space. Next, the image size is scaled and adjusted using the affine transformation matrix to unify the multi-source image input to a standard resolution, meet the scale consistency requirements of the downstream module, and provide a basis for spatial alignment between the image and the point cloud. At the same time, the lidar point cloud data is denoised by using a radius outlier removal (ROR) method based on neighborhood density to remove abnormal isolated points with insufficient number of neighbors within a certain radius, thereby removing noise interference caused by environmental dust, dynamic occlusions or sensing errors, thereby improving the integrity and spatial stability of the point cloud structure and providing a high-quality input basis for subsequent cross-modal feature alignment and perception fusion.

[0012] Furthermore, in step S2, the preprocessed camera image is first fed into the MSEII. An image pyramid is constructed to extract image structural features at multiple scale levels. Low-resolution layers focus on capturing the overall scene outline and layout, while high-resolution layers focus on preserving high-frequency features such as object boundaries and texture details. Subsequently, a shallow convolutional neural network is used to extract edge and gradient information from the high-resolution image, highlighting key contours, boundaries, and regions of sudden change. Simultaneously, a deep network is used to encode features from the low-resolution image, acquiring more semantically structured contextual information, thereby improving structural perception in weakly textured or complex backgrounds. An edge attention mechanism (EdgeAttentionModule) is further introduced to enhance the network's response weights to boundary regions. Lateral connections and upsampling are combined to fuse feature maps of different scales, generating a structure-enhanced feature map that exhibits spatial continuity and multi-scale perception. This structural map serves as a geometric prior and is fed into the semantic compensation module, providing accurate boundary guidance for the subsequent projection of image semantic features into the point cloud space, thereby improving modal alignment accuracy and edge matching in the BEV space.

[0013] Furthermore, in step S3, a high-dimensional semantic embedding feature map of the camera image is first extracted through a deep semantic segmentation network. The feature map has pixel-level category prediction and global context information, and can fully express the semantic boundaries and category distribution of objects in the image. Subsequently, a perspective geometric mapping relationship is constructed based on the camera intrinsic and extrinsic parameters, and the image semantic features are back-projected. The SPE is constructed by mapping the original point cloud into the lidar coordinate system using back-projection (B-SPE), thereby injecting dense two-dimensional semantic information into the sparse three-dimensional point cloud space. Then, the original point cloud is used as the geometric skeleton, and the corresponding pixel on the image is found at each point cloud position to obtain its semantic category representation, thereby realizing semantic point injection. Bilinear interpolation, K-nearest neighbor feature aggregation (KNN) and mask-guided filling strategies are introduced for the sparse areas that are not directly covered, thereby realizing image-guided point cloud semantic completion and structural alignment, thereby effectively bridging the perception blind spots of point clouds in low-density scenarios such as long distances, occluded areas or edge boundaries. Finally, the generated semantically enhanced point cloud maintains the original spatial topological structure while having a more complete and fine-grained category distribution and contextual information, providing a structurally complete and semantically clear input basis for subsequent multimodal fusion and three-dimensional perception tasks in the BEV space.

[0014] Furthermore, in step S4, the semantically completed point cloud features and the structurally enhanced image features are first mapped to a unified BEV coordinate system, and the spatial geometric structure is encoded using the Voxel Pooling method. The image features are transformed into a perspective form through homography or depth-guided back-projection to achieve spatial alignment and distribution unification of the two modalities. Subsequently, the point cloud BEV features and image BEV features are dimensionally unified using the channel matching and spatial resampling mechanisms, and the sparse areas are repaired using the mask-guided completion technology to enhance the expressive ability of the multimodal feature map in terms of structural continuity and boundary stability. Next, a cross-modal attention fusion module is introduced to perform semantic collaborative modeling while maintaining the independent features of the modalities, so that the category context in the image and the spatial topological relationship in the point cloud can be complementary and fused. Finally, the generated fused BEV feature map has high-resolution expression capability, scale consistency and spatial topological integrity, which can not only accurately describe the target structure in complex industrial environments, but also provide highly robust, multimodal collaborative feature input support for subsequent three-dimensional detection, path planning and semantic perception tasks.

[0015] Furthermore, in step S5, the fused multimodal BEV feature map is first fed into SMPConv. Sparse convolution operations and a moving point aggregation mechanism are combined to perform sparse regional structural modeling and context enhancement on the feature map, improving boundary continuity and the spatial representation of distant objects. Stacked sparse residual blocks are then used to further enhance the semantic response of object edges, occluded regions, and weakly structured areas, extracting stable, topology-aware fused features. This feature map is then fed into the decoupled detection head, which performs position, size, and orientation regression of the 3D bounding box and object category prediction, achieving parallel modeling and output of structure and semantics. Finally, 3D object detection results are obtained, including center coordinates, dimensions, orientation, azimuth, velocity, and semantic category. This provides a structurally complete and semantically clear environmental perception foundation for downstream path planning, obstacle avoidance, and object understanding in complex scenes.

[0016] Furthermore, in step S6, the trained BEV multimodal perception model is compressed and deployed using mixed-precision quantization to adapt to the quadruped robot system with limited computing resources in heterogeneous hardware platforms. This quantization method dynamically selects the convolution backbone module and feature downsampling module for INT8 integer processing based on the computational sensitivity and semantic feature distribution of each network layer, while retaining FP16 precision for structures such as the image semantic projection module and the BEV fusion module, thereby significantly reducing computing and storage overhead while ensuring model accuracy. The model after mixed-precision quantization can achieve real-time inference on the edge computing platform and complete tasks such as 3D target detection, semantic segmentation, and path planning with lower latency and power consumption, ensuring that the intelligent perception system deployed on the mobile robot platform can operate stably in complex environments.

[0017] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0018] (1) To address the challenges of existing BEV perception methods in terms of point cloud sparsity, unclear image features, and insufficient modal alignment accuracy, this technology proposes a method framework based on semantic projection compensation and multi-scale edge enhancement. Through structural alignment and semantic completion, the robustness and expressiveness of the perception model in complex scenes are improved. This method demonstrates stronger structural integrity and semantic consistency in long-range detection, occluded area recognition, and low-contrast scenes. It also has good computational efficiency and is suitable for deployment and application in resource-constrained mobile robot platforms.

[0019] (2) This paper proposes a perception scheme that integrates semantic projection compensation and multi-scale edge enhancement. It uses semantic information in images to spatially complete sparse point clouds, improves the alignment accuracy between different modalities, and effectively integrates structural and semantic features in the BEV coordinate system, thereby enhancing the three-dimensional perception capability and task execution stability of quadruped robots in complex scenarios. By optimizing the feature alignment mechanism and enhancing the expression of edge information, the robustness of the quadruped robot's environmental perception in unstructured terrain and dynamic scenes is improved, and the operating efficiency and deployment feasibility of the BEV model on platforms with limited computing power are improved.

[0020] (3) The present invention introduces a multi-scale edge enhancement module (MSEII) and a semantic projection compensation mechanism (SPE) to improve the structural integrity and semantic expression ability of modal fusion features in the BEV space. Specifically, the MSEII module is first used to construct an image edge feature pyramid under convolution perception to enhance the boundary saliency of weak texture areas; then a deep semantic segmentation network is used to extract image semantic embedding, and geometric projection is performed based on camera-radar external and internal parameters to map the image semantic features to the point cloud space. The SPE module is constructed to complete semantic completion and structural alignment. After the image and point cloud features are fused in a unified BEV coordinate system, the spatial context relationship is further modeled through sparse moving point convolution (SMPConv), and the three-dimensional target bounding box and semantic label are output through the decoupled detection head. While ensuring the lightweight model and deployment efficiency, this method significantly improves the long-range target detection capability and semantic consistency in occlusion scenarios, and is suitable for multimodal BEV perception applications in other resource-constrained devices such as mobile robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the flow of a BEV perception optimization method based on semantic projection compensation and edge structure enhancement for a quadruped robot platform.

[0022] Figure 2 This is the improved BEV feature fusion flowchart. DETAILED DESCRIPTION

[0023] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0024] like Figure 1 As shown, a BEV perception optimization method based on semantic projection compensation and edge structure enhancement for a quadruped robot platform of the present invention comprises the following steps:

[0025] Step S1: Acquire camera images and lidar point cloud data from synchronized sensors. Perform color normalization, distortion correction, and resizing on the images to achieve uniform scale and visual quality. De-noise the point cloud data to improve spatial structure stability and modal alignment accuracy.

[0026] Step S2: The preprocessed image is input into MSEII, and an image edge feature pyramid is constructed through multi-resolution convolution and high-frequency information channels. This effectively enhances the structural boundary expression capability in weak texture or reflective areas, and provides a geometric boundary reference for the accurate placement of subsequent semantic features in three-dimensional space.

[0027] Step S3: A semantic segmentation network is used to extract deep semantic feature embeddings from the image. Based on the camera-radar calibration parameters, these image features are projected onto the point cloud space through geometric mapping to construct the SPE. This mechanism uses spatial projection of image semantics to semantically fill in and structurally align missing or sparse areas in the point cloud, significantly improving the geometric restoration and semantic resolution of incomplete areas of the point cloud.

[0028] Step S4: The semantically completed point cloud features and image structure information are projected into a unified bird's-eye view BEV coordinate system, and voxel pooling is used to generate a fused BEV feature map. For feature-sparse areas, interpolation or mask filling strategies are used to repair them to ensure feature continuity and modality alignment consistency.

[0029] Step S5: Use SMPConv to perform spatial modeling on the BEV feature map and extract context information; then use the decoupled detection head to complete the bounding box positioning and semantic classification of the three-dimensional target, and output the final detection and segmentation results.

[0030] Step S6: Use mixed precision quantization to optimize the BEVFusion model, quantizing different network modules to INT8 and FP16 precision respectively to improve the reasoning efficiency and perception stability of the model on the quadruped robot platform.

[0031] Furthermore, in step S1, first, the camera image data For color normalization, use the following formula:

[0032]

[0033] Where I represents the input original image pixel tensor; I ' is the normalized image; μ=[μ R ,μ G ,μ B ] is the pixel mean of RGB channels; σ=[σ R ,σ G,σ B ] is the standard deviation of the corresponding channel. Normalization is performed channel by channel, usually using preset parameters from the ImageNet dataset. Next, the image is distorted using a pixel coordinate remapping model based on the camera intrinsic parameter matrix:

[0034]

[0035] Among them, f x , f y is the focal length of the image in the x and y directions (in pixels); c x , c y is the position of the principal point (optical center) of the image in pixel coordinates. Then, to unify the image size to the standard resolution required by the network, the image is scaled using the following affine transformation matrix:

[0036]

[0037] Among them, (W o , H o ) is the width and height of the original image; (W t , H t ) is the target input size. The scaling matrix adjusts the original image to the target size in equal or non-equal proportions to ensure that the multimodal input has a uniform spatial resolution. Apply statistical outlier filtering (SOR) to denoise. Calculate the p value for each point i The average Euclidean distance to its k nearest neighbors is as follows:

[0038]

[0039] Among them, d i For point p i The mean local neighborhood distance, p j Represents its jth neighbor point. If the following abnormality judgment conditions are met:

[0040] d i >μ d +λ·σ d

[0041] Then p i The abnormal points are removed, where μ d and σ d They represent the average neighborhood distance and standard deviation of all points respectively, and λ is the empirical threshold, which is usually between 1.5 and 3.

[0042] Furthermore, in step S2, the input image is first downsampled to multiple scales using the image pyramid construction function, and the edge response map at each scale is calculated:

[0043]

[0044] Where I0 represents the preprocessed input image; I s represents the downsampled image of the sth layer; r s ∈(0,1] is the corresponding downsampling ratio, S is the number of scale layers; represents a two-dimensional convolution operation; K x and K y is the Sobel convolution kernel, which is defined as:

[0045]

[0046] G s Represents the edge intensity response map of the s-th layer image at each pixel position, which is used to capture structural information at different resolutions.

[0047] Subsequently, the edge response maps of all scales are upsampled and fused into the structure-enhanced feature map:

[0048]

[0049] Among them, E(x,y) represents the edge-enhanced image features after fusion; α s ∈[0,1] represents the attention weight coefficient of the edge feature of the sth layer, satisfying ∑α s =1; Upsample means that the response graph G s Interpolate back to the original resolution H×W.

[0050] Finally, the structure enhancement feature map E is combined with the image semantic feature map F sem Splicing to get the fusion feature map:

[0051] F fused =Concat(F sem ,E)

[0052] in, Represents the image semantic embedding feature; Concat(·) represents the splicing operation along the channel dimension, and the output fusion feature F fused It also has the ability of semantic expression and boundary guidance, providing a geometric structure reference for the precise projection of subsequent semantic features into the sparse point cloud space.

[0053] Furthermore, in step S3, the input image I is first encoded using a semantic segmentation network to extract the image semantic embedding feature map, which is expressed as:

[0054] F sem =SNet(I0)

[0055] Among them, SNet(·) represents the semantic segmentation network; This is the feature embedding result of the image in the semantic space.

[0056] Then, according to the calibration relationship between the camera and the lidar, the image coordinate points are back-projected into the three-dimensional point cloud space using the extrinsic parameter matrix [R|t] and the camera intrinsic parameter matrix K:

[0057] P i =R -1 ·(d i ·K -1 · i -t)

[0058] in, Represents the pixel coordinates in the image; d i Indicates the depth value of the pixel; is the camera intrinsic parameter matrix; denote the rotation and translation matrices respectively; Represents the 3D point cloud coordinates after back-projection.

[0059] Next, the image semantic features F are transformed into sem Mapped to the corresponding point position in the point cloud space, the preliminary point cloud semantic compensation features are obtained, which are expressed as:

[0060] F pc (P i )=Interp(F sem ,p i )

[0061] Among them, Interp(·) is the interpolation sampling function; output Represents the semantic compensation feature of the i-th point.

[0062] In order to further enhance the expression continuity of sparse areas of the point cloud, a semantic diffusion completion strategy based on K-nearest neighbors is introduced to perform neighborhood smoothing compensation on each point cloud position:

[0063]

[0064] in, Represents point P i The set of K nearest neighbor points of ; Represented as the completed features after KNN interpolation.

[0065] Finally, the preliminary semantic features and the completed features are weightedly fused to obtain the semantically enhanced representation of the sparse point cloud:

[0066]

[0067] Among them, β represents the fusion weight factor; Represents the final semantic completion result of the point cloud, which serves as the input for subsequent mapping to the BEV space.

[0068] Furthermore, in step S4, the semantically completed point cloud features and the structurally enhanced image features are first projected into the BEV space coordinate system. For the point cloud features, the maximum height projection is used to map the three-dimensional sparse features into the two-dimensional BEV feature map:

[0069]

[0070] in, It is the point cloud feature after semantic completion; is the projection feature of the point cloud in the BEV coordinate system; Represents the maximum pooling operation in the vertical direction.

[0071] Subsequently, for image features, the structure-enhanced feature map is projected into the BEV space using homography transformation:

[0072]

[0073] Among them, H is the homography transformation matrix; K is the camera intrinsic parameter matrix, R is the rotation matrix, t is the translation vector; n is the ground normal vector, and d is the projection plane distance; It is the structural feature map of the image under BEV.

[0074] Next, the two modal BEV feature maps are linearly weighted fused to form a preliminarily aligned multimodal feature map:

[0075]

[0076] Among them, λ represents the fusion weight hyperparameter, are the BEV features after preliminary fusion.

[0077] Finally, for the hole areas in BEV, bilinear interpolation and K-nearest neighbor (KNN) interpolation are used to jointly complete them to improve the spatial continuity and contextual integrity of the fused features:

[0078]

[0079] Among them, w ij is the bilinear interpolation weight; (x k ,y k ) are the coordinates of the K nearest neighbor grids; F1(x,y) and F2(x,y) are the interpolation completion and KNN repair results respectively; Represents the BEV multimodal feature map after final fusion optimization.

[0080] Furthermore, in step S5, SMPConv is first used to perform continuous space modeling on the fused BEV feature map:

[0081]

[0082] in, BEV feature map after fusion optimization; is the context-enhanced feature map after sparse convolution, Ω is the sparse perceptual neighborhood; W(Δx, Δy) is the sparse convolution kernel weight in the corresponding direction.

[0083] Subsequently, the decoupled detection head is used to perform target detection and semantic recognition on the enhanced feature map:

[0084]

[0085] in, represents the input features of the bounding box branch of the lth layer; Conv(·) represents a two-dimensional convolution operation; BN(·) represents a batch normalization operation; ReLU(·) represents a linear rectification activation function; x c ,y c represents the center coordinate of the target in the BEV plane; z c represents the center height of the target in the vertical direction; w, l, h represent the width, length, and height of the target; P(x, y, c) represents the normalized probability, indicating the confidence that it belongs to semantic category c; S(x, y, c) represents the prediction score that the BEV grid position (x, y) belongs to category c; c represents the semantic category index.

[0086] Finally, the regression and classification results are combined into a complete multimodal detection output:

[0087] O(x,y)={b(x,y),P(x,y,c)}

[0088] Furthermore, in step S6, the trained BEV multimodal perception model is compressed and deployed using a mixed precision quantization method, and INT8 quantization is performed:

[0089]

[0090] Among them, F BEV ' represents the BEV spatial multimodal fusion feature map after INT8 quantization; S is the scaling factor, which compresses the 32-bit floating point value to the INT8 integer range; Z is the zero point offset, which ensures the symmetrical representation of INT8; Y cls '(i,j) represents the category classification probability heat map of each BEV pixel; b' represents the three-dimensional target bounding box parameters (center point coordinates, size, direction angle, speed).

[0091] Use FP16 quantization to quantize the trained BEV model:

[0092]

[0093] Among them, F SPE represents the semantic embedding feature map output by the semantic projection compensation module (SPE); F MSEII Represents the structural pyramid feature map output by MSEII; Indicates feature concatenation or fusion operation; S FP16 Represents the high-precision semantic graph output by the U-Net network.

Claims

1. A BEV perception optimization method based on semantic projection compensation and edge structure enhancement, characterized in that: The steps include: Step S1: Acquire camera images and lidar point cloud data from a synchronous sensor; perform color normalization, distortion correction, and size scaling on the camera image, and perform denoising on the lidar point cloud data to obtain preprocessed image data; Step S2: The pre-processed image data is input into the multi-scale edge enhancement module MSEII, and an image edge feature pyramid is constructed through multi-resolution convolution and high-frequency information channels to enhance the structural boundary expression ability in weak texture or reflective areas; Step S3: Extract deep semantic features of the image through the semantic segmentation network, project the image features into the point cloud space through geometric mapping according to the camera-radar calibration parameters, and construct a semantic projection compensation enhancement module SPE to obtain semantically completed point cloud features and image structure information; Step S4: project the semantically completed point cloud features and image structure information into a unified bird's-eye view BEV coordinate system. For feature-sparse areas, use interpolation or mask filling strategies to repair them, and use voxel pooling to generate a fused BEV feature map. Step S5: Use the sparse point convolutional network SMPConv to perform spatial modeling on the BEV feature map, build a BEVFusion model, and extract context information; complete the bounding box positioning and semantic classification of the three-dimensional target through the decoupled detection head, and output the final detection and segmentation results; Step S6: Optimize the BEVFusion model using a mixed precision quantization method.

2. The BEV perception optimization method based on semantic projection compensation and edge structure enhancement according to claim 1 is characterized in that: Step S1 is specifically: For color normalization, use the following formula: Where I represents the input original image pixel tensor; I' is the normalized image; μ = [μ R ,μ G ,μ B ] is the pixel mean of RGB channels; σ=[σ R ,σ G ,σ B ] is the standard deviation of the corresponding channel, and normalization is performed channel by channel; The image is distorted and a pixel coordinate remapping model based on the camera intrinsic parameter matrix is ​​used: Among them, f x , f y is the focal length of the image in the x and y directions; c x , c y is the position of the principal point of the image, i.e. the optical center, in pixel coordinates; to unify the image size to the standard resolution required by the network, the following affine transformation matrix is ​​used to resize the image: Among them, (W o , H o ) is the width and height of the original image; (W t , H t ) is the target input size. The scaling matrix adjusts the original image to the target size in equal or non-equal proportions. Apply statistical outlier filter SOR to denoise; calculate each point p i The average Euclidean distance to its k nearest neighbors is as follows: Among them, d i For point p i The mean local neighborhood distance, p j Indicates its jth neighbor point, if the following abnormality judgment conditions are met: d i >m d +l·s d Then p i The abnormal points are removed, where μ d and σ d They represent the average neighborhood distance and standard deviation of all points respectively, and λ is the empirical threshold.

3. The BEV perception optimization method based on semantic projection compensation and edge structure enhancement according to claim 1 is characterized in that: Step S2 is specifically as follows: use the image pyramid construction function to perform multi-scale downsampling representation on the input image and calculate the edge response map at each scale: Where I0 represents the preprocessed input image; I s represents the downsampled image of the sth layer; r s ∈(0,1] is the corresponding downsampling ratio, S is the number of scale layers; represents a two-dimensional convolution operation; K x and K y is the Sobel convolution kernel, which is defined as: G s Represents the edge intensity response map of the s-th layer image at each pixel position, which is used to capture structural information at different resolutions; The edge response maps of all scales are upsampled and fused into structure-enhanced feature maps: Among them, E(x,y) represents the edge-enhanced image features after fusion; α s ∈[0,1] represents the attention weight coefficient of the edge feature of the sth layer, satisfying ∑α s =1; Upsample means that the response graph G s Interpolate back to the original resolution H×W; Finally, the structure enhancement feature map E is combined with the image semantic feature map F sem Splicing to get the fusion feature map: F fused =Concat(F sem ,E) in, Represents the image semantic embedding feature; Concat(·) represents the splicing operation along the channel dimension, and the output fusion feature F fused It also has the ability of semantic expression and boundary guidance, providing a geometric structure reference for the precise projection of subsequent semantic features into the sparse point cloud space.

4. The BEV perception optimization method based on semantic projection compensation and edge structure enhancement according to claim 1 is characterized in that: Step S3 is as follows: first, the pre-processed input image I0 is encoded using the semantic segmentation network, and the image semantic embedding feature map is extracted, which is expressed as: F sem =SNet(I0) Among them, SNet(·) represents the semantic segmentation network; The feature embedding result of the image in the semantic space; According to the calibration relationship between the camera and the lidar, the image coordinate points are back-projected into the three-dimensional point cloud space using the extrinsic parameter matrix [R|t] and the camera intrinsic parameter matrix K: P i =R -1 ·(d i ·K -1 · i -t) Among them, p i =[u i ,v i ,1] T Represents the pixel coordinates in the image; d i Indicates the depth value of the pixel; is the camera intrinsic parameter matrix; denote the rotation and translation matrices respectively; Represents the three-dimensional point cloud coordinates after back projection; Use bilinear interpolation function to transform image semantic features F sem Mapped to the corresponding point position in the point cloud space, the preliminary point cloud semantic compensation features are obtained, which are expressed as: F pc (P i )=Interp(F sem ,p i ) Among them, Interp(·) is the interpolation sampling function; output Represents the semantic compensation feature of the i-th point; A semantic diffusion completion strategy based on K-nearest neighbors is introduced to enhance the expression continuity of sparse areas of the point cloud and perform neighborhood smoothing compensation for each point cloud position: in, Represents point P i The set of K nearest neighbor points of ; Represented as the completed features after KNN interpolation; Finally, the preliminary semantic features and the completed features are weightedly fused to obtain the semantically enhanced representation of the sparse point cloud: Among them, β represents the fusion weight factor; Represents the final semantic completion result of the point cloud, which serves as the input for subsequent mapping to the BEV space.

5. The BEV perception optimization method based on semantic projection compensation and edge structure enhancement according to claim 1 is characterized in that: Step S4 is as follows: first, the semantically completed point cloud features and the structurally enhanced image features are projected into the BEV space coordinate system respectively. For the point cloud features, the maximum height projection is used to map the three-dimensional sparse features into the two-dimensional BEV feature map: in, It is the point cloud feature after semantic completion; is the projection feature of the point cloud in the BEV coordinate system; Represents the maximum pooling operation in the vertical direction; For image features, the structure-enhanced feature map is projected into the BEV space using homography transformation: Among them, H is the homography transformation matrix; K is the camera intrinsic parameter matrix, R is the rotation matrix, t is the translation vector; n is the ground normal vector, and d is the projection plane distance; It is the structural feature map of the image under BEV; The two modal BEV feature maps are linearly weighted fused to form a preliminarily aligned multimodal feature map: Among them, λ represents the fusion weight hyperparameter, is the BEV feature after preliminary fusion; For the hole areas in BEV, bilinear interpolation and K-nearest neighbor KNN interpolation are used to complete them: Among them, w ij is the bilinear interpolation weight; (x k ,y k ) are the coordinates of the K nearest neighbor grids; F1(x,y) and F2(x,y) are the interpolation completion and KNN repair results respectively; Represents the BEV multimodal feature map after final fusion optimization.

6. The BEV perception optimization method based on semantic projection compensation and edge structure enhancement according to claim 1 is characterized in that: Step S5 is specifically: using SMPConv to perform continuous space modeling on the fused BEV feature map: in, BEV feature map after fusion optimization; is the context-enhanced feature map after sparse convolution, Ω is the sparse perceptual neighborhood; W(Δx, Δy) is the sparse convolution kernel weight in the corresponding direction; Use the decoupled detection head to perform target detection and semantic recognition on the enhanced feature map: in, represents the input features of the bounding box branch of the lth layer; Conv(·) represents a two-dimensional convolution operation; BN(·) represents a batch normalization operation; ReLU(·) represents a linear rectification activation function; x c ,y c represents the center coordinate of the target in the BEV plane; z c represents the center height of the target in the vertical direction; w, l, h represent the width, length, and height of the target; P(x, y, c) represents the normalized probability, indicating the confidence that it belongs to semantic category c; S(x, y, c) represents the prediction score that the BEV grid position (x, y) belongs to category c; c represents the semantic category index; Finally, the regression and classification results are combined into a complete multimodal detection output: O(x,y)={b(x,y),P(x,y,c)}.

7. The BEV perception optimization method based on semantic projection compensation and edge structure enhancement according to claim 1 is characterized in that: Step S6 is specifically as follows: compress and deploy the trained BEV multimodal perception model using mixed precision quantization and INT8 quantization: Among them, F BEV ' represents the BEV spatial multimodal fusion feature map after INT8 quantization; S is the scaling factor, which compresses the 32-bit floating point value to the INT8 integer range; Z is the zero point offset, which ensures the symmetrical representation of INT8; Y cls '(i,j) represents the category classification probability heat map of each BEV pixel; b' represents the three-dimensional target bounding box parameters including center point coordinates, size, direction angle, and speed; Use FP16 quantization to quantize the trained BEV model: Among them, F SPE represents the semantic embedding feature map output by the semantic projection compensation module SPE; F MSEII Represents the structural pyramid feature map output by MSEII; Indicates feature concatenation or fusion operation; S FP16 Represents the high-precision semantic graph output by the U-Net network.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to claim 1.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

Images