Bird's eye view feature generation method and apparatus, and electronic device

Abstract

The application provides a bird's eye view feature generation method and device and electronic equipment; in the method, a plurality of acquisition images synchronously acquired by all vehicle cameras of a vehicle at a current time are acquired; a bird's eye view and an initial feature tensor thereof are determined according to the plurality of acquisition images; a fusion feature tensor at the current time is generated according to the bird's eye view, the initial feature tensor and a preset attention; and a target feature tensor of the bird's eye view at the current time is generated according to a historical feature tensor at a previous time and the fusion feature tensor. The above generation mode generates the fusion feature tensor through double attention of spatial attention and semantic attention, significantly reduces redundant calculation, suppresses overexposure or overdarkness through the spatial attention, enhances high coupling with a semantic category, thereby improving the robustness of generating the target feature tensor under an extreme light scene such as strong light, rain and fog, avoiding noise interference, and further improving the target feature tensor generation accuracy and generation efficiency of the bird's eye view.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus and electronic device for generating bird's-eye view features. Background Technology

[0002] In the field of autonomous driving visual perception, bird's-eye view (BEV) is rapidly becoming a technological hotspot. Bird's-eye view projects multi-view images captured by cameras onto a scale-free, distortion-free planar coordinate system. This allows for a clear visualization of lane topology, passable areas, and the position, size, and orientation of surrounding road users in three-dimensional space. This "one-map-to-the-world" advantage makes bird's-eye view an efficient intermediary for communication between road perception, trajectory prediction, and even planning.

[0003] In recent years, the introduction of multi-view Transformer structures, sparse convolutional networks, and deformable attention mechanisms into pure visual bird's-eye view perception systems has significantly improved the performance of bird's-eye view systems. However, most existing bird's-eye view systems focus on aligning features from multiple cameras in spatial geometry, leading to the following problems in the feature generation process: ① Redundancy problem: Due to the potential overlap between the perception ranges of multiple cameras, the same target information is repeatedly calculated during feature fusion, resulting in feature redundancy, increased model complexity, slower training convergence, and potentially negatively impacting the accuracy of downstream tasks (such as object detection or segmentation); ② Noise interference problem: This mainly includes camera viewpoint differences and data sampling errors. The camera viewpoint difference problem primarily refers to... The same viewpoint may have differences in feature quality due to factors such as lighting conditions and occlusion. The data sampling error problem refers to the noise that may be introduced by depth estimation error or camera calibration error during the process of feature projection into the BEV space, which will cause low-quality or erroneous features to be mixed into the fusion result, affecting the overall reliability of the bird's-eye view system; ③ Limitations of weight allocation: In the existing technical solutions, weight allocation (such as simple average fusion or static weighted fusion) cannot dynamically adjust the importance of features according to the specific scene, and cannot highlight key viewpoints or features, resulting in insufficient robustness in small target detection and extreme lighting scenes such as rain, fog, and strong light. Summary of the Invention

[0004] In view of this, the object of the present invention is to provide a method, apparatus and electronic device for generating bird's-eye view features, so as to alleviate at least some of the above-mentioned technical problems.

[0005] In a first aspect, embodiments of the present invention provide a method for generating bird's-eye view features. The method includes: acquiring multiple images simultaneously captured by all on-board cameras of the vehicle at the current moment; determining a bird's-eye view and its initial feature tensor based on the multiple captured images; generating a fused feature tensor for the current moment based on the bird's-eye view, the initial feature tensor, and a preset attention; wherein the preset attention includes spatial attention calculation and semantic attention; acquiring the historical feature tensor of the previous moment, and generating a target feature tensor for the bird's-eye view at the current moment based on the historical feature tensor and the fused feature tensor.

[0006] Optionally, the step of generating the fused feature tensor at the current moment based on the bird's-eye view, the initial feature tensor, and the preset attention includes: determining the spatial weight tensor based on the bird's-eye view and spatial attention; determining the semantically reweighted feature tensor based on the initial feature tensor and semantic attention; and generating the fused feature tensor at the current moment based on the spatial weight tensor, the semantically reweighted feature tensor, and the initial feature tensor.

[0007] Optionally, the bird's-eye view includes multiple bird's-eye view grids; the step of determining the spatial weight tensor based on the bird's-eye view and spatial attention includes: for any bird's-eye view grid, obtaining its center position information, and calculating the offset of the bird's-eye view grid based on preset pyramid features; learning the offset and corresponding weights based on backpropagation, and calculating the spatial attention coefficient of the bird's-eye view grid; and determining the spatial weight tensor based on the spatial attention coefficients of multiple bird's-eye view grids.

[0008] Optionally, the step of determining the semantically reweighted feature tensor based on the initial feature tensor and semantic attention includes: performing global pooling on the initial feature tensor and generating a channel weight vector through a preset function; wherein, the global pooling process includes: global average pooling and global max pooling; and generating the semantically reweighted feature tensor based on the channel weight vector and the initial feature tensor.

[0009] Optionally, the step of generating the fused feature tensor at the current time based on the spatial weight tensor, the semantically reweighted feature tensor, and the initial feature tensor includes: generating the concatenated feature tensor at the current time based on the spatial weight tensor and its corresponding first weight, the semantically reweighted feature tensor and its corresponding second weight; wherein the sum of the first weight and the second weight is 1; and generating the fused feature tensor at the current time based on the concatenated feature tensor and the initial feature tensor.

[0010] Optionally, the step of generating the fused feature tensor at the current time based on the spliced ​​feature tensor and the initial feature tensor includes: summing the residuals of the spliced ​​feature tensor and the initial feature tensor to obtain the summation result; and performing regularization processing on the summation result to generate the fused feature tensor at the current time.

[0011] Optionally, the step of generating the target feature tensor of the current bird's-eye view based on the historical feature tensor and the fused feature tensor includes: compensating the historical feature tensor to obtain the compensated historical feature tensor; wherein the compensation includes: pose compensation and rotation compensation; and fusing the compensated historical feature tensor and the fused feature tensor based on deformable self-attention to generate the target feature tensor of the current bird's-eye view.

[0012] Secondly, embodiments of the present invention also provide a bird's-eye view feature generation apparatus, the apparatus comprising:

[0013] The image acquisition module is used to acquire multiple images simultaneously captured by all the vehicle's onboard cameras at the current moment;

[0014] The bird's-eye view determination module is used to determine the bird's-eye view and its initial feature tensor based on multiple acquired images;

[0015] The fusion feature tensor generation module is used to generate the fusion feature tensor at the current moment based on the bird's-eye view, the initial feature tensor, and the preset attention; wherein, the preset attention includes spatial attention calculation and semantic attention;

[0016] The target feature tensor generation module is used to obtain the historical feature tensor of the previous time step and generate the target feature tensor of the bird's-eye view at the current time step based on the historical feature tensor and the fused feature tensor.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the method described in the first aspect.

[0019] The embodiments of the present invention bring the following beneficial effects:

[0020] This invention provides a method, apparatus, and electronic device for generating bird's-eye view features. The method involves acquiring multiple images simultaneously captured by all onboard cameras of the vehicle at the current moment; determining a bird's-eye view and its initial feature tensor based on the multiple images; generating a fused feature tensor for the current moment based on the bird's-eye view, the initial feature tensor, and a preset attention mechanism; wherein the preset attention mechanism includes spatial attention calculation and semantic attention; acquiring the historical feature tensor from the previous moment; and generating the target feature tensor of the bird's-eye view at the current moment based on the historical feature tensor and the fused feature tensor. This generation method, through dual attention of spatial and semantic attention, generates a fused feature tensor, significantly reducing redundant computation and suppressing overexposure or underexposure through spatial attention. It also enhances the high coupling with semantic categories, thereby improving the robustness of generating the target feature tensor under extreme lighting conditions such as strong light, rain, and fog, while avoiding noise interference, thus improving the accuracy and efficiency of generating the target feature tensor of the bird's-eye view.

[0021] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 A flowchart of a bird's-eye view feature generation method provided in an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of the Sigmoi function provided in an embodiment of the present invention;

[0026] Figure 3 A flowchart of another method for generating bird's-eye view features provided in an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of a bird's-eye view feature generation device provided in an embodiment of the present invention;

[0028] Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] To facilitate understanding of this embodiment, the embodiments of the present invention will be described in detail below.

[0031] Example 1

[0032] This invention provides a method for generating bird's-eye view features, which mainly involves multi-camera visual perception technology in intelligent vehicles, ADAS (Advanced Driving Assistance System) and AD (Autonomous Driving), and can be widely applied to passenger cars, commercial vehicles, logistics trucks, low-speed unmanned vehicles in parks, and roadside multi-sensor fusion systems.

[0033] like Figure 1 As shown, the method includes the following steps:

[0034] Step S102: Obtain multiple images simultaneously captured by all on-board cameras of the vehicle at the current moment.

[0035] In practical applications, vehicles are typically equipped with multiple onboard cameras (such as front-view, side-view, and rear-view cameras). During vehicle operation, electronic devices can acquire images simultaneously from these cameras in real-time or periodically. Taking the current moment as an example, these multiple images represent the vehicle's (using this vehicle as an example) views from multiple perspectives at that moment. The image information is then converted from the image control to the BEV space, simplifying the complex three-dimensional environment around the vehicle into a two-dimensional image (i.e., a bird's-eye view). For example, a bird's-eye view provides a comprehensive view of the vehicle's surroundings, including the front, rear, left, and right sides, facilitating fault detection and path planning during autonomous driving. Therefore, improving the feature generation accuracy and efficiency of bird's-eye views is crucial for road perception and fault detection.

[0036] Furthermore, after acquiring multiple images, the electronic device performs preprocessing on them. This preprocessing includes first performing linear distortion correction, exposure leveling, and other visual enhancements on the images; then downsampling the resolution to one-quarter of the original size to reduce memory and bandwidth overhead; finally, by processing the real-time CAN (Controller Area Network) bus and IMU (Inertial Measurement Unit) attitude, real-time fine-tuning of extrinsic parameters is performed between the camera coordinate system and the vehicle coordinate system. This aligns the images acquired by multiple vehicle cameras to the same vehicle coordinate system, thereby improving the accuracy of the acquired images and avoiding noise interference caused by differences in camera perspectives, thus improving the feature generation accuracy of the bird's-eye view. It should be noted that the above preprocessing process can refer to existing technologies, and this embodiment of the invention will not elaborate on it in detail.

[0037] Step S104: Determine the bird's-eye view and its initial feature tensor based on multiple acquired images.

[0038] For the preprocessed multiple acquired images described above, the electronic device determines the bird's-eye view and its initial feature tensor based on the multiple acquired images. This is achieved by inputting the processed multiple acquired images into a BEV encoder, which can also be called a Deformable Ray module. In practical applications, the Deformable Ray module first divides each projection ray into four equally spaced depth layers within the view frustum (i.e., a three-dimensional cone-shaped region extending forward from the camera's optical center as the vertex and the image plane's border as the base, reaching the maximum distance of interest), discretizing the originally continuous space into four levels of "sampling planes." Subsequently, at each depth layer, surrounding pixels are dynamically resampled based on a learnable offset, and weighted fusion is performed on the corresponding features from the acquired images from multiple viewpoints. This "ray-cross-viewpoint-hierarchical" aggregation strategy, while ensuring spatial geometric consistency, adaptively highlights pixel information truly related to the grid position. It should be noted that the specific working principle of the BEV encoder can be referenced from existing BEV encoders; the embodiments of this invention will not be described in detail here.

[0039] Therefore, the deformable ray module processes multiple pre-processed acquired images to obtain a bird's-eye view corresponding to the acquired images at the current moment. This bird's-eye view generation process also involves rasterization, meaning the generated bird's-eye view comprises multiple bird's-eye view grids, the size of which can be set according to actual conditions. Simultaneously, by weighting the corresponding features of the multiple acquired images, the initial feature tensor F of the bird's-eye view can be obtained. in The initial feature tensor F inIt not only preserves the global layout of the vehicle's surrounding environment, but also incorporates multi-scale deep semantics, laying a unified and high-quality foundation for the subsequent generation of initial target features, thereby improving the feature generation accuracy of the bird's-eye view.

[0040] In multi-scale deep semantics, multi-scale refers to the initial feature tensor F. in The channel dimension is not homogeneous features, but rather integrates information from different scale layers, which can describe both detailed textures (i.e., small objects) and large-area structures (such as road boundaries) at the same time; deep semantics refers to the fact that the channel vector contains descriptions of "high semantic layers" such as color texture, object category, edge, drivability, dynamic and static attributes, rather than just "shallow" features such as grayscale or edge intensity.

[0041] Step S106: Generate the fused feature tensor for the current moment based on the bird's-eye view, the initial feature tensor, and the preset attention.

[0042] The preset attention includes spatial attention computation and semantic attention. Therefore, by generating a fusion feature tensor through dual attention of spatial attention and semantic attention, not only is redundant computation significantly reduced, but also overexposure or underexposure is suppressed through spatial attention, and the high coupling with semantic categories is enhanced. This improves the robustness of generating target feature tensors under extreme lighting scenarios such as strong light, rain and fog, while avoiding noise interference, thereby improving the accuracy and efficiency of generating target feature tensors in bird's-eye view.

[0043] Step S108: Obtain the historical feature tensor of the previous moment, and generate the target feature tensor of the bird's-eye view at the current moment based on the historical feature tensor and the fused feature tensor.

[0044] The bird's-eye view feature generation method provided in this invention acquires multiple images simultaneously captured by all onboard cameras of the vehicle at the current moment; determines the bird's-eye view and its initial feature tensor based on the multiple captured images; generates a fused feature tensor for the current moment based on the bird's-eye view, the initial feature tensor, and a preset attention; and generates the target feature tensor of the bird's-eye view at the current moment based on the historical feature tensor and the fused feature tensor from the previous moment. This generation method, through dual attention of spatial attention and semantic attention, generates the fused feature tensor, significantly reducing redundant computation and suppressing overexposure or underexposure through spatial attention. It also enhances the high coupling with semantic categories, thereby improving the robustness of generating the target feature tensor under extreme lighting conditions such as strong light, rain, and fog, while avoiding noise interference, thus improving the accuracy and efficiency of generating the target feature tensor of the bird's-eye view.

[0045] In one implementation, the step of generating the fused feature tensor at the current moment based on the bird's-eye view, the initial feature tensor, and the preset attention includes: (A1) determining the spatial weight tensor based on the bird's-eye view and spatial attention; (A2) determining the semantically reweighted feature tensor based on the initial feature tensor and semantic attention; and (A3) generating the fused feature tensor at the current moment based on the spatial weight tensor, the semantically reweighted feature tensor, and the initial feature tensor. The specific processes of (A1) to (A3) are as follows:

[0046] (A1) The bird's-eye view includes multiple bird's-eye view grids; the step of determining the spatial weight tensor based on the bird's-eye view and spatial attention includes: for any bird's-eye view grid, obtaining its center position information and calculating the offset of the bird's-eye view grid based on the preset pyramid features; learning the offset and the corresponding weight based on backpropagation, and calculating the spatial attention coefficient of the bird's-eye view grid; determining the spatial weight tensor based on the spatial attention coefficients of multiple bird's-eye view grids.

[0047] Specifically, for any bird's-eye view grid, its center position information is (x, y), where x and y represent the coordinates of the center point of the bird's-eye view grid in the two-dimensional bird's-eye view. Then, the pyramid feature is preset to a 3-level pyramid feature. Where, p l Let l represent the feature map of the l-th layer of the pyramid. There are three layers in total, i.e., l = 0, 1, 2, with the resolution decreasing and the semantics increasing layer by layer. Then, the center location information (x, y) is a feature map of the 3rd layer of the pyramid. Calculate the offset Δp of the bird's-eye view grid. l,k Here, k = 1, 2, 3, 4, where k represents the sampling point, that is, the two-dimensional offset of the kth sampling point in the lth layer relative to the center position information (x, y). Therefore, the offset is initialized as a cross. For the center position information of any bird's-eye view grid in the bird's-eye view, four sampling points (up, down, left, and right) are initialized on each layer of pyramid features to form an orthogonal cross template.

[0048] The above determines the offset Δp of the bird's-eye view grid. l,k Then, the offset Δp is learned based on backpropagation. l,k and the corresponding weight w l,k The spatial attention coefficient of this bird's-eye view grid can be calculated. The specific formula for calculating the spatial attention coefficient is as follows:

[0049]

[0050] Where, α s (x, y) represents the spatial attention coefficient, reflecting the characteristic intensity of the final convergence of the bird's-eye view grid. (x, y) represents the center position information of the bird's-eye view grid, that is, the horizontal and vertical coordinates of the center in the two-dimensional bird's-eye view plane coordinate system (XY coordinate system). l,k Indicates the offset Δpl,k The corresponding weights, and Δp l,k Learnable weights that appear in pairs, F l 'Represents the feature map p of the l-th pyramid level l The feature map is obtained after geometric alignment such as rotation and translation. This indicates the component of the offset on the X-axis. The offset component on the Y-axis is represented by l, which represents the number of layers in the pyramid feature map, and k represents the sampling points on each layer of the pyramid feature map.

[0051] Therefore, the spatial attention coefficients of multiple bird's-eye view grids can be obtained according to formula (1), and the spatial weight tensor α can be determined based on the spatial attention coefficients of multiple bird's-eye view grids. s Furthermore, the offset can be automatically adjusted using the above formula (1) to focus the sampling points along the same viewing direction onto pixels that actually have texture or moving edges. Here, the viewing direction is the ray direction from the camera to the center of the bird's-eye view grid. Aggregating along this direction helps maintain geometric consistency; at the same time, the weight w is learned. l,k It can amplify features containing motion traces or co-linear textures, assign less weight to meaningless regions, improve the accuracy of spatial attention coefficient calculation for bird's-eye view grids, thereby improving the accuracy of fused feature tensors and ultimately improving the accuracy of target feature tensor generation for bird's-eye view grids.

[0052] (A2) The step of determining the semantically reweighted feature tensor based on the initial feature tensor and semantic attention includes: performing global pooling on the initial feature tensor and generating channel weight vectors through a preset function; wherein, the global pooling process includes: global average pooling and global max pooling; generating the semantically reweighted feature tensor based on the channel weight vector and the initial feature tensor.

[0053] Specifically, for the initial feature tensor F of the bird's-eye view in First, Global Average Pooling (GAP) and Global Max Pooling (GMP) are performed to map the two-dimensional features to the channel domain to extract channel statistics. These statistics are then compressed and expanded using two levels of one-dimensional fully connected layers, such as two fully connected layers with a compression ratio r = 8. Finally, a channel weight vector α is generated using a preset function. c This is done to adjust the activation intensity of the initial feature tensor while suppressing redundant information and preserving the differences between information.

[0054] The preset functions include the SiLU function and the Sigmoi function. In practical applications, the Sigmoi function compresses the output value to the (0,1) interval, such as... Figure 2The Sigmoi function shown outputs values ​​within the (0,1) interval, achieving soft filtering of channel features and suppressing irrelevant channels. It should be noted that specific SiLU and Sigmoi functions, as well as specific global pooling processing, can be found in existing technologies; these will not be elaborated upon further in this embodiment.

[0055] Furthermore, the aforementioned channel weight vector α c After generation, based on the channel weight vector α c and the initial feature tensor F in Generate semantically reweighted feature tensors The formula for calculating the semantically reweighted feature tensor is as follows:

[0056]

[0057] in, Let α represent the semantically reweighted feature tensor. c F represents the channel weight vector. in represents the initial feature tensor, and ⊙ represents back multiplication.

[0058] Therefore, through the channel weight vector α c Multiply back the initial feature tensor F in Generate semantically reweighted feature tensors The aim is to dynamically suppress noise channels and highlight channels related to key target categories, so that in extreme lighting, rainy nights, or long-tailed scenes, we can still focus on truly discriminative features, thereby improving the generation accuracy and efficiency of target feature tensors in bird's-eye view images.

[0059] (A3) The step of generating the fusion feature tensor at the current time based on the spatial weight tensor, the semantic reweighted feature tensor, and the initial feature tensor includes: generating the concatenated feature tensor at the current time based on the spatial weight tensor and the corresponding first weight, the semantic reweighted feature tensor and the corresponding second weight; wherein the sum of the first weight and the second weight is 1; and generating the fusion feature tensor at the current time based on the concatenated feature tensor and the initial feature tensor.

[0060] Specifically, the channels for spatial attention and semantic attention are concatenated, and weights are adaptively assigned using 1×1 convolutional gating units. Here, we set the spatial weight tensor α. s The corresponding first weight is β s Semantic reweighted feature tensor The corresponding second weight is β c Then the characteristic tensor F out The calculation formula is as follows:

[0061]

[0062] Where, β s +β c =1. In practical applications, the first weight is β. s The second weight is β c It is worth learning.

[0063] Furthermore, according to the splicing feature tensor F out and the initial feature tensor F in Step F in generating the fused feature tensor at the current time step inal Specifically, firstly, the splicing feature tensor F... out and the initial feature tensor F in The residuals are summed to obtain the summation result; then, the summation result is regularized to generate the fused feature tensor F at the current time step. inal Thus, through regularization, the spatial and semantic information are dynamically balanced, improving the fusion feature tensor F. inal The calculation accuracy.

[0064] Among them, the fusion feature tensor F inal The calculation formula is as follows:

[0065] F inal =LN(F out +F in (4)

[0066] Wherein, LN represents regularization. In this embodiment of the invention, the preferred regularization is LayerNorm regularization, but it can be adjusted according to the actual situation.

[0067] In summary, generating fusion feature tensors through dual attention of spatial attention and semantic attention not only significantly reduces redundant computation, but also suppresses overexposure or underexposure through spatial attention, and enhances the high coupling with semantic categories. This improves the robustness of generating target feature tensors in extreme lighting scenarios such as strong light, rain, and fog, while avoiding noise interference, thereby improving the accuracy and efficiency of target feature tensor generation in bird's-eye view images.

[0068] In one implementation, the step of generating the target feature tensor of the current bird's-eye view based on the historical feature tensor and the fused feature tensor includes: compensating the historical feature tensor to obtain the compensated historical feature tensor; wherein the compensation includes: pose compensation and rotation compensation; and fusing the compensated historical feature tensor and the fused feature tensor based on deformable self-attention to generate the target feature tensor of the current bird's-eye view.

[0069] To compensate for coordinate drift caused by vehicle motion, this embodiment of the invention compensates for the historical feature tensor of the previous moment. This compensation includes pose compensation and rotation compensation; specifically, the historical feature tensor of the previous moment is compensated based on compensation amounts (Δx, Δy, Δθ), where Δx and Δy are the translational (or pose) compensation amounts of the vehicle, expressed as the displacement of the vehicle's center of mass on the ground plane, and represent the yaw angle increment; Δθ is the rotation angle of the vehicle body about the vertical axis, thus obtaining the compensated historical feature tensor F. t-1 Here, t represents the current time, and t-1 represents the previous time.

[0070] Furthermore, the compensated historical feature tensor F t-1 However, differences in the motion of dynamic objects or errors in pose estimation still exist, thus requiring feature-level detail alignment. This invention employs local windows and deformable self-attention for alignment. Local windows refer to searching for corresponding key-value pairs within a small 9×9 window in the neighboring area of ​​any query location within the bird's-eye view grid. Deformable self-attention, based on the center of the local window, introduces 4-8 learnable offsets to enable the attention to jump to a more precise semantic alignment point. For example, the pixels at the top of a pedestrian's head in the previous frame are not strictly on the same grid as the pixels at the top of the head in the current frame.

[0071] Therefore, the historical feature tensor F t-1 The fusion feature tensor F at the current moment inal Trans-temporal feature fusion is performed using a deformable self-attention (SA), with a window size of 9. For example, the fused feature tensor F at the current time step is used. inal As a query, with historical feature tensor F t-1 Features of the same window and its offset position are used as Key and Value. After obtaining the weights through softmax, the Value is weighted and summed, and then added to the Query residual to obtain the target feature tensor F of the aligned, temporally consistent bird's-eye view at the current time step. t This effectively reduces ghosting and drift caused by multi-frame overlay, improving the generation accuracy of the target feature tensor. It should be noted that the specific deformable self-attention processing procedure can be found in existing technologies, and will not be described in detail here.

[0072] Furthermore, the target feature tensor generated from the bird's-eye view can be used for task decoding, including but not limited to road recognition, fault detection, and path planning, thereby improving the control accuracy of the vehicle and providing a more efficient, robust, and easier-to-deploy new paradigm for vehicle-side spatiotemporal fusion perception tasks.

[0073] In summary, the bird's-eye view feature generation method provided in this embodiment of the invention, by introducing dual attention of spatial attention and semantic attention, not only significantly reduces redundant computation, but also suppresses overexposure or underexposure through spatial attention, and enhances the high coupling with semantic categories, thereby improving the robustness of the generated target feature tensor under extreme lighting scenarios such as strong light, rain and fog, while avoiding noise interference, thus improving the accuracy and efficiency of target feature tensor generation in bird's-eye view.

[0074] Example 2

[0075] Based on the above-described method embodiments, this invention provides another method for generating bird's-eye view features. This method, under the dual attention of spatial attention and semantic attention, details the generation process of the target feature tensor. For example... Figure 3 As shown, the method includes the following steps:

[0076] (1) Acquire multiple images simultaneously captured by all vehicle cameras at the current moment.

[0077] (2) Preprocess multiple acquired images.

[0078] (3) Ray projection is performed on multiple acquired images using the BEV encoder to generate a bird's-eye view and initial feature tensors;

[0079] (4) For any bird’s-eye view grid, firstly calculate the offset of the bird’s-eye view grid through the 3-level pyramid features, then learn the offset and corresponding weights based on backpropagation, calculate the spatial attention coefficient of the bird’s-eye view grid, and finally determine the spatial weight tensor based on the spatial attention coefficients of multiple bird’s-eye view grids.

[0080] (5) Perform global average pooling and global max pooling on the initial feature tensor, generate channel weight vectors through a preset function, and finally generate semantically reweighted feature tensors based on the channel weight vectors and the initial feature tensor.

[0081] (6) Generate the fusion feature tensor at the current time based on the spatial weight tensor, the semantic reweighted feature tensor and the initial feature tensor.

[0082] (7) Align the fused feature tensor with the historical feature tensor of the previous moment to generate the target feature tensor of the bird's-eye view at the current moment.

[0083] It should be noted that the specific process of each of the above steps can be referred to the foregoing embodiments, and the embodiments of the present invention will not be described in detail here.

[0084] The bird's-eye view feature generation method provided in this invention generates a fused feature tensor through dual attention of spatial attention and semantic attention. This not only significantly reduces redundant computation, but also suppresses overexposure or underexposure through spatial attention, and enhances the high coupling with semantic categories. This improves the robustness of generating target feature tensors under extreme lighting conditions such as strong light, rain, and fog, while avoiding noise interference, thereby improving the accuracy and efficiency of generating target feature tensors for bird's-eye views.

[0085] Example 3

[0086] Corresponding to the above method embodiments, this invention also provides a bird's-eye view feature generation device, such as... Figure 4 As shown, the device includes: an image acquisition module 41, a bird's-eye view determination module 42, a fusion feature tensor generation module 43, and a target feature tensor generation module 44; the functions of each module are as follows:

[0087] Image acquisition module 41 is used to acquire multiple images simultaneously captured by all on-board cameras of the vehicle at the current moment;

[0088] The bird's-eye view determination module 42 is used to determine the bird's-eye view and its initial feature tensor based on multiple acquired images;

[0089] The fusion feature tensor generation module 43 is used to generate the fusion feature tensor at the current moment based on the bird's-eye view, the initial feature tensor, and the preset attention; wherein, the preset attention includes spatial attention calculation and semantic attention;

[0090] The target feature tensor generation module 44 is used to obtain the historical feature tensor of the previous moment and generate the target feature tensor of the bird's-eye view at the current moment based on the historical feature tensor and the fused feature tensor.

[0091] The bird's-eye view feature generation device provided in this embodiment of the invention generates a fused feature tensor through dual attention of spatial attention and semantic attention. This not only significantly reduces redundant computation, but also suppresses overexposure or underexposure through spatial attention, and enhances the high coupling with semantic categories. This improves the robustness of generating target feature tensors under extreme lighting scenarios such as strong light, rain, and fog, while avoiding noise interference, thereby improving the accuracy and efficiency of generating target feature tensors for bird's-eye views.

[0092] Optionally, the fusion feature tensor generation module 43 is further configured to: determine the spatial weight tensor based on the bird's-eye view and spatial attention; determine the semantic reweighted feature tensor based on the initial feature tensor and semantic attention; and generate the fusion feature tensor at the current moment based on the spatial weight tensor, the semantic reweighted feature tensor, and the initial feature tensor.

[0093] Optionally, the bird's-eye view includes multiple bird's-eye view grids; determining the spatial weight tensor based on the bird's-eye view and spatial attention includes: for any bird's-eye view grid, obtaining its center position information, and calculating the offset of the bird's-eye view grid based on preset pyramid features; learning the offset and corresponding weights based on backpropagation, and calculating the spatial attention coefficient of the bird's-eye view grid; determining the spatial weight tensor based on the spatial attention coefficients of multiple bird's-eye view grids.

[0094] Optionally, determining the semantically reweighted feature tensor based on the initial feature tensor and semantic attention includes: performing global pooling on the initial feature tensor and generating a channel weight vector through a preset function; wherein, the global pooling process includes: global average pooling and global max pooling; and generating the semantically reweighted feature tensor based on the channel weight vector and the initial feature tensor.

[0095] Optionally, generating the fusion feature tensor at the current time based on the spatial weight tensor, the semantic reweighted feature tensor, and the initial feature tensor includes: generating the concatenated feature tensor at the current time based on the spatial weight tensor and its corresponding first weight, the semantic reweighted feature tensor and its corresponding second weight; wherein the sum of the first weight and the second weight is 1; and generating the fusion feature tensor at the current time based on the concatenated feature tensor and the initial feature tensor.

[0096] Optionally, generating the fused feature tensor at the current time based on the spliced ​​feature tensor and the initial feature tensor includes: summing the residuals of the spliced ​​feature tensor and the initial feature tensor to obtain the summation result; and performing regularization processing on the summation result to generate the fused feature tensor at the current time.

[0097] Optionally, the target feature tensor generation module 44 is further configured to: compensate the historical feature tensor to obtain the compensated historical feature tensor; wherein the compensation includes: pose compensation and rotation compensation; and based on deformable self-attention, fuse the compensated historical feature tensor and the fused feature tensor to generate the target feature tensor of the bird's-eye view at the current moment.

[0098] The bird's-eye view feature generation device provided in this embodiment of the invention has the same technical features as the bird's-eye view feature generation method provided in the above embodiments, so it can also solve the same technical problems and achieve the same technical effects.

[0099] This invention also provides an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above-described bird's-eye view feature generation method.

[0100] See Figure 5As shown, the electronic device includes a processor 100 and a memory 101. The memory 101 stores machine-executable instructions that can be executed by the processor 100. The processor 100 executes the machine-executable instructions to implement the above-described bird's-eye view feature generation method.

[0101] Furthermore, Figure 5 The electronic device shown also includes a bus 102 and a communication interface 103, with the processor 100, the communication interface 103 and the memory 101 connected via the bus 102.

[0102] The memory 101 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 103 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network. The bus 102 may be an ISA (Industrial Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Enhanced Industry Standard Architecture) bus. These buses can be categorized as address buses, data buses, and control buses. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0103] Processor 100 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 100 or by instructions in software form. Processor 100 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 101, and the processor 100 reads the information from memory 101 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0104] This embodiment also provides a machine-readable storage medium storing machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the above-described bird's-eye view feature generation method.

[0105] The computer program products of the bird's-eye view feature generation method, apparatus and electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0108] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0110] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating features in a bird's-eye view, characterized in that, The method includes: Acquire multiple images simultaneously captured by all onboard cameras of this vehicle at the current moment; Determine the bird's-eye view and its initial feature tensor based on multiple acquired images; The fused feature tensor for the current moment is generated based on the bird's-eye view, the initial feature tensor, and the preset attention; wherein, the preset attention includes spatial attention calculation and semantic attention; Obtain the historical feature tensor of the previous moment, and generate the target feature tensor of the bird's-eye view at the current moment based on the historical feature tensor and the fused feature tensor. The step of generating the fused feature tensor at the current moment based on the bird's-eye view, the initial feature tensor, and the preset attention includes: determining a spatial weight tensor based on the bird's-eye view and the spatial attention; determining a semantically reweighted feature tensor based on the initial feature tensor and the semantic attention; and generating the fused feature tensor at the current moment based on the spatial weight tensor, the semantically reweighted feature tensor, and the initial feature tensor. The bird's-eye view includes multiple bird's-eye view grids; the step of determining the spatial weight tensor based on the bird's-eye view and the spatial attention includes: for any bird's-eye view grid, obtaining its center position information, and calculating the offset of the bird's-eye view grid based on preset pyramid features; learning the offset and the corresponding weight based on backpropagation, and calculating the spatial attention coefficient of the bird's-eye view grid; determining the spatial weight tensor based on the spatial attention coefficients of multiple bird's-eye view grids. The step of determining the semantically reweighted feature tensor based on the initial feature tensor and the semantic attention includes: performing global pooling on the initial feature tensor and generating a channel weight vector through a preset function; wherein, the global pooling includes: global average pooling and global max pooling; and generating the semantically reweighted feature tensor based on the channel weight vector and the initial feature tensor.

2. The method according to claim 1, characterized in that, The step of generating the fused feature tensor at the current time based on the spatial weight tensor, the semantically reweighted feature tensor, and the initial feature tensor includes: The concatenated feature tensor for the current time step is generated based on the spatial weight tensor and its corresponding first weight, the semantic reweighted feature tensor and its corresponding second weight; wherein the sum of the first weight and the second weight is 1. The fused feature tensor for the current moment is generated based on the spliced ​​feature tensor and the initial feature tensor.

3. The method according to claim 2, characterized in that, The step of generating the fused feature tensor at the current time based on the concatenated feature tensor and the initial feature tensor includes: The residuals of the spliced ​​feature tensor and the initial feature tensor are summed to obtain the summation result. The summation result is regularized to generate the fused feature tensor at the current time.

4. The method according to claim 1, characterized in that, The step of generating the target feature tensor of the bird's-eye view at the current moment based on the historical feature tensor and the fused feature tensor includes: The historical feature tensor is compensated to obtain the compensated historical feature tensor; wherein, the compensation includes: pose compensation and rotation compensation; Based on deformable self-attention, the compensated historical feature tensor and the fused feature tensor are fused to generate the target feature tensor of the bird's-eye view at the current moment.

5. A bird's-eye view feature generation device, characterized in that, The device includes: The image acquisition module is used to acquire multiple images simultaneously captured by all the vehicle's onboard cameras at the current moment; The bird's-eye view determination module is used to determine the bird's-eye view and its initial feature tensor based on multiple acquired images; A fusion feature tensor generation module is used to generate a fusion feature tensor at the current moment based on the bird's-eye view, the initial feature tensor, and a preset attention; wherein, the preset attention includes spatial attention calculation and semantic attention; The target feature tensor generation module is used to obtain the historical feature tensor of the previous moment, and generate the target feature tensor of the bird's-eye view at the current moment based on the historical feature tensor and the fused feature tensor. The process of generating the fused feature tensor at the current moment based on the bird's-eye view, the initial feature tensor, and the preset attention includes: determining a spatial weight tensor based on the bird's-eye view and the spatial attention; determining a semantically reweighted feature tensor based on the initial feature tensor and the semantic attention; and generating the fused feature tensor at the current moment based on the spatial weight tensor, the semantically reweighted feature tensor, and the initial feature tensor. The bird's-eye view includes multiple bird's-eye view grids; determining the spatial weight tensor based on the bird's-eye view and the spatial attention includes: for any bird's-eye view grid, obtaining its center position information, and calculating the offset of the bird's-eye view grid based on preset pyramid features; learning the offset and the corresponding weight based on backpropagation, and calculating the spatial attention coefficient of the bird's-eye view grid; determining the spatial weight tensor based on the spatial attention coefficients of multiple bird's-eye view grids. Determining a semantically reweighted feature tensor based on the initial feature tensor and the semantic attention includes: performing global pooling on the initial feature tensor and generating a channel weight vector through a preset function; wherein the global pooling process includes: global average pooling and global max pooling; and generating the semantically reweighted feature tensor based on the channel weight vector and the initial feature tensor.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the method described in any one of claims 1-4.

Images