Bird-eye view feature generation method and device and electronic equipment

Abstract

The invention provides an aerial view feature generation method and device and electronic equipment. The method comprises the following steps: acquiring a plurality of acquisition images synchronously acquired by all vehicle-mounted cameras of a vehicle at the current moment; determining a bird's-eye view and an initial feature tensor thereof according to the plurality of collected images; generating a fusion feature tensor at the current moment according to the aerial view, the initial feature tensor and preset attention; and generating a target feature tensor of the aerial view at the current moment according to the historical feature tensor at the previous moment and the fused feature tensor. According to the generation mode, the fusion feature tensor is generated through the double attention of the spatial attention and the semantic attention, redundant calculation is remarkably reduced, overexposure or dark channel passing is inhibited through the spatial attention, high coupling with semantic categories is enhanced, and the accuracy of the fusion feature tensor is improved. Therefore, the robustness of generating the target feature tensor in extreme illumination scenes such as strong light, rain and fog is improved, noise interference is avoided, and the target feature tensor generation precision and generation efficiency of the aerial view are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a bird's-eye view feature generation method and device and electronic equipment. BACKGROUND

[0002] In the field of automatic driving visual perception, bird's-eye view (BEV) is rapidly becoming a technical hotspot. Among them, the bird's-eye view uniformly projects the multi-view images captured by the camera onto a scale-distorted planar coordinate system, so it can intuitively present the lane topology, passable area and the position, size and orientation of surrounding traffic participants in three-dimensional space, etc. This advantage of "one map to see the whole picture" makes the bird's-eye view become an efficient intermediary for communication between road perception, trajectory prediction and planning.

[0003] In recent years, with the introduction of multi-view Transformer structure, sparse convolutional network and deformable attention mechanism into pure visual bird's-eye view perception system in succession, the performance of bird's-eye view has been greatly improved. However, the existing bird's-eye view system mostly focuses on aligning multi-camera features in spatial geometry, resulting in the following problems in the feature generation process of the bird's-eye view: ①redundant information problem; since the perception range of multi-view cameras may have a large overlap area, the features of different view cameras will repeatedly calculate the same target information when fused, which is manifested as feature redundancy, resulting in increased model complexity, slower training convergence, and possibly negatively affecting the accuracy of downstream tasks such as target detection or segmentation; ②noise interference problem; mainly including camera view angle difference problem and data sampling error problem, wherein the camera view angle difference problem mainly refers to the difference in feature quality caused by factors such as light conditions and occlusion in different view angles, and the data sampling error problem refers to the fact that depth estimation error or camera calibration error may introduce noise in the process of projecting features into BEV space, resulting in low-quality or incorrect features mixed into the fusion result, affecting the overall reliability of the bird's-eye view system; ③weight allocation limitation problem; the weight allocation (such as simple average fusion or static weighted fusion) in the existing technical solutions cannot dynamically adjust the importance of features according to specific scenarios, and cannot highlight key view angles or features, resulting in insufficient robustness in small target detection and extreme light conditions such as rain, fog and strong light. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a bird's-eye view feature generation method, device and electronic equipment to alleviate at least some of the above technical problems.

[0005] In a first aspect, the embodiments of the present application provide a bird's eye view feature generation method, which comprises: acquiring a plurality of acquisition images synchronously collected by all vehicle-mounted cameras of a vehicle at a current time; determining a bird's eye view and an initial feature tensor according to the plurality of acquisition images; generating a fusion feature tensor at the current time according to the bird's eye view, the initial feature tensor and preset attention; wherein the preset attention comprises spatial attention calculation and semantic attention; acquiring a historical feature tensor at a previous time, and generating a target feature tensor of the bird's eye view at the current time according to the historical feature tensor and the fusion feature tensor.

[0006] Optionally, the step of generating the fusion feature tensor at the current time according to the bird's eye view, the initial feature tensor and the preset attention comprises: determining a spatial weight tensor according to the bird's eye view and the spatial attention; determining a semantic re-weighted feature tensor according to the initial feature tensor and the semantic attention; and generating the fusion feature tensor at the current time according to the spatial weight tensor, the semantic re-weighted feature tensor and the initial feature tensor.

[0007] Optionally, the bird's eye view comprises a plurality of bird's eye view grids; the step of determining the spatial weight tensor according to the bird's eye view and the spatial attention comprises: for any bird's eye view grid, acquiring center position information thereof, and calculating an offset of the bird's eye view grid according to a preset pyramid feature; calculating a spatial attention coefficient of the bird's eye view grid based on the offset and a corresponding weight learned by back propagation; and determining the spatial weight tensor according to the spatial attention coefficients of the plurality of bird's eye view grids.

[0008] Optionally, the step of determining the semantic re-weighted feature tensor according to the initial feature tensor and the semantic attention comprises: performing global pooling processing on the initial feature tensor, and generating a channel weight vector by a preset function; wherein the global pooling processing comprises global average pooling processing and global maximum pooling processing; and generating the semantic re-weighted feature tensor according to the channel weight vector and the initial feature tensor.

[0009] Optionally, the step of generating the fusion feature tensor at the current time according to the spatial weight tensor, the semantic re-weighted feature tensor and the initial feature tensor comprises: generating a splicing feature tensor at the current time according to the spatial weight tensor and a corresponding first weight, the semantic re-weighted feature tensor and a corresponding second weight; wherein the sum of the first weight and the second weight is 1; and generating the fusion feature tensor at the current time according to the splicing feature tensor and the initial feature tensor.

[0010] Optionally, the step of generating the fusion feature tensor at the current time according to the splicing feature tensor and the initial feature tensor comprises: performing residual sum on the splicing feature tensor and the initial feature tensor to obtain a sum result; and performing regularization processing on the sum result to generate the fusion feature tensor at the current time.

[0011] Optionally, the step of generating the target feature tensor of the bird's eye view at the current moment according to the historical feature tensor and the fused feature tensor comprises: compensating the historical feature tensor to obtain a compensated historical feature tensor; wherein the compensation comprises: pose compensation and rotation compensation; and fusing the compensated historical feature tensor and the fused feature tensor based on deformable self-attention to generate the target feature tensor of the bird's eye view at the current moment.

[0012] In a second aspect, the embodiments of the present application further provide a bird's eye view feature generation device, which comprises:

[0013] An image acquisition module is configured to acquire a plurality of acquisition images synchronously collected by all vehicle-mounted cameras of the vehicle at a current moment.

[0014] A bird's eye view determination module is configured to determine a bird's eye view and an initial feature tensor thereof according to the plurality of acquisition images.

[0015] A fused feature tensor generation module is configured to generate a fused feature tensor at the current moment according to the bird's eye view, the initial feature tensor and a preset attention; wherein the preset attention comprises spatial attention calculation and semantic attention.

[0016] A target feature tensor generation module is configured to acquire a historical feature tensor at a previous moment, and generate a target feature tensor of the bird's eye view at the current moment according to the historical feature tensor and the fused feature tensor.

[0017] In a third aspect, the embodiments of the present application further provide an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method of the first aspect when executing the computer program.

[0018] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program, and the computer program is executable on the processor to execute the steps of the method of the first aspect.

[0019] The embodiments of the present application have the following beneficial effects:

[0020] The embodiment of the present application provides a bird's eye view feature generation method, device and electronic equipment, multiple acquisition images synchronously collected by all vehicle cameras of a vehicle at a current moment are acquired; a bird's eye view and an initial feature tensor thereof are determined according to the multiple acquisition images; a fusion feature tensor at the current moment is generated according to the bird's eye view, the initial feature tensor and preset attention; wherein the preset attention includes spatial attention calculation and semantic attention; a historical feature tensor at a previous moment is acquired, and a target feature tensor of the bird's eye view at the current moment is generated according to the historical feature tensor and the fusion feature tensor. The above generation mode generates the fusion feature tensor through double attention of the spatial attention and the semantic attention, not only significantly reduces redundant calculation, but also suppresses overexposure or overdarkness through the spatial attention, enhances high coupling with a semantic category, thereby improving the robustness of generating the target feature tensor under an extreme illumination scene such as strong light, rain and fog, meanwhile, noise interference is avoided, and then the target feature tensor generation precision and generation efficiency of the bird's eye view are improved.

[0021] Other features and advantages of the present application will be set forth in the descriptions below, and in part will become apparent to those skilled in the art, and will be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by the structures particularly pointed out in the description and the appended drawings.

[0022] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0024] Figure 1 A flow chart of a bird's eye view feature generation method provided by the embodiment of the present application;

[0025] Figure 2 A curve diagram of a Sigmoi function provided by the embodiment of the present application;

[0026] Figure 3 A flow chart of another bird's eye view feature generation method provided by the embodiment of the present application;

[0027] Figure 4 A schematic diagram of a bird's eye view feature generation device provided by the embodiment of the present application;

[0028] Figure 5A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0030] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0031] Embodiment one

[0032] The embodiment of the present application provides a bird's eye view feature generation method, which mainly relates to multi-camera visual perception technology in intelligent vehicles, ADAS (Advanced Driving Assistance System, Advanced Driving Assistance System) and AD (Autonomous Driving, Autonomous Driving), and can be widely applied to passenger cars, commercial vehicles, logistics trucks, low-speed unmanned vehicles in parks and roadside multi-sensor fusion systems.

[0033] As shown in Figure 1 The method comprises the following steps:

[0034] In step S102, a plurality of acquisition images synchronously acquired by all vehicle-mounted cameras of the vehicle at the current time are acquired.

[0035] In actual application, the vehicle is usually equipped with a plurality of vehicle-mounted cameras (for example, including front view, side view and rear view cameras), and the electronic device can acquire the acquisition images synchronously acquired by the plurality of vehicle-mounted cameras in real time or periodically during the driving of the vehicle. Here, taking the current time as an example, the plurality of acquisition images are images of the vehicle (taking the vehicle as an example) at the current time from multiple perspectives, and then the image information is converted from the image control to the BEV space, so as to simplify the complex three-dimensional environment around the vehicle into a two-dimensional image (i.e. a bird's eye view). For example, the bird's eye view can display the environment around the vehicle in all directions, including the front, rear, left and right environments, so as to facilitate fault detection and path planning during autonomous driving, and therefore, improving the feature generation precision and generation efficiency of the bird's eye view is of great significance to road perception and fault detection.

[0036] Further, the electronic device also pre-processes the plurality of collected images after obtaining the plurality of collected images. In the pre-processing process, first, linear distortion and exposure leveling are performed on the plurality of collected images for visual enhancement, and then the resolution is down-sampled to one quarter of the original size to reduce the memory and bandwidth overhead. Finally, through real-time CAN (Controller Area Network) bus and IMU (Inertial Measurement Unit) attitude, real-time fine adjustment of the external parameters between the camera coordinate system and the vehicle coordinate system is completed, so as to align the collected images collected by the plurality of vehicle-mounted cameras to the same vehicle coordinate system, thereby improving the accuracy of the collected images, avoiding noise interference problems caused by differences in camera viewing angles, and further improving the feature generation accuracy of the bird's eye view. It should be noted that the above pre-processing process can refer to the prior art, and the embodiments of the present application will not be described in detail.

[0037] In step S104, a bird's eye view and an initial feature tensor thereof are determined according to the plurality of collected images.

[0038] For the plurality of pre-processed collected images, the electronic device determines a bird's eye view and an initial feature tensor thereof according to the plurality of collected images, such as inputting the plurality of processed collected images into a BEV encoder, which can also be referred to as a Deformable Ray module. In actual application, the Deformable Ray module first divides four equally spaced depth layers in the viewing cone (i.e., a three-dimensional cone region with the camera optical center as the vertex, the edge frame of the image plane as the bottom surface, and extending forward to the maximum distance of interest) along each projection ray, discretizing the originally continuous space into four levels of "sampling planes"; then, on each depth layer, the surrounding pixels are dynamically resampled according to the learnable offset, and the corresponding features of the collected images from multiple perspectives are weighted and fused, so that by means of this "ray-by-ray, cross-perspective, hierarchical" aggregation strategy, the pixel information that is truly related to the grid position can be adaptively highlighted while ensuring spatial geometric consistency. It should be noted that the working principle of the specific BEV encoder can refer to the existing BEV encoder, and the embodiments of the present application will not be described in detail.

[0039] Therefore, the Deformable Ray module processes the plurality of pre-processed collected images to obtain a bird's eye view corresponding to the plurality of collected images at the current time. The bird's eye view generation process also performs rasterization, that is, the generated bird's eye view includes a plurality of bird's eye grids, and the size of the bird's eye grid can be set according to actual conditions. At the same time, by weighting the corresponding features of the plurality of collected images, an initial feature tensor F in of the bird's eye view can be obtained. inNot only the global layout of the environment around the vehicle is reserved, but also multi-scale depth semantics are combined, which lays a unified and high-quality foundation for subsequent initial feature generation of the target, thereby improving the feature generation accuracy of the bird's eye view.

[0040] Among them, for the multi-scale depth semantics, the multi-scale refers to the internal channel dimension of the initial feature tensor F in is not a homogeneous feature, but information of different scale layers are fused, which can describe both detailed texture (i.e. small target) and large area structure (such as road boundary); the depth semantics refers to that the channel vector contains "high semantic level" description such as color texture, target category, edge, drivability, dynamic and static attribute, rather than "shallow" features such as gray level or edge intensity.

[0041] Step S106, generating a fusion feature tensor of the current moment according to the bird's eye view, the initial feature tensor and the preset attention.

[0042] Among them, the preset attention includes spatial attention calculation and semantic attention; therefore, the fusion feature tensor is generated through the dual attention of spatial attention and semantic attention, which not only significantly reduces redundant calculation, but also suppresses overexposure or overdark road through spatial attention, enhances high coupling with semantic categories, thereby improving the robustness of generating target feature tensor under extreme illumination scenes such as strong light, rain and fog, while avoiding noise interference, and further improving the target feature tensor generation accuracy and generation efficiency of the bird's eye view.

[0043] Step S108, obtaining a historical feature tensor of the previous moment, and generating a target feature tensor of the bird's eye view of the current moment according to the historical feature tensor and the fusion feature tensor.

[0044] The bird's eye view feature generation method provided by the embodiment of the application obtains multiple acquisition images synchronously collected by all vehicle-mounted cameras of the vehicle at the current moment; determines a bird's eye view and an initial feature tensor thereof according to the multiple acquisition images; generates a fusion feature tensor of the current moment according to the bird's eye view, the initial feature tensor and a preset attention; and generates a target feature tensor of the bird's eye view of the current moment according to a historical feature tensor of the previous moment and the fusion feature tensor. The above generation method generates the fusion feature tensor through the dual attention of spatial attention and semantic attention, which not only significantly reduces redundant calculation, but also suppresses overexposure or overdark road through spatial attention, enhances high coupling with semantic categories, thereby improving the robustness of generating target feature tensor under extreme illumination scenes such as strong light, rain and fog, while avoiding noise interference, and further improving the target feature tensor generation accuracy and generation efficiency of the bird's eye view.

[0045] In an embodiment, the step of generating the fusion feature tensor of the current moment according to the bird's eye view, the initial feature tensor and the preset attention comprises: (A1) determining a spatial weight tensor according to the bird's eye view and the spatial attention; (A2) determining a semantic reweighted feature tensor according to the initial feature tensor and the semantic attention; and (A3) generating the fusion feature tensor of the current moment according to the spatial weight tensor, the semantic reweighted feature tensor and the initial feature tensor. The specific processes of (A1) to (A3) are as follows respectively:

[0046] (A1) The bird's eye view comprises a plurality of bird's eye view grids. The step of determining the spatial weight tensor according to the bird's eye view and the spatial attention comprises: for any bird's eye view grid, obtaining the center position information thereof, and calculating the offset of the bird's eye view grid according to the preset pyramid feature; calculating the spatial attention coefficient of the bird's eye view grid based on the offset and the corresponding weight learned by back propagation; and determining the spatial weight tensor according to the spatial attention coefficients of the plurality of bird's eye view grids.

[0047] Specifically, for any bird's eye view grid, the center position information is (x, y), where x and y represent the coordinate position of the center point of the bird's eye view grid in the two-dimensional bird's eye view, and the preset pyramid feature is a 3-level pyramid feature where p l represents the l-th layer pyramid feature map, where there are three layers, i.e. l = 0, 1, 2, and the resolution decreases layer by layer and the semantics increases layer by layer, and the center position information (x, y) is for the 3-level pyramid feature The offset Δp l,k of the bird's eye view grid is calculated as follows: where k = 1, 2, 3, 4, k represents a sampling point, i.e. the two-dimensional offset of the k-th sampling point in the l-th layer relative to the center position information (x, y), therefore, the offset is initialized as a positive cross, and for the center position information of any bird's eye view grid in the bird's eye view, 4 sampling points (up, down, left and right) are initialized on each pyramid feature, forming a positive cross template.

[0048] After the offset Δp l,k of the bird's eye view grid is determined, the spatial attention coefficient of the bird's eye view grid can be calculated based on the offset Δp l,k and the corresponding weight w l,k learned by back propagation, and the specific calculation formula of the spatial attention coefficient is as follows:

[0049]

[0050] where α s (x, y) represents the spatial attention coefficient, reflecting the final convergence feature strength of the bird's eye view grid, (x, y) represents the center position information of the bird's eye view grid, i.e. the horizontal and vertical coordinates of the center in the two-dimensional bird's eye plane coordinate system (XY coordinate system), and w l,k represents the offset Δpl,k corresponding weight, and Δp l,k pair-appeared learnable weight, F l denotes the pyramid feature map p l feature map obtained after geometric alignment such as rotation and translation, denotes the component of the offset in the X-axis, denotes the component of the offset in the Y-axis, l denotes the number of layers of the pyramid feature map, and k denotes the sampling points on each layer of the pyramid feature.

[0051] Therefore, the spatial attention coefficients of multiple bird's eye views can be obtained according to formula (1), and the spatial weight tensor α s In addition, by formula (1) above, the offset can be automatically adjusted so that the sampling points are focused on the pixels with real texture or motion edges along the same line-of-sight direction, and the line-of-sight direction is the direction of the ray from the camera to the center of the bird's eye view, and the aggregation along this direction helps to maintain geometric consistency; meanwhile, the learning weight w l,k The features containing motion traces or textures along the line of sight can be amplified, and smaller weights are given to meaningless areas, which improves the calculation accuracy of the spatial attention coefficients of the bird's eye view, thereby improving the accuracy of the fused feature tensor, and further improving the generation accuracy of the target feature tensor of the bird's eye view.

[0052] (A2) The step of determining the semantic reweighted feature tensor according to the initial feature tensor and the semantic attention, comprising: performing global pooling processing on the initial feature tensor, and generating a channel weight vector through a preset function; wherein the global pooling processing comprises: global average pooling processing and global maximum pooling processing; generating a semantic reweighted feature tensor according to the channel weight vector and the initial feature tensor.

[0053] Specifically, for the initial feature tensor F in of the bird's eye view, first, global average pooling (GAP) processing and global maximum pooling (GMP) processing are performed respectively, the two-dimensional feature is mapped to the channel domain to extract channel statistics, and then compressed and expanded through two one-dimensional fully connected layers, such as two fully connected layers with a compression rate r = 8, and then a channel weight vector α c is generated through a preset function to adjust the activation strength of the initial feature tensor, suppress redundant information and retain the differences between information.

[0054] wherein the preset function comprises a SiLU function and a Sigmoi function, and in actual application, the Sigmoi function compresses the output value to the interval (0, 1), such as Figure 2The output value of the Sigmoi function shown satisfies the interval (0, 1), realizes soft screening of the channel features, and suppresses irrelevant channels. It should be noted that the specific SiLU function and Sigmoi function can refer to the prior art, and the specific global pooling processing can refer to the prior art, and the embodiments of the present application will not be described in detail here.

[0055] In addition, the channel weight vector α c After generation, according to the channel weight vector α c And the initial feature tensor F in Generate a semantic reweighted feature tensor The calculation formula of the semantic reweighted feature tensor is as follows:

[0056]

[0057] Wherein, The semantic reweighted feature tensor is represented by α c The channel weight vector is represented by F in The initial feature tensor is represented by F

[0058] Therefore, by multiplying the initial feature tensor F c By the channel weight vector α in Generate a semantic reweighted feature tensor The purpose is to dynamically suppress noise channels and highlight channels related to key target categories, so that in extreme light, rainy night or long tail scenes, the features that can truly distinguish can still be focused on, thereby improving the generation accuracy and generation efficiency of the target feature tensor of the bird's eye view.

[0059] (A3) The step of generating the fusion feature tensor at the current moment according to the spatial weight tensor, the semantic reweighted feature tensor and the initial feature tensor, comprising: generating a splicing feature tensor at the current moment according to the spatial weight tensor and the corresponding first weight, the semantic reweighted feature tensor and the corresponding second weight; wherein the sum of the first weight and the second weight is 1; generating the fusion feature tensor at the current moment according to the splicing feature tensor and the initial feature tensor.

[0060] Specifically, the channels with spatial attention and the channels with semantic attention are also spliced, and the weights are adaptively allocated through a 1x1 convolutional gating unit. Here, the spatial weight tensor α s The corresponding first weight is β s The semantic reweighted feature tensor is represented by F The corresponding second weight is β c Then the calculation formula of the splicing feature tensor F out

[0061] ​

[0062] wherein β s + β c = 1. In practical applications, the first weight is β s and the second weight is β c , which can be learned.

[0063] In addition, the step of generating the fusion feature tensor at the current moment F out from the spliced feature tensor F in and the initial feature tensor F inal ; specifically, first, residual sum is performed on the spliced feature tensor F out and the initial feature tensor F in to obtain a sum result; then, the sum result is subjected to regularization processing to generate the fusion feature tensor F inal at the current moment, so that through the regularization processing, the spatial and semantic two-way information is dynamically balanced, and the calculation accuracy of the fusion feature tensor F inal is improved.

[0064] wherein the calculation formula of the fusion feature tensor F inal is as follows:

[0065] F inal = LN(F out +F in ) (4)

[0066] wherein LN represents regularization processing, and the regularization processing of the embodiments of the present application is preferably LayerNorm regularization, which can be adaptively adjusted according to actual conditions.

[0067] In summary, the fusion feature tensor is generated through the dual attention of spatial attention and semantic attention, which not only significantly reduces redundant calculation, but also suppresses overexposure or overdarkness through spatial attention, enhances high coupling with semantic categories, thereby improving the robustness of generating target feature tensors in extreme light scenes such as strong light, rain and fog, while avoiding noise interference, and further improving the target feature tensor generation accuracy and generation efficiency of the bird's eye view.

[0068] In one embodiment, the step of generating the target feature tensor of the bird's eye view at the current moment according to the historical feature tensor and the fusion feature tensor includes: compensating the historical feature tensor to obtain a compensated historical feature tensor; wherein the compensation includes: pose compensation and rotation compensation; and based on the deformable self-attention, fusing the compensated historical feature tensor and the fusion feature tensor to generate the target feature tensor of the bird's eye view at the current moment.

[0069] To offset the coordinate drift caused by the vehicle body motion, the embodiment of the present application compensates the historical feature tensor of the previous time, which includes pose compensation and rotation compensation, i.e. compensating the historical feature tensor of the previous time according to the compensation amount (Δx, Δy, Δθ), wherein Δx and Δy are the translation (or pose) compensation amount of the vehicle, expressed as the displacement of the vehicle mass center on the ground plane, and Δθ is the rotation angle of the vehicle body around the vertical axis, so as to obtain the compensated historical feature tensor F t-1 Here t represents the current time, and t-1 represents the previous time.

[0070] In addition, there is still a dynamic object motion difference or pose estimation error in the compensated historical feature tensor F t-1 , so a feature-level detail alignment is still needed. The embodiment of the present application uses a local window and a deformable self-attention for alignment; wherein the local window refers to searching for the corresponding key-value pair within a small window of 9x9 in the field of any query position in the bird's eye grid, and the deformable self-attention is based on the center of the local window, and introduces 4-8 learnable offsets to enable the attention to jump to more accurate semantic alignment points, for example, the top of the person in the previous frame and the top pixel in the current frame are not strictly the same grid.

[0071] Therefore, the historical feature tensor F t-1 and the fusion feature tensor F inal of the current time are fused through a deformable self-attention (Deformable Self-Attn) with a window size of 9, for example, taking the fusion feature tensor F inal of the current time as Query, and taking the features of the same window and their offset positions in the historical feature tensor F t-1 as Key and Value, and then obtaining the target feature tensor F t of the current time bird's eye view after alignment and time consistency through softmax to get the weight, weighting and summing the Value, and adding the Query residual, so as to effectively reduce the ghosting and drift caused by multi-frame superposition, and improve the generation accuracy of the target feature tensor. It should be noted that the specific processing process of the deformable self-attention can refer to the prior art, and the embodiment of the present application will not be described in detail here.

[0072] Further, after generating the target feature tensor of the bird's eye view, it can be used for task decoding, including but not limited to road recognition, fault detection and path planning, etc., thereby improving the control accuracy of the vehicle and providing a more efficient, more robust and easier-to-deploy new paradigm for vehicle end spatiotemporal fusion perception tasks.

[0073] In summary, the aerial view feature generation method provided by the embodiment of the present application introduces double attention of spatial attention and semantic attention, not only significantly reduces redundant calculation, but also suppresses overexposure or overdarkness channel through spatial attention, enhances high coupling with semantic categories, thereby improving the robustness of generating target feature tensor in extreme illumination scenes such as strong light, rain and fog, while avoiding noise interference, and further improving the target feature tensor generation accuracy and generation efficiency of the aerial view.

[0074] Embodiment two

[0075] On the basis of the above method embodiment, the embodiment of the present application provides another aerial view feature generation method, which details the generation process of the target feature tensor under the double attention of spatial attention and semantic attention. As shown in the formula (1), the method comprises the following steps: Figure 3

[0076] (1) Obtain multiple acquisition images synchronously collected by all vehicle-mounted cameras at the current time.

[0077] (2) Preprocess the multiple acquisition images.

[0078] (3) Perform ray projection on the multiple acquisition images through the BEV encoder to generate an aerial view and an initial feature tensor.

[0079] (4) For any aerial view grid, first calculate the offset of the aerial view grid through the 3-level pyramid feature, then calculate the spatial attention coefficient of the aerial view grid based on the back propagation learning offset and the corresponding weight, and finally determine the spatial weight tensor according to the spatial attention coefficients of multiple aerial view grids.

[0080] (5) Perform global average pooling processing and global maximum pooling processing on the initial feature tensor, and generate a channel weight vector through a preset function, and finally generate a semantic reweighted feature tensor according to the channel weight vector and the initial feature tensor.

[0081] (6) Generate a fusion feature tensor at the current time according to the spatial weight tensor, the semantic reweighted feature tensor and the initial feature tensor.

[0082] (7) Align the fusion feature tensor according to the historical feature tensor of the previous time to generate a target feature tensor of the aerial view at the current time.

[0083] It should be noted that the specific process of each step can refer to the foregoing embodiments, and the embodiment of the present application will not be described in detail here.

[0084] ​The bird's-eye view feature generation method provided in this invention generates a fused feature tensor through dual attention of spatial attention and semantic attention. This not only significantly reduces redundant computation, but also suppresses overexposure or underexposure through spatial attention, and enhances the high coupling with semantic categories. This improves the robustness of generating target feature tensors under extreme lighting conditions such as strong light, rain, and fog, while avoiding noise interference, thereby improving the accuracy and efficiency of generating target feature tensors for bird's-eye views.

[0085] Example 3

[0086] Corresponding to the above method embodiments, this invention also provides a bird's-eye view feature generation device, such as... Figure 4 As shown, the device includes: an image acquisition module 41, a bird's-eye view determination module 42, a fusion feature tensor generation module 43, and a target feature tensor generation module 44; the functions of each module are as follows:

[0087] Image acquisition module 41 is used to acquire multiple images simultaneously captured by all on-board cameras of the vehicle at the current moment;

[0088] The bird's-eye view determination module 42 is used to determine the bird's-eye view and its initial feature tensor based on multiple acquired images;

[0089] The fusion feature tensor generation module 43 is used to generate the fusion feature tensor at the current moment based on the bird's-eye view, the initial feature tensor, and the preset attention; wherein, the preset attention includes spatial attention calculation and semantic attention;

[0090] The target feature tensor generation module 44 is used to obtain the historical feature tensor of the previous moment and generate the target feature tensor of the bird's-eye view at the current moment based on the historical feature tensor and the fused feature tensor.

[0091] The bird's-eye view feature generation device provided in this embodiment of the invention generates a fused feature tensor through dual attention of spatial attention and semantic attention. This not only significantly reduces redundant computation, but also suppresses overexposure or underexposure through spatial attention, and enhances the high coupling with semantic categories. This improves the robustness of generating target feature tensors under extreme lighting scenarios such as strong light, rain, and fog, while avoiding noise interference, thereby improving the accuracy and efficiency of generating target feature tensors for bird's-eye views.

[0092] Optionally, the fusion feature tensor generation module 43 is further configured to: determine the spatial weight tensor based on the bird's-eye view and spatial attention; determine the semantic reweighted feature tensor based on the initial feature tensor and semantic attention; and generate the fusion feature tensor at the current moment based on the spatial weight tensor, the semantic reweighted feature tensor, and the initial feature tensor.

[0093] Optionally, the bird's eye view includes a plurality of bird's eye view grids; the spatial weight tensor is determined according to the bird's eye view and the spatial attention, including: for any bird's eye view grid, obtaining the center position information thereof, and calculating the offset of the bird's eye view grid according to a preset pyramid feature; the spatial attention coefficient of the bird's eye view grid is calculated based on the offset and the corresponding weight learned by back propagation; and the spatial weight tensor is determined according to the spatial attention coefficients of the plurality of bird's eye view grids.

[0094] Optionally, the semantic reweighted feature tensor is determined according to the initial feature tensor and the semantic attention, including: performing global pooling processing on the initial feature tensor, and generating a channel weight vector through a preset function; wherein the global pooling processing includes global average pooling processing and global maximum pooling processing; and the semantic reweighted feature tensor is generated according to the channel weight vector and the initial feature tensor.

[0095] Optionally, the fusion feature tensor of the current moment is generated according to the spatial weight tensor, the semantic reweighted feature tensor and the initial feature tensor, including: generating a spliced feature tensor of the current moment according to the spatial weight tensor and a corresponding first weight, the semantic reweighted feature tensor and a corresponding second weight; wherein the sum of the first weight and the second weight is 1; and the fusion feature tensor of the current moment is generated according to the spliced feature tensor and the initial feature tensor.

[0096] Optionally, the fusion feature tensor of the current moment is generated according to the spliced feature tensor and the initial feature tensor, including: performing residual sum on the spliced feature tensor and the initial feature tensor to obtain a sum result; and performing regularization processing on the sum result to generate the fusion feature tensor of the current moment.

[0097] Optionally, the target feature tensor generation module 44 is further configured to: compensate the historical feature tensor to obtain a compensated historical feature tensor; wherein the compensation includes pose compensation and rotation compensation; and fuse the compensated historical feature tensor and the fusion feature tensor based on deformable self-attention to generate the target feature tensor of the bird's eye view of the current moment.

[0098] The bird's eye view feature generation device provided by the embodiment of the application has the same technical features as the bird's eye view feature generation method provided by the above-mentioned embodiment, can solve the same technical problems, and achieve the same technical effects.

[0099] The embodiment of the application further provides an electronic device including a processor and a memory, the memory storing machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned bird's eye view feature generation method.

[0100] Referring to Figure 5As shown, the electronic device includes a processor 100 and a memory 101 storing machine executable instructions executable by the processor 100 to implement the bird's eye view feature generation method described above.

[0101] Further, Figure 5 As shown, the electronic device further includes a bus 102 and a communication interface 103, and the processor 100, the communication interface 103 and the memory 101 are connected through the bus 102.

[0102] The memory 101 can include a high-speed random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 102 can be an ISA (Industrial Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Enhanced Industry Standard Architecture) bus, etc. The above buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0103] The processor 100 can be an integrated circuit chip with processing capability. In implementation process, each step of the above method can be completed by integrated logic circuit of hardware in the processor 100 or instruction in the form of software. The processor 100 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiment of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiment of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101, and combines the hardware to complete the steps of the method of the above embodiment.

[0104] The embodiment also provides a machine readable storage medium, the machine readable storage medium stores machine executable instructions, when the machine executable instructions are called and executed by the processor, the machine executable instructions cause the processor to implement the above bird's eye view feature generation method.

[0105] The computer program product of the bird's eye view feature generation method, device and electronic equipment provided by the embodiment of the present application includes a computer readable storage medium storing program codes, and the instructions included in the program codes can be used to execute the method described in the foregoing method embodiment. The specific implementation can be referred to the method embodiment, and will not be described here.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above described system and device can refer to the corresponding process in the foregoing method embodiment, and will not be described here.

[0107] In addition, in the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be direct connection, can also be indirect connection through intervening medium, can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0108] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application or the part of the prior art or the part of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and various program code storage media.

[0109] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0110] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the same. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that any person skilled in the art can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features, within the technical scope disclosed by the present application. The modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An aerial view feature generation method, characterized by, The method comprises: acquiring multiple images synchronously collected by all vehicle-mounted cameras of a vehicle at a current time point; determining an aerial view and an initial feature tensor thereof according to the multiple images; generating a fusion feature tensor at the current time point according to the aerial view, the initial feature tensor, and preset attention; wherein the preset attention comprises spatial attention calculation and semantic attention; acquiring a historical feature tensor at a previous time point, and generating a target feature tensor of the aerial view at the current time point according to the historical feature tensor and the fusion feature tensor.

2. The method of claim 1, wherein, The step of generating a fusion feature tensor at the current time point according to the aerial view, the initial feature tensor, and preset attention comprises: determining a spatial weight tensor according to the aerial view and the spatial attention; determining a semantic re-weighted feature tensor according to the initial feature tensor and the semantic attention; generating the fusion feature tensor at the current time point according to the spatial weight tensor, the semantic re-weighted feature tensor, and the initial feature tensor.

3. The method of claim 2, wherein, The aerial view comprises multiple aerial view grids; the step of determining a spatial weight tensor according to the aerial view and the spatial attention comprises: for any aerial view grid, acquiring center position information thereof, and calculating an offset of the aerial view grid according to a preset pyramid feature; calculating a spatial attention coefficient of the aerial view grid based on back propagation learning of the offset and a corresponding weight; determining the spatial weight tensor according to spatial attention coefficients of multiple aerial view grids.

4. The method of claim 2, wherein, The step of determining a semantic re-weighted feature tensor according to the initial feature tensor and the semantic attention comprises: performing global pooling processing on the initial feature tensor, and generating a channel weight vector through a preset function; wherein the global pooling processing comprises global average pooling processing and global maximum pooling processing; generating the semantic re-weighted feature tensor according to the channel weight vector and the initial feature tensor.

5. The method of claim 2, wherein, The step of generating the fusion feature tensor at the current time point according to the spatial weight tensor, the semantic re-weighted feature tensor, and the initial feature tensor comprises: generating a splicing feature tensor at the current time point according to the spatial weight tensor and a corresponding first weight, the semantic re-weighted feature tensor and a corresponding second weight; wherein the sum of the first weight and the second weight is 1; generating the fusion feature tensor at the current time point according to the splicing feature tensor and the initial feature tensor.

6. The method of claim 5, wherein, The step of generating the fusion feature tensor at the current time point according to the splicing feature tensor and the initial feature tensor comprises: performing residual sum on the splicing feature tensor and the initial feature tensor to obtain a sum result; performing regularization processing on the sum result to generate the fusion feature tensor at the current time point.

7. The method of claim 1, wherein, The step of generating a target feature tensor of the aerial view at the current time point according to the historical feature tensor and the fusion feature tensor comprises: compensating the historical feature tensor to obtain a compensated historical feature tensor; wherein the compensation comprises pose compensation and rotation compensation; Based on deformable self-attention, the compensated history feature tensor and the fusion feature tensor are fused to generate a target feature tensor of the bird's eye view at the current time.

8. An aerial view feature generation apparatus characterized by comprising: The device comprises: An image acquisition module is configured to acquire a plurality of acquisition images synchronously collected by all vehicle-mounted cameras of the vehicle at the current time. A bird's eye view determination module is configured to determine a bird's eye view and an initial feature tensor thereof according to the plurality of acquisition images. A fusion feature tensor generation module is configured to generate a fusion feature tensor at the current time according to the bird's eye view, the initial feature tensor and a preset attention, wherein the preset attention comprises spatial attention calculation and semantic attention. A target feature tensor generation module is configured to acquire a history feature tensor at a previous time, and generate a target feature tensor of the bird's eye view at the current time according to the history feature tensor and the fusion feature tensor.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the method of any one of claims 1-7.

Images