Aerial view target detection method and system, storage medium and equipment

Abstract

The invention discloses an aerial view target detection method and system, a storage medium and equipment, and belongs to the field of BEV visual detection map reconstruction.The method comprises the steps that S1, images from different camera visual angles are input, and features of the different camera visual angles are extracted through a backbone network; s2, redesigning a time encoder through deformation and splicing by using long-period time information, and processing the characteristics of different camera visual angles through the designed time encoder; s3, based on a processing result of the step S2, further utilizing dynamic deformable attention to extract spatial features of the image; s4, processing the features obtained in the step S3 by using a feedforward network, and generating a final BEV feature map; and S5, performing target detection by using the final BEV feature map. According to the method, dynamic deformable attention is fused in aerial view angle detection, so that image query is more comprehensive, the calculation overhead is low, meanwhile, time sequence coding is improved, long-time historical information can be better integrated, and the robustness and accuracy of detection are improved.

Description

Technical Field

[0001] The present invention relates to the field of BEV visual detection map reconstruction, to technologies such as Transformer neural network, time series and spatial information fusion, and in particular to a bird's-eye view target detection method, system, storage medium and device. Background Art

[0002] Bird's-Eye-View (BEV) is a commonly used representation in autonomous driving. It converts the three-dimensional scene around the vehicle into a two-dimensional plane image, providing a "bird's-eye view" perspective from above. This representation can intuitively display the position of the vehicle in the environment and clearly show key information such as the distribution of surrounding objects, road structure, lane lines, etc. Therefore, BEV representation is very useful in autonomous driving tasks, especially in perception, planning and control.

[0003] In recent years, with the rapid development of autonomous driving and BEV visual detection technology, environmental perception tasks and the realization of high-precision autonomous driving systems have become key links. The existing BEV algorithm proposes a new framework called BEVFormer, which uses spatiotemporal Transformer to learn a unified BEV representation to support a variety of autonomous driving perception tasks. In the BEV algorithm, the use of temporal information is to better handle object perception and tracking tasks in dynamic environments. Using continuous temporal information, the BEV algorithm can effectively infer the motion state and trajectory of objects and handle complex scene problems such as occlusion. Especially in autonomous driving, vehicles need to perceive and predict the dynamic behavior of surrounding objects in real time during driving, rather than relying solely on the static information of a single frame image. Therefore, the integration of temporal information is crucial to improving the robustness and accuracy of the perception system. At present, the BEV algorithm uses a repetitive temporal self-attention mechanism to integrate historical BEV features. Its temporal encoder cannot make full use of long-term temporal information, and simply increasing the cycle step size cannot obtain additional performance improvements.

[0004] The traditional BEV algorithm uses a self-attention mechanism, which needs to pay attention to every position of the entire input feature map during the calculation process. Although this can capture global context information, the computational overhead increases sharply when the feature map resolution is high. In addition, the fixed receptive field limits the ability of the self-attention module to capture different scales or local features. Summary of the invention

[0005] The purpose of the present invention is to add a deformable attention mechanism to the existing BEV algorithm and integrate temporal information, thereby providing a bird's-eye view target detection method, system, storage medium and device.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] In a first aspect, a method for detecting a target from a bird's-eye view is provided, comprising the following steps:

[0008] S1, input images from different camera perspectives, and extract features from different camera perspectives through the backbone network;

[0009] S2, using long-period time information, redesigning the time encoder through deformation and splicing, and processing the features of the different camera perspectives through the designed time encoder;

[0010] S3, based on the processing result of step S2, further using dynamic deformable attention to extract spatial features of the image;

[0011] S4, using the feedforward network to process the features obtained in step S3 to generate a final BEV feature map;

[0012] S5. Use the final BEV feature map to perform target detection.

[0013] In some possible embodiments based on the first aspect, the using of long-period time information includes:

[0014] Increase the time interval of the sequence.

[0015] In some possible embodiments based on the first aspect, the redesigning of the time encoder by deformation and splicing includes:

[0016] For the features of any frame image at the past moment, firstly, according to the self-motion transfer matrix T from the past moment to the current moment t k = [R|t]∈SE3, align the features of the past moment to the current moment; then concatenate the aligned features of multiple frames of historical moments and the features of the current moment in the channel dimension; finally, use the residual module to reduce the number of channels.

[0017] Preferably, bilinear interpolation is used to align features at past moments to the current moment.

[0018] Preferably, the features of the different camera perspectives are processed by a designed temporal encoder, including:

[0019] The improved time encoder is used to query and fuse the temporal information from the features of historical moments to obtain the features of different camera perspectives after the first correction.

[0020] Preferably, the step S3 specifically includes:

[0021] The features of different camera perspectives after the first correction are queried and the spatial domain information is fused from the features extracted in step S1 through the deformable attention module to obtain the features of different camera perspectives after the first correction.

[0022] Preferably, the step S4 specifically includes:

[0023] The features of different camera perspectives that have been corrected again are subjected to multiple rounds of FF calculations through a feedforward network.

[0024] In a second aspect, a bird's-eye view target detection system is provided, comprising:

[0025] A feature extraction module is used to receive images from different camera perspectives and extract features from different camera perspectives through a backbone network;

[0026] A time sequence information extraction module, used to utilize long-period time information to redesign a time encoder through deformation and splicing, and to process the features of the different camera perspectives through the designed time encoder;

[0027] A spatial information extraction module, which receives the output of the temporal feature extraction module and further uses dynamic deformable attention to extract the spatial features of the image;

[0028] A feedforward network module, used to generate a final BEV feature map using the features obtained in the feedforward network processing step S3;

[0029] The target detection module is used to perform target detection using the final BEV feature map.

[0030] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the bird's-eye view target detection method described in the first aspect is implemented.

[0031] In a fourth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, and the processor executes the bird's-eye view target detection method described in the first aspect when executing the computer instructions.

[0032] It should be further explained that the technical features corresponding to the above options can be combined or replaced with each other to form a new technical solution if there is no conflict.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention integrates dynamic deformable attention in bird's-eye view detection and avoids the shortcomings of traditional self-attention mechanism, such as complex calculation and fixed receptive field, by dynamically selecting a small number of key points to focus attention. It has higher computational efficiency and flexible receptive field, making the query image more comprehensive and with low computational overhead. At the same time, it improves temporal coding, can better integrate long-term historical information, make full use of long-period temporal information, and improve the robustness and accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flow chart of a bird's-eye view target detection method shown in an embodiment of the present invention;

[0036] Figure 2 The overall network structure diagram of the embodiment of the present invention is shown;

[0037] Figure 3 An improved time encoder shown as an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of dynamic deformable attention shown in an embodiment of the present invention;

[0039] Figure 5 A module diagram of a bird's-eye view target detection system shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various configurations. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] It should be noted that the defects existing in the solutions in the above-mentioned prior art are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above-mentioned problems and the solutions proposed in the embodiments of the present application for the above-mentioned problems below should all be the contributions made by the inventor to the present application in the process of invention and creation, and should not be understood as technical contents known to technical personnel in this field.

[0042] In view of the technical problems pointed out in the background technology, the embodiments provided by the present invention are as follows:

[0043] Embodiment 1

[0044] Reference Figure 1 In an exemplary embodiment, a method for detecting a target from a bird's-eye view is provided, comprising the following steps:

[0045] S1, input images from different camera perspectives, and extract features from different camera perspectives through the backbone network;

[0046] S2, using long-period time information, redesigning the time encoder through deformation and splicing, and processing the features of the different camera perspectives through the designed time encoder;

[0047] S3, based on the processing result of step S2, further using dynamic deformable attention to extract spatial features of the image;

[0048] S4, using the feedforward network to process the features obtained in step S3 to generate a final BEV feature map;

[0049] S5. Use the final BEV feature map to perform target detection.

[0050] Specifically, this method is improved based on the existing BEVFormer framework. On the one hand, in step S2, the temporal coding is improved and the time encoder is redesigned to better integrate long-term historical information, make full use of long-term temporal information, and improve the robustness and accuracy of detection. In step S3, dynamic deformable attention is used to replace the original self-attention mechanism. By dynamically selecting a small number of key points to focus attention, the shortcomings of the traditional self-attention mechanism, such as complex calculation and fixed receptive field, are avoided. It has higher computational efficiency and flexible receptive field, making the query image more comprehensive and the computational overhead lower. The overall structure of the improved network is as follows: Figure 2 shown.

[0051] Embodiment 2

[0052] Based on the inventive concept of Example 1, this example introduces a target detection method in detail, including the following contents.

[0053] In step S1, the autonomous driving scene in the NuScenes dataset is selected as the data input. The Nu Scenes dataset is a large-scale autonomous driving dataset that contains 1,000 autonomous driving scenes. Each scene lasts 20 seconds and covers different weather, time and traffic conditions. It is widely used in research in this field. ResNet-101 is selected as the backbone network, and the output is a multi-channel feature tensor after network processing. These tensors then enter the next step of processing. Specifically, images from multiple different camera perspectives are input, and features are extracted through the backbone network. Features are represented as feature sets from different camera perspectives. in, is the feature of the i-th view camera, N view is the total number of camera views.

[0054] In step S2, a simple feature deformation and splicing strategy is adopted to redesign the temporal encoder for time domain fusion. The improved temporal encoder is as follows: Figure 3 Specifically, for any frame of the BEV feature at the past moment, firstly according to the self-motion transfer matrix from the past moment to the current moment t Align (bilinear interpolation) to the current moment to get Then, the aligned multi-frame historical BEV features and the current moment BEV features are concatenated in the channel dimension, and finally the residual module is used to reduce the number of channels. In order to maintain a similar computational complexity to the original design, we use the same number of historical BEV features but increase the sampling interval.

[0055] By increasing the time interval of the sequence, we can better utilize the long-period temporal information. The new temporal encoder also opens the possibility of utilizing future BEV features in an offline 3D detection setting.

[0056] In step S3, the proposed dynamic deformable attention module can effectively expand the query domain of BEV features on the image, avoiding the shortcomings of high computational consumption, small receptive field and insufficient information exchange caused by large feature resolution. Figure 4 As shown, in order to make the query image more comprehensive, the query sample of the present invention is represented as The feature map of the network input is represented as in The normalized coordinates of each query reference point are represented as The dynamic deformable attention module is used for:

[0057]

[0058] Among them, x represents the input feature, P is the reference point, represents the offset from the reference point. The input feature level is represented by l, and the attention head is represented by m. k and K represent the number of sampling points and the total number of sampling points. P mlqk A represents the offset of the mth attention head and the lth feature layer at the kth sampling point. mlqk represents the attention weight at the same position, and its regularized weight can be expressed as

[0059]

[0060] In step S4, the feedforward network appears after the self-attention layer of each encoder and decoder. The main function of the feedforward network is to extract the complex features of the input data through linear transformation and activation function, thereby enhancing the model's nonlinear expression ability and ability to understand data; among them, multiple rounds of FF calculation refer to Figure 2 The process in the above is iterated for multiple rounds, and FF calculation is performed after each round of iterative adjustment. The result of FF calculation processing by the current round encoder is used as the input of the next round encoder, and so on. After multiple rounds of iterative adjustment, the BEV feature map at the current moment is obtained.

[0061] In step S5, the final bird's eye view (BEV) feature map processed by the feed-forward network is used as the input for the downstream 3D object detection or semantic segmentation task.

[0062] Specifically, combined Figure 2 Describe the overall feature processing process:

[0063] At time t, images from multiple viewpoint cameras on the vehicle are input to the backbone, and multi-scale features of each image are output: At the same time, we also need to keep BEVFeaturesB at past times such as tT and t-nT t-T 、BEV Features B t-nT Etc. Where T refers to the period length we selected later;

[0064] In the coding layer, BEV Queries Q is first used to pass the improved temporal encoder from BEVFeaturesB at the past moment t-T 、BEV Features B t-n T etc. and fuse the time series information to obtain the modified BEVQueriesQ';

[0065] Then in the same encoding layer, the BEV Queries Q' passed through the temporal module is queried and fused with spatial domain information from multi-scale features through the deformable attention module to obtain further BEV Queries Q";

[0066] The current coding layer performs FF calculation on the twice-corrected BEV features Q″ and then outputs it as the input of the next coding layer;

[0067] After stacking 6 layers in this way, i.e., after 6 rounds of fine-tuning, the unified BEV feature map at time t is generated;

[0068] Finally, the final BEV feature map is used as input to predict the perception result through 3D object detection (detection head) and semantic segmentation (map segmentation head).

[0069] Among them, FF calculation is done through a feedforward network ( Figure 2 The FeedForward layer is a fully connected neural network that contains two linear transformations and an activation function. The data after residual connection and normalization (Add&Norm) enters the FeedForward layer. Its main purpose is to further extract and deepen the features. The working principle is as follows: After receiving the input, the first linear transformation is performed to perform preliminary processing on the data. Then, the data is nonlinearly transformed through the ReLU activation function, because the data in the real world is often nonlinear. This is to capture more complex features of the input data. Finally, a second linear transformation is performed to generate the final output. Residual connection and normalization (Add&Norm) layer ( Figure 2 Add&Norm in the Residual Network). The residual network uses residual connections to allow the input information to directly cross one or more layers and be added to the output of the subsequent layers, thereby alleviating the gradient vanishing and gradient exploding problems in deep networks and allowing the network to be expanded to deeper layers. Normalization is a data preprocessing technique that aims to map the input data or the output data of a neural network layer to a specific numerical range or distribution through linear or nonlinear transformations. This method uses layer normalization, which normalizes the outputs of all neurons in the layer so that the output distribution has a stable mean and variance. Layer normalization normalizes the output after the residual connection, and then uses learnable parameters to scale and translate the normalized output. This can maintain the stability of the data distribution while retaining a certain degree of flexibility.

[0070] Embodiment 3

[0071] Based on the same inventive concept as that of the first embodiment, a bird's-eye view target angle detection system is provided, comprising:

[0072] A feature extraction module is used to receive images from different camera perspectives and extract features from different camera perspectives through a backbone network;

[0073] A time sequence information extraction module, used to utilize long-period time information to redesign a time encoder through deformation and splicing, and to process the features of the different camera perspectives through the designed time encoder;

[0074] A spatial information extraction module, which receives the output of the temporal feature extraction module and further uses dynamic deformable attention to extract the spatial features of the image;

[0075] A feedforward network module, used to generate a final BEV feature map using the features obtained in the feedforward network processing step S3;

[0076] The target detection module is used to use the final BEV features to perform target detection.

[0077] It should be noted that each module in the system implements the functions corresponding to each step of the aforementioned method, which will not be elaborated here.

[0078] Embodiment 4

[0079] Based on the same inventive concept as that of Embodiment 1, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the bird's-eye view target detection method provided by the embodiment of the present invention is implemented. Based on such an understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0080] Embodiment 5

[0081] Based on the same inventive concept as that of the first embodiment, a network device is provided, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and when the processor executes the computer instructions, the bird's-eye view target detection method provided by the embodiment of the present invention is executed.

[0082] The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0083] Embodiments of the subject matter and functional operations described in this specification may be implemented in: tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode and transmit information to a suitable receiver device for execution by the data processing device.

[0084] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0085] Processors suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, just to name a few.

[0086] It should be understood that each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] The above specific implementation methods are detailed descriptions of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A bird's-eye view target detection method, characterized in that: The following steps are involved: S1, input images from different camera perspectives, and extract features from different camera perspectives through the backbone network; S2, using long-period time information, redesigning the time encoder through deformation and splicing, and processing the features of the different camera perspectives through the designed time encoder; S3, based on the processing result of step S2, further using dynamic deformable attention to extract spatial features of the image; S4, using the feedforward network to process the features obtained in step S3 to generate a final BEV feature map; S5. Use the final BEV feature map to perform target detection.

2. The bird's-eye view target detection method according to claim 2, characterized in that: The time information using a long period includes: Increase the time interval of the sequence.

3. The bird's-eye view target detection method according to claim 1, characterized in that: The time encoder is redesigned by deformation and splicing, including: For the features of any frame image at the past moment, firstly, according to the self-motion transfer matrix T from the past moment to the current moment t k = [R|t]∈SE3, align the features of the past moment to the current moment; then concatenate the aligned features of multiple frames of historical moments and the features of the current moment in the channel dimension; finally, use the residual module to reduce the number of channels.

4. The bird's-eye view target detection method according to claim 2, characterized in that: Bilinear interpolation is used to align the features of the past moment to the current moment.

5. The bird's-eye view target detection method according to claim 3, characterized in that: The features of different camera perspectives are processed by the designed temporal encoder, including: The improved time encoder is used to query and fuse the temporal information from the features of historical moments to obtain the features of different camera perspectives after the first correction.

6. The bird's-eye view target detection method according to claim 5, characterized in that: The step S3 specifically includes: The features of different camera perspectives after the first correction are queried and the spatial domain information is fused from the features extracted in step S1 through the deformable attention module to obtain the features of different camera perspectives after the first correction.

7. The bird's-eye view target detection method according to claim 6, characterized in that: The step S4 specifically includes: The features of different camera perspectives that have been corrected again are subjected to multiple rounds of FF calculations through a feedforward network.

8. A bird's-eye view target detection system, characterized in that: include: A feature extraction module is used to receive images from different camera perspectives and extract features from different camera perspectives through a backbone network; A time sequence information extraction module, used to utilize long-period time information to redesign a time encoder through deformation and splicing, and to process the features of the different camera perspectives through the designed time encoder; A spatial information extraction module, which receives the output of the temporal feature extraction module and further uses dynamic deformable attention to extract the spatial features of the image; A feedforward network module, used to generate a final BEV feature map using the features obtained in the feedforward network processing step S3; The target detection module is used to perform target detection using the final BEV feature map.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the bird's-eye view target detection method described in any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, wherein: When the processor runs the computer instructions, it executes the bird's-eye view target detection method described in any one of claims 1-7.

Images