BEV-based target object detection method and system

Abstract

The invention provides a BEV-based target object detection method and system, and the method comprises the steps: obtaining a to-be-detected image and corresponding target point cloud data, carrying out the feature extraction of the to-be-detected target image and target point cloud data through a feature extraction network, and converting the features into BEV image features and BEV point cloud features, the BEV image features and the BEV point cloud features are subjected to feature enhancement through a deformable attention mechanism, so that the feature representation capability is improved; and finally, the BEV image features and the BEV point cloud features after feature enhancement are fused to be further used for target detection, so that more accurate target object detection can be realized, and the precision of target object detection is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and more specifically relates to a BEV-based target object detection method and system. Background Art

[0002] With the rapid development of computer vision, many novel methods have been proposed to solve the problem of multi-sensor-centric multimodal unmanned driving bird's eye view (BEV) perception. In this perspective, the vehicle can "see" the surrounding road conditions from a bird's eye view, fusing data from different sensors into a unified top-down view. The bird's eye view image not only intuitively shows the distribution and location of elements such as roads, vehicles, and pedestrians, but also contains rich semantic information, such as lane lines, traffic signs, and traffic lights.

[0003] As an important technology in the field of autonomous driving, the core technology of bird's-eye view perception lies in the deep fusion of data from different sensors, including images captured by high-definition cameras, precise point cloud data generated by lidar, and distance and speed information provided by millimeter-wave radar, from the unique perspective of a bird's-eye view. In autonomous driving technology, there are differences in data collection, processing and time synchronization between different sensors, which requires solving the resulting inconsistency problems during data fusion to ensure the accuracy and reliability of the fusion results. In addition, autonomous driving vehicles need to cope with complex and changing road environments, including bad weather, night driving, and interference from obstructions. These factors place extremely high demands on sensor performance and the robustness of fusion algorithms.

[0004] However, in the prior art, there are still problems that the camera cannot accurately detect the target object and predict the future trajectory of the target object when it is moving at high speed during autonomous driving, and the target object detection is unclear. Summary of the invention

[0005] In order to solve the problem that the camera cannot accurately detect the target object and predict the future trajectory of the target object when moving at high speed during autonomous driving, and the problem that the target object detection is unclear, the present invention proposes a target object detection method and system based on BEV to improve the effect of autonomous driving target detection and prediction.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] A target object detection method based on BEV includes the following steps:

[0008] Acquire the image to be detected and the corresponding target point cloud data;

[0009] Performing target object feature recognition on the target image to be detected through a first feature extraction network and converting the target object feature into a BEV image feature;

[0010] Extract radar point cloud features from the target point cloud data through a second feature extraction network and convert the radar point cloud features into BEV point cloud features;

[0011] Performing feature enhancement on the BEV image features and the BEV point cloud features based on a deformable attention mechanism;

[0012] The enhanced BEV image features and BEV point cloud features are fused to obtain fused features;

[0013] The fused features are used for target detection to obtain a detection result.

[0014] As a preferred solution, the present invention also provides a target object detection system based on BEV, comprising:

[0015] A data acquisition module is used to acquire the image to be detected and the corresponding target point cloud data;

[0016] A first feature extraction module, used for identifying target object features of the target image to be detected through a first feature extraction network, and converting the features into BEV image features;

[0017] A second feature extraction module, used for extracting radar point cloud features from the target point cloud data through a second feature extraction network, and converting the radar point cloud features into BEV point cloud features;

[0018] A feature enhancement module, used for performing feature enhancement on the BEV image features and the BEV point cloud features based on a deformable attention mechanism;

[0019] A fusion module is used to fuse the BEV image features and BEV point cloud features after feature enhancement to obtain fusion features;

[0020] Result output module: used to perform target detection on the fusion features to obtain detection results.

[0021] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0022] The present invention proposes a target object detection method and system based on BEV. First, an image to be detected and corresponding target point cloud data are obtained. Feature extraction is performed on the target image and target point cloud data to be detected through a feature extraction network and they are converted into BEV image features and BEV point cloud features respectively. The BEV image features and BEV point cloud features are enhanced through a deformable attention mechanism, so as to improve the feature representation capability and predict the next trajectory of the target object. Finally, the feature-enhanced BEV image features and BEV point cloud features are fused and further used for target detection, so as to achieve more accurate target object detection and improve the accuracy of target object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The flowchart of the target object detection method based on BEV is shown in an embodiment of the present invention.

[0024] Figure 2 The diagram is an architecture diagram of a BEV-based target object detection method according to an embodiment of the present invention.

[0025] Figure 3 The present invention is a flowchart of adjusting the camera posture according to an embodiment of the present invention.

[0026] Figure 4 The flowchart of obtaining BEV image features according to an embodiment of the present invention is shown.

[0027] Figure 5 This is a flow chart of obtaining BEV point cloud features according to an embodiment of the present invention.

[0028] Figure 6 This is a diagram of the spatiotemporal pyramid prediction model architecture shown in an embodiment of the present invention.

[0029] Figure 7 The feature fusion flow chart of BEV image features and BEV point cloud features shown in an embodiment of the present invention.

[0030] Figure 8 The overall flow chart of BEV-based target object detection according to an embodiment of the present invention is shown.

[0031] Fig. 9 The architecture diagram of a BEV-based target object detection system is shown for an embodiment of the present invention. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0033] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0034] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0035] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Example 1

[0037] This embodiment proposes a target object detection method based on BEV, such as Figure 1 , 2 , which is a flow chart and architecture diagram of a target object detection method based on BEV of this embodiment.

[0038] The present embodiment proposes a method for detecting a target object based on BEV, including the following steps:

[0039] Acquire the image to be detected and the corresponding target point cloud data;

[0040] Performing target object feature recognition on the target image to be detected through a first feature extraction network and converting the target object feature into a BEV image feature;

[0041] Extract radar point cloud features from the target point cloud data through a second feature extraction network and convert the radar point cloud features into BEV point cloud features;

[0042] Performing feature enhancement on the BEV image features and the BEV point cloud features based on a deformable attention mechanism;

[0043] The enhanced BEV image features and BEV point cloud features are fused to obtain fused features;

[0044] The fused features are used for target detection to obtain a detection result.

[0045] This embodiment first obtains the image to be detected and the corresponding target point cloud data, extracts features of the target image to be detected and the target point cloud data through a feature extraction network and converts them into BEV image features and BEV point cloud features respectively, and enhances the BEV image features and BEV point cloud features through a deformable attention mechanism, thereby improving the feature representation capability and predicting the next trajectory of the target object; finally, the feature-enhanced BEV image features and BEV point cloud features are fused and further used for target detection, which can achieve more accurate target object detection and improve the accuracy of target object detection.

[0046] In an optional embodiment, the image and point cloud data to be detected are obtained from a nuScenes dataset.

[0047] The nuScenes dataset is the first large-scale dataset that provides data from the entire sensor suite of an autonomous vehicle (6 cameras, 1 LIDAR, 5 RADARs, GPS, inertial sensors). The dataset collects 1,000 driving scenes in Boston and Singapore.

[0048] In another optional embodiment, the image to be detected is collected using a camera; the posture of the camera is adjusted through inertial sensor data and visual data, and the inertial sensor data and visual data are brought into a visual inertial odometer model to obtain a posture trajectory for adjusting the posture of the camera.

[0049] This example trains a visual inertial odometry model based on a sliding window. The sliding window is only responsible for optimizing a certain number of frames of data. After the number of frames in the window reaches the set value, each new frame is thrown into the front frame of the window. However, the front frame cannot be thrown away directly because it contains valid observation data and information. Therefore, the previously observed information needs to be integrated into a priori information and put into the subsequent data optimization.

[0050] The system state quantity in the visual inertial odometry model is defined as follows:

[0051] X=[x f ,x f+1 ...x f+N ,λ s ,λs+1 ... s+M ]

[0052]

[0053] Where X is the camera’s position, λ is the observed landmark point; f represents the camera's position at the fth frame, λ s represents the sth landmark, that is, the sliding window starts from the sth landmark of the fth frame; N is the number of key frames in the sliding window, M is the number of landmarks observed by all key frames in the sliding window; X i Including the position p, posture q, velocity v of the inertial measurement unit body in the inertial coordinate system at time i, and the acceleration bias estimate b in the inertial measurement unit body coordinate system a and the angular velocity bias estimate w g ,in represents the acceleration and angular velocity of the camera at the i-th moment, represents the acceleration and angular velocity of the camera at the i-th moment, w represents the angular velocity, b represents the acceleration, z i represents the speed at the i-th moment, It is expressed as the estimated acceleration bias at the i-th moment in the body coordinate system of the inertial measurement unit, It is expressed as the estimate of the angular velocity bias at the i-th moment in the inertial measurement unit body coordinate system.

[0054] In situations such as fast motion or complex scenes, it is difficult to process camera data using purely visual sensors. By adjusting the camera posture through inertial sensors combined with visual information, the camera's position can be adjusted according to the movement of the vehicle, improving the accuracy and reliability of the data.

[0055] In an optional embodiment, the camera posture is adjusted by inertial sensor data and visual data, and the process is as follows: Figure 3 As shown, the following steps are included:

[0056] Step 1: Use a camera to capture a calibration image and extract feature points from the calibration image to obtain the two-dimensional coordinates (u, v) of the feature points; use the inverse projection of the feature points on the image as feature parameters and convert them into three-dimensional coordinates (x, y, z) in the camera coordinate system;

[0057] Step 2: Establish the coordinate relationship between the feature points in the normalized camera coordinate system and the camera coordinate system through inverse depth parameterization. The expression is:

[0058]

[0059] Among them, λ = 1 / z is the inverse depth parameter, that is, the inverse of the feature point depth;

[0060] Step 3: Based on the inverse depth of the feature points, i The observed value in the frame, the coordinates of the predicted feature point at the jth moment are:

[0061]

[0062] is the camera coordinate system coordinate of the feature point at the jth moment, T (b,c) is the external parameter from the camera coordinate system to the inertial measurement unit coordinate system, For the f i The transformation matrix from the inertial measurement unit coordinate system of the frame to the world coordinate system, For the f j The transformation matrix from the inertial measurement unit coordinate system of the frame to the world coordinate system; is the inverse depth of the feature point at the fth i The observations in the frame;

[0063] Take the feature point at the fth j The coordinates in the frame camera coordinate system are the predicted values, and the feature points are j The coordinates in the frame pixel coordinate system are the observation values, and the visual reprojection error is calculated as:

[0064]

[0065] Among them, x cj ,y cj , z cj is the feature point at the fth j The predicted coordinates in the frame camera coordinate system, u cj , v cj is the feature point at the fth j The observation coordinates in the frame pixel coordinate system;

[0066] Set the actual values ​​of the angular velocity and acceleration of the inertial measurement unit to w, a, and the measured values ​​to Then we have:

[0067]

[0068] The superscript g indicates the gyroscope, the superscript a indicates the accelerometer, the superscript w indicates the world coordinate system, the b indicates the inertial measurement unit body coordinate system, and the q bw The conversion relationship from the world coordinate system to the inertial measurement unit coordinate system, g represents the acceleration due to gravity;

[0069] The derivatives of position p, posture q, and velocity v with respect to time are expressed as:

[0070]

[0071] Integrate the measurement values ​​of the inertial measurement unit from p, v, q at the i-th moment to obtain p, v, q at the i-th moment; update the camera's posture according to the calculated position p, posture q, and velocity v.

[0072] In this embodiment, the data is associated with the point features in the image, and constraints are constructed through reprojection errors. Through IMU pre-integration and BA optimization, consistent with the classic VIO model, IMU and vision are combined to achieve visual information acquisition in complex environments.

[0073] In this embodiment, it is proposed to identify the target object features of the target image to be detected through a feature extraction network and convert it into BEV image features, and then enhance the BEV image features based on a deformable attention mechanism. The flowchart is as follows: Figure 4 shown.

[0074] In an optional embodiment, the first feature extraction network includes a ResNet network; and the second feature extraction network includes PointPillars.

[0075] Further optionally, the ResNet network includes a first input layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first residual block, a second residual block, a third residual block, a fourth residual block, a fifth residual block, a fifth residual block, a sixth residual block, a seventh residual block, an eighth residual block and a first global average pooling layer connected in sequence.

[0076] First, the first input layer receives an RGB image of size 224x224; then the data is input into the convolution layer, each convolution layer uses a 3x3 convolution kernel and a ReLU activation function to extract image features; then the image features are input into the residual block layer, each residual block consists of two convolution layers and a skip connection, which is used to solve the gradient vanishing and gradient exploding problems in deep convolutional neural networks; finally, the image features are input into the global average pooling layer, the feature map is globally average pooled, and the feature map is converted into one dimension.

[0077] ResNet introduces a residual module into the network, in which a jump connection is added between the input and output, adding the input directly to the output. This jump connection design makes it easier for the network to learn the residual, thus solving the problem of gradient disappearance and model degradation.

[0078] In an optional embodiment, the process of converting the target object features into BEV image features by using the LSS detection algorithm includes the following steps:

[0079] Use sensor data to assign a depth value to each image pixel, and combine the camera's internal and external parameters to convert the two-dimensional image features into three-dimensional image features under three-dimensional space coordinates; project all three-dimensional image features with depth information onto the BEV plane and aggregate them according to the grid on the BEV plane to form BEV image features.

[0080] Traditional object detection algorithms based on two-dimensional images often cannot accurately obtain the height information of objects and have limitations when processing three-dimensional scenes. The PointPillars algorithm can handle object detection tasks under different weather, lighting and environmental conditions and has strong robustness. By projecting point cloud data onto a bird's-eye view and combining voxelization and feature extraction techniques, the PointPillars algorithm can better utilize the three-dimensional information of point cloud data, thereby improving the accuracy of object detection while achieving faster detection speed.

[0081] In this embodiment, the ResNet network is used to extract features of the image to be detected. After the image features are extracted, the LSS algorithm is used to convert the image features into BEV features. After multi-scale information is obtained through multi-layer encoding of the input features, a deformable attention mechanism is used to obtain more accurate BEV image feature information.

[0082] In this embodiment, it is proposed to extract radar point cloud features from the target point cloud data by using PointPillars as the second feature extraction network, and convert it into BEV point cloud features. The flow chart is as follows: Figure 5 shown.

[0083] The radar point cloud features of the target point cloud data are extracted through the PointPillars algorithm, and feature mapping is performed after the point cloud features are obtained. Then, the multi-scale feature extraction information and the spliced ​​fusion feature map are subjected to BN and ReLU operations. Finally, a deformable attention mechanism is used to obtain more accurate BEV point cloud feature information.

[0084] In an optional embodiment, the feature extraction conversion of the radar point cloud uses the PointPillars algorithm to extract features in the point cloud data and convert them into BEV features. The specific steps are as follows:

[0085] First, the space of a point cloud sample is divided into grid-like vertical columns, where the size of each vertical column is a fixed value on the X-axis and Y-axis. All points in the sample will be divided into their respective vertical columns, and vertical columns without points will be regarded as invalid vertical columns.

[0086] Second, we need to add an additional 5 dimensions of data to each point in the vertical column, including x c ,y c 、z c 、xp and p Among them, x c ,y c 、z c Represents the coordinate offset of each point relative to the average coordinate of all points in the vertical column where the point is located. p and p represents the X and Y offset of the point relative to the geometric center of the vertical column. After data enhancement, the dimension of each point becomes 9 dimensions. Specifically, the encoding of each point contains x, y, z, x c ,y c 、z c 、x p and p It should be noted that an additional z is added p , that is, the offset of the point on the Z axis relative to the vertical column.

[0087] For example, the sample contains P valid vertical columns, and each vertical column can contain at most N points. For a vertical column, if the number of points is less than N, it is filled with 0; if it exceeds N, N points are randomly selected from the vertical column for sampling. For each point in each vertical column, it is necessary to encode, including the coordinates of the point, reflection intensity, vertical column geometric center, and the positional relationship of the point relative to the vertical column geometric center. The encoding length of each point obtained is D. The entire point cloud sample can be represented as a tensor with a shape of (P, N, D).

[0088] After obtaining the vertical column representation of the point cloud, the SA module in PointNet is used to extract features. First, multi-layer perception is applied to each point in each vertical column to map the encoding of each point from D dimension to C dimension, thereby obtaining a tensor of shape (P, N, C). Next, maximum pooling is applied to the points in each vertical column to obtain the feature vector of the vertical column. This step also eliminates the N dimension in the tensor, and finally obtains a feature map of shape (P, C).

[0089] Finally, the feature map of (P, C) is expanded into pseudo image features according to the position of the vertical column, and P is expanded to (H, W) to obtain a feature representation in the form of (C, H, W) similar to the image.

[0090] Through the above mapping operation, the obtained tensor data is subjected to multi-scale feature extraction and splicing fusion using the feature extraction method in two dimensions. The backbone network contains two sub-networks:

[0091] (1) A top-down network is used to generate feature maps of different scales. The network consists of a series of blocks (S, L, F), where S is the stride of each block, L is the 3×3 convolution, and F is the output channel. Each block contains BatchNorm and ReLU.

[0092] (2) The second network obtains the feature map through upsampling and concatenation operations. The output of each block in the top-down network is upsampled. in ,s out ,F)s in Where is the input stride, s out is the output stride, F is the output channel, and each block of the bottom-up network corresponds to an upsampling operation. A two-dimensional transposed convolution is used to obtain the upsampled feature map. Subsequently, BN and ReLU operations are performed on the feature map. All feature maps are concatenated to obtain the final BEV feature map.

[0093] In an optional embodiment, the deformable attention mechanism is used to enhance the BEV image features and the BEV point cloud features, respectively, including enhancing the input features using a spatiotemporal pyramid prediction model with a deformable attention mechanism, and the specific steps are as follows:

[0094] Step 1: Given the input graph x∈R H×W×C , generating a point As a reference, N represents the batch size, C represents the number of channels, W represents the width, and H represents the height. These dimensional information describes and defines the shape of the input data and feature maps in the convolutional neural network.

[0095] Specifically, the grid size is downsampled by a factor r from the input feature map size. G and W G is the height and width after downsampling, H G =H / r,W G =W / r, the reference point value is a linearly spaced two-dimensional coordinate {(0; 0),..., H G -1; W G -1)}, and then normalize them to [-1; +1] according to the grid shape H G ×W G , where (-1; -1) represents the upper left corner and (+1; +1) represents the lower right corner. To obtain the offset of each reference point, the feature map is linearly projected onto the query token q = xW q , which is then input into a lightweight sub-network θoffset(q) to generate an offset △p = θoffset(q).

[0096] In order to stabilize the training process, a predefined factor s is used to scale the amplitude of △p to prevent too large an offset, that is, △p←s tanh(Δp). The features are then sampled at the position of the deformation point as keys and values, and then the projection matrix is ​​sampled:

[0097]

[0098] Denote the deformed key embedding and value embedding respectively. Specifically, let the sampling function Φ() be transformed into bilinear interpolation:

[0099]

[0100] In g(a,b)=max(0,1-|ab|) and (r y ,r x ) indexes all positions on z2r H×W×C. Because g is only in the position closest to (p x ,p y ), which simplifies Eq. (8) to a weighted average of 4 positions. As mentioned earlier, the offset generation adopts a subnetwork that consumes the query features and outputs the offset values ​​of the reference points respectively. Considering that each reference point covers a local s×s area (s is the maximum value of the offset), the generation network should also have the perception of local features to learn reasonable offsets. Therefore, the subnetwork is implemented as two convolution modules with nonlinear activation, and the input features are first passed through a 5×5 depthwise convolution to capture local features. Then, GELU activation and 1×1 convolution are used to obtain the two-dimensional offset. The bias in the 1×1 convolution is deleted to alleviate the forced shift of all positions.

[0101] Step 2: Set the offset group. To promote the diversity of deformation points, the feature channels are divided into G groups. The features of each group use a shared subnet to generate corresponding offsets. The number of attention modules M is set to be multiple times the size of the offset group G to ensure that multiple attention heads are assigned to a set of deformed keys and values.

[0102] Deformable relative position bias. Relative position bias encodes the relative position between each pair of query and key, thereby increasing the attention to spatial information. For a feature map of shape H×W, its relative coordinate displacement ranges are [-H; H] and [-W; W] in two dimensions, respectively.

[0103] Construct a relative position offset table The relative position bias B is obtained by indexing the table with the relative displacement in both directions. Since the deformable attention has continuous key positions, the relative displacement in the normalized range [-1; +1] is calculated and then interpolated In the parameterized bias table Continuous relative displacement is performed to cover all possible offset values.

[0104] The computational effort of Deformed Multi-Head Attention (DMHA) is similar to that of PVT or Swin Transformer. The only additional overhead comes from the subnet used to generate the offset. The complexity of the entire module can be summarized as:

[0105] Ω(DMHA)=2HW N s C+2HWC 2 +2N s C 2 +(k 2 +2)N s C

[0106] In the above formula, N s =HGWG = HW / r 2 is the number of sampling points. The computational cost of the offset network has a linear complexity with the channel size, which is quite small compared to the cost of attention calculation. The complexity will be further reduced by choosing a larger downsampling factor r, which makes it suitable for tasks with higher resolution input.

[0107] Step three: Add the temporal BEV pyramid features to obtain multi-scale information and more powerful feature acquisition capabilities.

[0108] The input temporal BEV feature map B is encoded in four layers. Each encoder layer consists of an optional convolutional layer with stride 2 (for downsampling the features) and a deformable DAT transformer encoder, which is a stack of deformable DCN transformer blocks.

[0109] Step 4: By adding future BEV queries, the next trajectory of the target object can be effectively predicted. By defining future BEV queries to represent the future BEV state and query the generated multi-scale spatiotemporal features. There is a set of learnable future queries {Q (t)}t=0,...,T t , where Q (t) Same spatial dimensions as B3 Future queries use separate learned embeddings to distinguish predicted BEV states over time. In addition, a map feature generator is applied to extract the BEV state from the B (t) Generate high-dimensional features to extract information about the scene geometry.

[0110] Exemplarily, the spatiotemporal pyramid prediction model architecture is as follows: Figure 6 shown.

[0111] In this embodiment, it is proposed to fuse the BEV image features and BEV point cloud features after feature enhancement, and the flow chart thereof is as follows: Figure 7 shown.

[0112] In order to better align BEV features, this embodiment uses the feature fusion method of UniBEV: Multi-modal 3D Object Detection with Uniform BEVEncoders for Robustness against Missing Sensor Modalities, adopts connection, averaging and the proposed channel normalized weights (CNW), realizes dynamic weight allocation of different information sources, and ensures performance in various input scenarios.

[0113] The feature fusion expression is:

[0114] A C (i),A L (i) = softmax(A C (i),A L (i))

[0115]

[0116] Among them, A C (i) is the weight A C The i-th element in A represents the relative importance of modality C to the fusion result of the i-th channel. L (i) is the weight A L The i-th element in represents the relative importance of the modality L to the fusion result of the i-th channel, ⊙ represents the channel-wise multiplication using implicit broadcasting in the spatial dimension, is the BEV feature of the image, is the BEV feature of the point cloud. CNW adds a small number of learnable parameters to give the model more flexibility between these special cases, enabling it to optimize the relative importance of each modality for fusion and still allow a single modality to input meaningful values.

[0117] In this embodiment, it is proposed to use the fusion features for target detection. For example, Figure 8 As shown, it is the overall flow chart of target detection of this embodiment.

[0118] Exemplarily, the fused features are input into a two-dimensional network to obtain a fused feature processing result, and then the feature processing result is input into a detection head to obtain a three-dimensional target detection box result.

[0119] Example 2

[0120] This embodiment proposes a target object detection system based on BEV, and applies the target object detection method based on BEV proposed in Embodiment 1. Fig. 9 , which is an architecture diagram of a BEV-based target object detection system of this embodiment.

[0121] The present embodiment proposes a BEV-based target object detection system, including:

[0122] A data acquisition module is used to acquire the image to be detected and the corresponding target point cloud data;

[0123] A first feature extraction module, used for identifying target object features of the target image to be detected through a first feature extraction network, and converting the features into BEV image features;

[0124] A second feature extraction module, used for extracting radar point cloud features from the target point cloud data through a second feature extraction network, and converting the radar point cloud features into BEV point cloud features;

[0125] A feature enhancement module, used for performing feature enhancement on the BEV image features and the BEV point cloud features based on a deformable attention mechanism;

[0126] A fusion module is used to fuse the BEV image features and BEV point cloud features after feature enhancement to obtain fusion features;

[0127] The result output module is used to perform target detection on the fusion features to obtain detection results.

[0128] It can be understood that the system of this embodiment corresponds to the method of the above-mentioned embodiment 1, and the options in the above-mentioned embodiment 1 are also applicable to this embodiment, so they will not be described repeatedly here.

[0129] Example 3

[0130] This embodiment proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes all or part of the steps of a BEV-based target object detection method proposed in Example 1.

[0131] Example 4

[0132] This embodiment proposes a storage medium on which computer-readable instructions are stored, wherein the computer-readable instructions, when executed by a processor, implement all or part of the steps of a BEV-based target object detection method proposed in Embodiment 1.

[0133] Exemplarily, the storage medium includes, but is not limited to, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0134] Exemplarily, the instructions, programs, code sets or instruction sets may be implemented using conventional programming languages.

[0135] Exemplarily, the processor includes but is not limited to a smart phone, a personal computer, a server, a network device, etc., and is used to execute all or part of the steps of a BEV-based target object detection method described in Example 1.

[0136] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely exemplary, in which the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same or one or more software and / or hardware when implementing the scheme of the present invention. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the scheme of this embodiment.

[0137] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A target object detection method based on BEV, characterized in that: The method comprises: Acquire the image to be detected and the corresponding target point cloud data; Performing target object feature recognition on the target image to be detected through a first feature extraction network and converting the target object feature into a BEV image feature; Extract radar point cloud features from the target point cloud data through a second feature extraction network and convert the radar point cloud features into BEV point cloud features; Performing feature enhancement on the BEV image features and the BEV point cloud features based on a deformable attention mechanism; The enhanced BEV image features and BEV point cloud features are fused to obtain fused features; The fused features are used for target detection to obtain a detection result.

2. The target object detection method based on BEV according to claim 1, characterized in that: The image to be detected is collected by a camera; the posture of the camera is adjusted by inertial sensor data and visual data, and the inertial sensor data and visual data are brought into the visual inertial odometer model to obtain a posture trajectory for adjusting the posture of the camera; the system state quantity in the visual inertial odometer model is defined as follows: X=[x f ,x f+1 ...x f+N ,l s ,l s+1 ...l s+M ] Where X is the camera’s position, λ is the observed landmark point; f represents the camera's position at the fth frame, λ s represents the sth landmark, that is, the sliding window starts from the sth landmark of the fth frame; N is the number of key frames in the sliding window, M is the number of landmarks observed by all key frames in the sliding window; X i Including the position p, posture q, velocity v of the inertial measurement unit body in the inertial coordinate system at time i, and the acceleration bias estimate b in the inertial measurement unit body coordinate system a and the angular velocity bias estimate w g ,in represents the acceleration and angular velocity of the camera at the i-th moment, represents the acceleration and angular velocity of the camera at the i-th moment, w represents the angular velocity, b represents the acceleration, and v i represents the speed at the i-th moment, It is expressed as the estimated acceleration bias at the i-th moment in the body coordinate system of the inertial measurement unit, It is expressed as the estimate of the angular velocity bias at the i-th moment in the inertial measurement unit body coordinate system.

3. The target object detection method based on BEV according to claim 2, characterized in that: The camera posture is adjusted by inertial sensor data and visual data, including the following steps: Step 1: Use a camera to capture a calibration image and extract feature points from the calibration image to obtain the two-dimensional coordinates (u, v) of the feature points; use the inverse projection of the feature points on the image as feature parameters and convert them into three-dimensional coordinates (x, y, z) in the camera coordinate system; Step 2: Establish the coordinate relationship between the feature points in the normalized camera coordinate system and the camera coordinate system through inverse depth parameterization. The expression is: Among them, λ = 1 / z is the inverse depth parameter, that is, the inverse of the feature point depth; Step 3: Based on the inverse depth of the feature points, i The observed value in the frame, the coordinates of the predicted feature point at the jth moment are: is the camera coordinate system coordinate of the feature point at the jth moment, T (b,c) is the external parameter from the camera coordinate system to the inertial measurement unit coordinate system, For the f i The transformation matrix from the inertial measurement unit coordinate system of the frame to the world coordinate system, For the f j The transformation matrix from the inertial measurement unit coordinate system of the frame to the world coordinate system; is the inverse depth of the feature point at the fth i The observations in the frame; Take the feature point at the fth j The coordinates in the frame camera coordinate system are the predicted values, and the feature points are j The coordinates in the frame pixel coordinate system are the observation values, and the visual reprojection error is calculated as: Among them, x cj ,y cj , z cj is the feature point at the fth j The predicted coordinates in the frame camera coordinate system, u cj , v cj is the feature point at the fth j The observation coordinates in the frame pixel coordinate system; Set the true values ​​of the angular velocity and acceleration of the inertial measurement unit to w, a, and the measured values ​​to Then we have: The superscript g indicates the gyroscope, the superscript a indicates the accelerometer, the superscript w indicates the world coordinate system, the b indicates the inertial measurement unit body coordinate system, and the q bw The conversion relationship from the world coordinate system to the inertial measurement unit coordinate system, g represents the acceleration due to gravity; The derivatives of position p, posture q, and velocity v with respect to time are expressed as: Integrate the measurement values ​​of the inertial measurement unit from p, v, q at the i-th moment to obtain p, v, q at the j-th moment; update the camera's posture according to the calculated position p, posture q, and velocity v.

4. The target object detection method based on BEV according to claim 1, characterized in that: The first feature extraction network includes a ResNet network; the second feature extraction network includes PointPillars.

5. The target object detection method based on BEV according to claim 1, characterized in that: The target object features are converted into BEV image features through the LSS detection algorithm, which includes the following steps: Use sensor data to assign a depth value to each image pixel, and combine the camera's internal and external parameters to convert the two-dimensional image features into three-dimensional image features under three-dimensional space coordinates; project all three-dimensional image features with depth information onto the BEV plane and aggregate them according to the grid on the BEV plane to form BEV image features.

6. The target object detection method based on BEV according to claim 1, characterized in that: The feature extraction conversion of the radar point cloud uses a second feature extraction network to extract features in the point cloud data and converts them into BEV features. The specific steps are as follows: Step 1: Divide the space of a point cloud sample into grid-like vertical columns, extract features from the vertical columns of the point cloud through the SA module in PointNet, and obtain point cloud data features; the size of each vertical column is a fixed value on the X-axis and Y-axis; Step 2: Perform multi-scale feature extraction and splicing fusion on the point cloud data features to obtain a BEV feature map of the point cloud.

7. The target object detection method based on BEV according to claim 1, characterized in that: The deformable attention mechanism is used to enhance the BEV image features and the BEV point cloud features, respectively, including enhancing the input features using a spatiotemporal pyramid prediction model with a deformable attention mechanism; The spatiotemporal pyramid prediction model includes at least 4 stacked encoder layers, a future query layer, and decoder layers corresponding to the encoder layers one by one; The encoder layer consists of a convolutional layer and a deformable DAT transformer block, which is used to perform BEV feature extraction on the input BEV image features and / or BEV point cloud features; The future query layer is used to query and generate future BEV states, and to generate high-dimensional map features from the encoded BEV features using a map feature generator, and to fuse them with the BEV features to generate multi-scale spatiotemporal features; The decoder layer consists of a deconvolution layer and a deformable DAT transformer block, which is used to decode multi-scale spatiotemporal features to obtain feature-enhanced BEV image features and / or BEV point cloud features.

8. The target object detection method based on BEV according to claim 7, characterized in that: In the deformable DAT transformer block, the following steps are performed: Map the input features into a linear projection to obtain the query vector q; The feature channels are divided into G groups, and the features of each group use a shared offset network to generate corresponding offsets △p; The amplitude of the offset △p is scaled by a predefined factor s, and the input feature is sampled at the deformation point using the scaled offset △p' to obtain the deformed key vector Sum value vector Encode the relative position between each pair of query and key, pass the query vector, key vector and value vector into the deformable attention mechanism, perform feature fusion based on the deformable attention mechanism, and complete feature enhancement; The encoder and decoder layers at the same level use the same key vector and value vector.

9. A target object detection method based on BEV according to any one of claims 1 to 8, characterized in that: The method of fusing the feature-enhanced BEV image features and BEV point cloud features includes: using connection, average and channel normalization weights to dynamically allocate weights to different information sources.

10. A BEV-based target object detection system, using the BEV-based target object detection method according to any one of claims 1 to 9, characterized in that: include: A data acquisition module is used to acquire the image to be detected and the corresponding target point cloud data; A first feature extraction module, used for identifying target object features of the target image to be detected through a first feature extraction network, and converting the features into BEV image features; A second feature extraction module, used for extracting radar point cloud features from the target point cloud data through a second feature extraction network, and converting the radar point cloud features into BEV point cloud features; A feature enhancement module, used for performing feature enhancement on the BEV image features and the BEV point cloud features based on a deformable attention mechanism; A fusion module is used to fuse the BEV image features and BEV point cloud features after feature enhancement to obtain fusion features; The result output module is used to perform target detection on the fusion features to obtain detection results.

Images