Image detection model training and target detection method and device based on BEV

Abstract

The present disclosure provides a BEV-based image detection model training and target detection method and device, relating to the field of artificial intelligence technology, particularly the field of autonomous driving technology. The method comprises: extracting features from N perspective images to obtain N multi-layer image feature maps, and performing depth estimation based on the N multi-layer image feature maps to obtain N depth feature maps; training the image detection model for the l+1th time based on the N multi-layer image feature maps, the N depth feature maps, and the bird's-eye view BEV feature map l output from the lth training, outputting the BEV feature map l+1 and the target detection result; adjusting the model parameters of the image detection model based on the depth feature map and the target detection result, and continuing to train the adjusted image detection model until a target image detection model is obtained. The present disclosure can improve the accuracy of feature space conversion and perform accurate and efficient 3D target detection in the BEV space.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the field of autonomous driving technology. Background Art

[0002] In autonomous driving, it is often necessary to use images captured by on-board cameras in a bird's eye view (BEV) space to perform subsequent operations such as driving planning.

[0003] As the diversity and number of sensors equipped on self-driving vehicles (SDVs) become increasingly complex, it becomes crucial to represent features from different viewpoints in a unified way. The well-known bird's-eye view (BEV) is a natural and direct candidate for a unified representation. Compared to the front view or perspective view, which have been widely studied in the field of two-dimensional vision, the BEV representation has several inherent advantages. First, it does not suffer from the occlusion and scale issues that are prevalent in two-dimensional tasks. Recognizing vehicles with occlusions or cross traffic can be better solved. In addition, representing objects or road elements in this form will facilitate the development and deployment of subsequent modules (such as planning and control).

[0004] Therefore, how to further improve the efficiency and accuracy of image detection, enhance the accuracy of feature space conversion, and perform accurate and efficient 3D target detection in BEV space has become one of the important research directions. Summary of the Invention

[0005] The present disclosure provides a BEV-based image detection model training and target detection method and device.

[0006] According to one aspect of the present disclosure, a method for training an image detection model is provided, the method comprising:

[0007] Performing feature extraction on the N perspective images to obtain N multi-layer image feature maps, and performing depth estimation based on the N multi-layer image feature maps to obtain N depth feature maps, where N is a positive integer;

[0008] Based on the N multi-layer image feature maps, the N depth feature maps, and the bird's-eye view BEV feature map l outputted from the lth training, the image detection model is trained for a l+1th time, and the BEV feature map l+1 and the target detection result are outputted, where l is a positive integer;

[0009] According to the depth feature map and the target detection result, the model parameters of the image detection model are adjusted, and the adjusted image detection model is continuously trained until the target image detection model is obtained.

[0010] According to one aspect of the present disclosure, a target detection method is provided, the method comprising:

[0011] Obtain N perspective images to be detected at the current moment, and extract N multi-layer image feature maps of the N perspective images to be detected;

[0012] Performing depth estimation based on the N multi-layer image feature maps to obtain N depth feature maps;

[0013] Inputting the N multi-layer image feature maps and the N depth feature maps into a target image detection model, the target image detection model performs an attention mechanism based on the N multi-layer image feature maps, the N depth feature maps and the previous BEV feature map of the model to obtain a target BEV feature map;

[0014] Perform target detection on the target BEV feature map to obtain the target detection result;

[0015] The target image detection model is trained using the image detection model training method of the first embodiment.

[0016] The present disclosure can improve the efficiency and accuracy of image detection, enhance the accuracy of feature space conversion, introduce depth information in the process of mapping multi-view images from different image spaces to a unified BEV space, improve the accuracy of feature space conversion, and perform accurate and efficient 3D target detection in the BEV space.

[0017] According to another aspect of the present disclosure, a training device for an image detection model is provided, comprising:

[0018] A feature map acquisition module is used to extract features from N perspective images to obtain N multi-layer image feature maps, and perform depth estimation based on the N multi-layer image feature maps to obtain N depth feature maps, where N is a positive integer;

[0019] A training module is configured to train the image detection model for the l+1th time based on the N multi-layer image feature maps, the N depth feature maps, and the bird's-eye view (BEV) feature map l output from the lth training, and output the BEV feature map l+1 and the target detection result, where l is a positive integer;

[0020] The adjustment module is used to adjust the model parameters of the image detection model according to the depth feature map and the target detection results, and continue to train the adjusted image detection model until the target image detection model is obtained.

[0021] According to another aspect of the present disclosure, there is provided an object detection device, comprising:

[0022] An extraction module is used to obtain N perspective images to be detected at the current moment and extract N multi-layer image feature maps of the N perspective images to be detected;

[0023] A depth estimation module is used to perform depth estimation based on N multi-layer image feature maps to obtain N depth feature maps;

[0024] An acquisition module is used to input N multi-layer image feature maps and N depth feature maps into the target image detection model, and the target image detection model performs an attention mechanism based on the N multi-layer image feature maps and N depth feature maps and the previous BEV feature map of the model to obtain a target BEV feature map;

[0025] The target detection module is used to perform target detection on the target BEV feature map and obtain the target detection result;

[0026] The target image detection model is trained using a training device such as an image detection model.

[0027] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor, and

[0028] a memory communicatively connected to at least one processor; wherein,

[0029] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to execute the image detection model training method of the first aspect embodiment of the present disclosure or the target detection method of the second aspect embodiment.

[0030] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the image detection model training method of the first aspect embodiment of the present disclosure or the target detection method of the second aspect embodiment.

[0031] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the image detection model training method of the first aspect embodiment of the present disclosure or the target detection method of the second aspect embodiment.

[0032] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0034] Figure 1 is a flowchart of a method for training an image detection model according to one embodiment of the present disclosure;

[0035] Figure 2 is a flowchart of a method for training an image detection model according to one embodiment of the present disclosure;

[0036] Figure 3 is a flowchart of a method for training an image detection model according to one embodiment of the present disclosure;

[0037] Figure 4 is a schematic diagram of a training method for an image detection model according to an embodiment of the present disclosure;

[0038] Figure 5 is a schematic diagram of a training method for an image detection model according to an embodiment of the present disclosure;

[0039] Figure 6 is a flow chart of a target detection method according to one embodiment of the present disclosure;

[0040] Figure 7 is a structural diagram of a training device for an image detection model according to an embodiment of the present disclosure;

[0041] Figure 8 is a structural diagram of an object detection device according to an embodiment of the present disclosure;

[0042] Figure 9 It is a block diagram of an electronic device used to implement the training method of the image detection model of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0044] The embodiments of the present disclosure relate to the fields of artificial intelligence technologies such as computer vision and deep learning.

[0045] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence.

[0046] Unmanned driving technology is a combination of multiple cutting-edge disciplines such as sensors, computers, artificial intelligence, communications, navigation and positioning, pattern recognition, machine vision, and intelligent control.

[0047] Autonomous driving generally refers to an automated driving system that utilizes advanced communications, computing, networking, and control technologies to achieve real-time, continuous control of trains. Modern communication methods, directly connected to the train, enable two-way data communication between trains and the ground, with fast transmission rates and large amounts of information. This allows tracking trains and control centers to instantly determine the exact location of the preceding train, making operational management more flexible and control more efficient, making it more adaptable to the demands of autonomous trains.

[0048] The following describes the BEV-based image detection model training and target detection method and device disclosed in the present invention with reference to the accompanying drawings.

[0049] Figure 1 is a flowchart of a method for training an image detection model according to an embodiment of the present disclosure, such as Figure 1 As shown, the method includes the following steps:

[0050] S101, performing feature extraction on N perspective images to obtain N multi-layer image feature maps, and performing depth estimation based on the N multi-layer image feature maps to obtain N depth feature maps, where N is a positive integer.

[0051] Optionally, in an embodiment of the present disclosure, N image acquisition devices, such as cameras, are called to capture images from N perspectives. That is, the N image acquisition devices are deployed at different positions, and any image acquisition device corresponds to one perspective. The image of the current scene is captured to obtain N perspective images, and then features of the N perspective images are extracted through a neural network to obtain N multi-layer image feature maps.

[0052] Optionally, in an embodiment of the present disclosure, a residual network is used to extract image features. For example, ResNet50 or ResNet101 can be used to extract features from N perspective images to obtain N multi-layer image feature maps.

[0053] In some implementations, in order to optimize the feature extraction of multi-scale targets, Feature Pyramid Networks (FPN) can also be used to perform multi-scale fusion on the extracted multi-layer image feature maps.

[0054] When mapping from image space to BEV space without considering depth information, it is possible that a series of false detection frames will be predicted along the depth ray direction in the BEV space. For this reason, in the embodiment of the present disclosure, depth estimation is performed based on N multi-layer image feature maps to obtain N depth feature maps, so as to facilitate the subsequent adaptive fusion of features of multiple input images and mapping them to a unified BEV space, thereby reducing the problem of false detection.

[0055] In some implementations, a multi-class classification task is performed on the pixels of each of the N multi-layer image feature maps, multiple depths are predicted for each pixel, and the probabilities of each depth are weighted to obtain the final depth value. This process is repeated to obtain N depth feature maps. Each pixel in the depth feature map has a depth value.

[0056] S102, based on N multi-layer image feature maps, N depth feature maps and the bird's-eye view BEV feature map l output from the lth training, the image detection model is trained for the l+1th time, and the BEV feature map l+1 and the target detection result are output, where l is a positive integer.

[0057] In the embodiment of the present disclosure, the initial BEV feature map of the first training is randomly initialized and the output BEV feature map is trained based on N multi-layer image feature maps, N depth feature maps and the initial BEV feature map. Figure 1 .

[0058] Optionally, the attention mechanism network in the image detection model performs an attention mechanism, such as a global cross-attention mechanism or a deformable cross-attention mechanism, on the N multi-layer image feature maps, the N depth feature maps, and the BEV feature map l to obtain a BEV feature map l+1.

[0059] Optionally, the image detection model further includes a target detection network, which performs target detection on the BEV feature map l+1 to obtain a target detection result. Optionally, the target detection network can be a deep neural network.

[0060] S103, adjusting the model parameters of the image detection model according to the depth feature map and the target detection result, and continuing to train the adjusted image detection model until the target image detection model is obtained.

[0061] In the disclosed embodiment, a loss function may be obtained based on the depth feature map and the target detection results to adjust the model parameters of the image detection model.

[0062] In some implementations, a first loss function is obtained between a depth feature map and a reference depth feature map, and a second loss function is obtained between an object detection result and a reference detection result. A total loss function of the image detection model is obtained based on the sum of the first loss function and the second loss function, and model parameters of the image detection model are adjusted based on the total loss function.

[0063] In some implementations, a first loss function between the depth feature map and the reference depth feature map is obtained, and the image detection model is adjusted based on the first loss function.

[0064] In some implementations, a second loss function between the target detection result and the reference detection result is obtained, and the image detection model is adjusted based on the second loss function.

[0065] Optionally, the adjusted image detection model is continuously trained until a preset number of training times is reached or the error reaches a preset range, thereby obtaining a target image detection model.

[0066] In an embodiment of the present disclosure, feature extraction is performed on N perspective images to obtain N multi-layer image feature maps, and depth estimation is performed based on the N multi-layer image feature maps to obtain N depth feature maps, where N is a positive integer; based on the N multi-layer image feature maps, the N depth feature maps, and the bird's-eye view BEV feature map l output from the lth training, the image detection model is trained for the l+1th time, and the BEV feature map l+1 and the target detection result are output, where l is a positive integer; based on the depth feature map and the target detection result, the model parameters of the image detection model are adjusted, and the adjusted image detection model is continuously trained until the target image detection model is obtained. The present disclosure can improve the efficiency and accuracy of image detection and the accuracy of feature space conversion. Depth information is introduced in the process of mapping multi-perspective images from different image spaces to a unified BEV space, which can improve the accuracy of feature space conversion and perform accurate and efficient 3D target detection in the BEV space.

[0067] Figure 2 is a flowchart of a method for training an image detection model according to an embodiment of the present disclosure, such as Figure 2 As shown, the method includes the following steps:

[0068] S201, performing feature extraction on N perspective images to obtain N multi-layer image feature maps, and performing depth estimation based on the N multi-layer image feature maps to obtain N depth feature maps, where N is a positive integer.

[0069] For the introduction of step S201, please refer to the relevant content in the above embodiment, which will not be repeated here.

[0070] It should be noted that, in the embodiment of the present disclosure, after obtaining N multi-layer image feature maps, multi-scale fusion is performed based on S FPN layers, with the aim of optimizing the extraction of features of multi-scale targets.

[0071] S202, the attention mechanism network in the image detection model performs an attention mechanism on N multi-layer image feature maps, N depth feature maps and BEV feature map l to obtain BEV feature map l+1.

[0072] In some implementations, taking the full attention mechanism to obtain the BEV feature map l+1 as an example, global attention is calculated pixel by pixel for each bev query and the image it can be projected on, that is, the transformation matrix (lidar2img parameter) from the lidar coordinate system to the image coordinate system is determined. Based on the transformation matrix, the target image to which the T query vectors corresponding to the BEV feature map l can be projected is determined, where the target image is n images in the multi-view image, n is an integer, and 1≤n≤N, and T is a positive integer. Further, the value vector and key vector of each layer of the image feature map corresponding to the target image are determined, and a multi-head global attention mechanism is performed on the query vector and the value vector and key vector of each layer of the image feature map at each pixel position to obtain the BEV feature map l+1.

[0073] Alternatively, the BEV feature map can be expressed as F bev ∈R C*X*Y , where T = X*Y, X and Y are positive integers, X*Y represents the shape of the query, and C is the feature dimension of each query.

[0074] Alternatively, the multi-view image can be represented as I∈R N*3*H*W , H represents the length of each multi-view image, W represents the width of each multi-view image, and H*W represents the size of each multi-view image.

[0075] Alternatively, the image feature map can be represented as F img ∈R N*C*h*w , h represents the length of each image feature map, w represents the width of each image feature map, and h*w represents the size of each image feature map.

[0076] In some implementations, a multi-head global attention mechanism is applied pixel by pixel to the query vector and the value vector and key vector of each layer of the image feature map to obtain the BEV feature map l+1, including: for query vector i among the T query vectors, based on the query vector i and the value vector and key vector of the s-th layer image feature map, obtaining the first matrix of the s-th layer corresponding to the target image, and adding the first matrices of each layer to obtain the second matrix of the attention head. Wherein, i and s are both integers, and 1≤s≤S, 1≤i≤T. Based on the second matrix of the attention head and the weight matrix of the attention head, a third matrix of the attention head is obtained, and the third matrices of each attention head are added to obtain the global attention matrix for query vector i. The global attention matrices of the T query vectors are added to obtain the BEV feature map l+1.

[0077] In some implementations, obtaining a first matrix of the sth layer corresponding to a target image based on a query vector i and a value vector and a key vector of an sth image feature map includes: determining a first depth value of position j on the target image based on a depth feature map of the target image. Determining a first weight of position j in a dimension of the query vector i based on the absolute value of the difference between the second depth value and the first depth value of position j in the query vector i. Obtaining a first matrix of the sth layer corresponding to the target image based on the query vector i, a key (Key) of position j in the sth layer image feature map, a weight of position j, and a value (Value) of position j in the sth layer image feature map.

[0078] Alternatively, the BEV characteristic map l+1 may be obtained using the following formula:

[0079]

[0080] Among them, DGGCA represents the BEV feature map l+1 obtained by the full attention mechanism, n is the number of target images that the query vector i can project, represents the second depth value of position j in query vector i, Represents the value of position j on the s-th layer image feature map, represents the first depth value of position j on the target image, represents the key of position j on the s-th layer image feature map, abs(...) represents the operation of taking the absolute value of the difference, σ(...) represents the normalized exponential (softmax) function, M represents the number of attention heads, W m represents the learnable parameters of the mth attention head.

[0081] In some implementations, using a deformable attention mechanism to obtain a BEV feature map l+1 as an example, the bev query is projected onto different image locations using lidar2img, and L points are sampled around the locations. Different weights are assigned to the sampled points based on their depth similarity to the bevquery depth. In other words, a transformation matrix is ​​determined from the lidar coordinate system to the image coordinate system. Based on the transformation matrix, the projected image positions of the T query vectors corresponding to the BEV feature map l onto the target image are determined. The target image is one of the n images in the multi-view image, where n is an integer with 1≤n≤N and T is a positive integer. Samples are taken around the projected image positions to obtain the sampling positions. A multi-head deformable attention mechanism is applied to the query vector, the key vector of each layer's image feature map, and the sampling positions to obtain the BEV feature map l+1.

[0082] In some implementations, a multi-head deformable attention mechanism is performed on the query vector, the key vectors of each layer of the image feature map, and the sampling positions to obtain the BEV feature map l+1, including: for query vector i among the T query vectors, based on the query vector i, the sampling position, and the image feature map of the sth layer, a fourth matrix for the sampling position at the sth layer is obtained. The fourth matrices of each layer are summed to obtain the fifth matrix of the attention head. The sixth matrix of the attention head is obtained by combining the fifth matrix of the attention head with the weight matrix of the attention head, and the sixth matrix of each attention head is summed to obtain the deformable attention matrix for query vector i. The deformable attention matrices of the T query vectors are summed to obtain the BEV feature map l+1.

[0083] In some implementations, obtaining a fourth matrix for the sampling position at the sth layer based on a query vector i, the sampling position, and an image feature map at the sth layer includes: determining a third depth value for the sampling position f based on a depth feature map of the target image; determining a second weight for the sampling position f in the dimension of the query vector i based on the absolute value of the difference between the fourth depth value and the third depth value for the sampling position f in the query vector i; performing coordinate stretching on the first coordinate of the query vector i to obtain a second coordinate at the sth layer size; and obtaining a fourth matrix for the sampling position at the sth layer based on the second coordinate, the second weight for the sampling position f, and the value in the image feature map at the sth layer.

[0084] Alternatively, the BEV characteristic map l+1 may be obtained using the following formula:

[0085]

[0086] Among them, DGDCA represents the BEV feature map l+1 obtained by the deformable attention mechanism, F represents the number of samples around each projected image position, Δp msqk and A msqk They represent the offset and attention weight of the kth sampling point of the sth layer image feature map of the mth attention head, represents the scale factor, represents the normalized coordinates of the query vector i, d f represents the third depth value of the sampling position f, represents the fourth depth value of the sampling position f in the query vector i, W S ′ represents the weight of the s-th layer image feature map (each feature scale), Represents the value of position l in the image feature map of the sth layer.

[0087] S203, the target detection network in the image detection model performs target detection on the BEV feature map l+1 to obtain a target detection result.

[0088] S204: Adjust the model parameters of the image detection model according to the depth feature map and the target detection result, and continue training the adjusted image detection model until the target image detection model is obtained.

[0089] For the introduction of steps S203 to S204 , please refer to the contents of the above embodiment and will not be repeated here.

[0090] In some implementations, a first loss function is obtained between the depth feature map and the reference depth feature map, and the image detection model is adjusted based on the first loss function. For example, for classification tasks, when generating a heat map with num_class channels, the 3D center point of each real target is projected into the BEV space during training to generate a 2D Gaussian distribution feature map, and a loss function (such as focalloss) is used to calculate the loss. For regression tasks, a heat map with 8 channels is generated, and the offset o∈R of the 3D center point is regressed for each pixel. 2 , the height of the target in the longitudinal direction h g ∈R 1 , length, width and height dimensions s∈R 3 and the orientation angle (sin(β), cos(β))∈[-1, 1] 2 . Use loss function (such as L1loss) to perform loss calculation.

[0091] In some implementations, a second loss function is obtained between the target detection result and the reference detection result, and the image detection model is adjusted based on the second loss function. For example, 900 target queries are randomly initialized in 3D space, and then the transformer is used to calculate the similarity between the 900 object queries and the bev feature map to predict the 3D information of the 900 targets, including category, position, dimension, and orientation information. The category loss is calculated using focal loss, and the other information loss is calculated using L1 loss. Multiple prediction results are output during testing.

[0092] The present disclosure can improve the efficiency and accuracy of image detection, enhance the accuracy of feature space conversion, introduce depth information in the process of mapping multi-view images from different image spaces to a unified BEV space, improve the accuracy of feature space conversion, and perform accurate and efficient 3D target detection in the BEV space.

[0093] Figure 3 is a flowchart of a method for training an image detection model according to an embodiment of the present disclosure, such as Figure 3As shown, in the embodiment of the present disclosure, N perspective images are obtained, feature extraction is performed on the N perspective images to obtain N multi-layer image feature maps, and depth estimation is performed based on the N multi-layer image feature maps to obtain N depth feature maps. The N multi-layer image feature maps, the N depth feature maps, and the initial BEV feature map are input into the image detection model (i.e., the depth-guided attention mechanism module), and a global attention mechanism or a deformable attention mechanism is performed to obtain a target BEV feature map. The N multi-layer image feature maps, the N depth feature maps, and the target BEV feature map are input into the depth-guided attention mechanism module again, and a global attention mechanism or a deformable attention mechanism is performed to update the target BEV feature map.

[0094] The N multi-layer image feature maps, the N depth feature maps, and the initial BEV feature map are input into the depth-guided attention mechanism module to perform the global attention mechanism process as follows: Figure 4 As shown, N multi-layer image feature maps, the N depth feature maps, and the initial BEV feature map are input into the linear layer (Linear Layer), and Query (Q), Key (K), and Value (V) are extracted. The Query and Key are subjected to the Softmax function operation and the standard attention (Cross-Attention) operation, and then input into the Linear Layer together with the Value for feature extraction to obtain the target BEV feature map.

[0095] The N multi-layer image feature maps, the N depth feature maps, and the initial BEV feature map are input into the depth-guided attention mechanism module to perform the deformable attention mechanism process as follows: Figure 5 As shown in the figure, the offset of N multi-layer image feature maps is input into Linear Layer for feature extraction, and bitwise added with the initial BEV feature map to obtain the fused feature map, which is then input into Linear Layer for feature extraction, and after performing Softmax function operation and Cross-Attention operation, the feature map is combined with N depth feature maps and N multi-layer image feature maps. Figure 1 Input the Linear Layer for feature extraction to obtain the target BEV feature map.

[0096] Optionally, the Linear Layer may be a multi-layer perceptron (MLP) layer.

[0097] The present disclosure can improve the efficiency and accuracy of image detection, enhance the accuracy of feature space conversion, introduce depth information in the process of mapping multi-view images from different image spaces to a unified BEV space, improve the accuracy of feature space conversion, and perform accurate and efficient 3D target detection in the BEV space.

[0098] Figure 6 is a flow chart of a target detection method according to an embodiment of the present disclosure. Figure 6 As shown, the method includes the following steps:

[0099] S601, obtaining N perspective images to be detected at the current moment, and extracting N multi-layer image feature maps of the N perspective images to be detected.

[0100] Optionally, in an embodiment of the present disclosure, N image acquisition devices, such as cameras, are called to capture images from N perspectives. That is, the N image acquisition devices are deployed at different positions, and any image acquisition device corresponds to one perspective. The image of the current scene is captured to obtain N perspective images, and then features of the N perspective images are extracted through a neural network to obtain N multi-layer image feature maps.

[0101] Optionally, in an embodiment of the present disclosure, a residual network is used to extract image features. For example, ResNet50 or ResNet101 can be used to extract features from N perspective images to obtain N multi-layer image feature maps.

[0102] In some implementations, in order to optimize the feature extraction of multi-scale targets, Feature Pyramid Networks (FPN) can also be used to perform multi-scale fusion on the extracted multi-layer image feature maps.

[0103] S602: Perform depth estimation based on N multi-layer image feature maps to obtain N depth feature maps.

[0104] In some implementations, a multi-class classification task is performed on the pixels of each of the N multi-layer image feature maps, multiple depths are predicted for each pixel, and the probabilities of each depth are weighted to obtain the final depth value. This process is repeated to obtain N depth feature maps. Each pixel in the depth feature map has a depth value.

[0105] S603, input N multi-layer image feature maps and N depth feature maps into the target image detection model, and the target image detection model performs an attention mechanism based on the N multi-layer image feature maps, N depth feature maps and the model's previous BEV feature map to obtain a target BEV feature map.

[0106] The target image detection model is trained using the same training method as the image detection model. The first BEV feature map is randomly initialized and obtained by the server.

[0107] S604: Perform target detection on the target BEV feature map to obtain a target detection result.

[0108] In some implementations, the image detection model may further include an object detection network, which performs object detection on the target BEV feature map to obtain an object detection result. Optionally, the object detection network may be a deep neural network.

[0109] In an embodiment of the present disclosure, N perspective images to be detected at the current moment are obtained, and N multi-layer image feature maps of the N perspective images to be detected are extracted. Depth estimation is performed based on the N multi-layer image feature maps to obtain N depth feature maps. The N multi-layer image feature maps and the N depth feature maps are input into a target image detection model. The target image detection model performs an attention mechanism based on the N multi-layer image feature maps and the N depth feature maps and the model's previous BEV feature map to obtain a target BEV feature map. Target detection is performed on the target BEV feature map to obtain a target detection result. The present disclosure can improve image detection efficiency and accuracy, and enhance the accuracy of feature space conversion. Depth information is introduced in the process of mapping multi-perspective images from different image spaces to a unified BEV space, which can improve the accuracy of feature space conversion and perform accurate and efficient 3D target detection in the BEV space.

[0110] Figure 7 FIG. 1 is a structural diagram of a training device for an image detection model according to an embodiment of the present disclosure. Figure 7 As shown, the training device 700 of the image detection model includes:

[0111] A feature map acquisition module 710 is configured to perform feature extraction on N perspective images to obtain N multi-layer image feature maps, and perform depth estimation based on the N multi-layer image feature maps to obtain N depth feature maps, where N is a positive integer.

[0112] A training module 720 is configured to perform a (l+1)th training on the image detection model based on the N multi-layer image feature maps, the N depth feature maps, and the bird's-eye view (BEV) feature map l output from the lth training run, and output the BEV feature map l+1 and the target detection result, where l is a positive integer;

[0113] The adjustment module 730 is used to adjust the model parameters of the image detection model according to the depth feature map and the target detection result, and continue to train the adjusted image detection model until the target image detection model is obtained.

[0114] In some implementations, the training module 720 is further configured to:

[0115] The attention mechanism network in the image detection model performs attention on N multi-layer image feature maps, N depth feature maps, and BEV feature map l to obtain BEV feature map l+1;

[0116] The target detection network in the image detection model performs target detection on the BEV feature map l+1 to obtain the target detection result.

[0117] In some implementations, the training module 720 is further configured to:

[0118] Determine the transformation matrix from the lidar coordinate system to the image coordinate system;

[0119] Based on the transformation matrix, determine the target images to which the T query vectors corresponding to the BEV feature map l can be projected, where the target images are n images in the multi-view image, n is an integer, and 1≤n≤N, and T is a positive integer;

[0120] Determine the value vector and key vector of each layer of image feature map corresponding to the target image;

[0121] A multi-head global attention mechanism is performed pixel by pixel on the query vector and the value vector and key vector of each layer of image feature map to obtain the BEV feature map l+1.

[0122] In some implementations, the training module 720 is further configured to:

[0123] For query vector i among T query vectors, obtain the first matrix of the sth layer corresponding to the target image based on the query vector i and the value vector and key vector of the sth layer image feature map, and add the first matrices of each layer to obtain the second matrix of the attention head; where i and s are both integers and 1≤i≤T;

[0124] According to the second matrix of the attention head and the weight matrix of the attention head, the third matrix of the attention head is obtained, and the third matrix of each attention head is added to obtain the global attention matrix of the query vector i;

[0125] The global attention matrices of T query vectors are added together to obtain the BEV feature map l+1.

[0126] In some implementations, the training module 720 is further configured to:

[0127] Determine a first depth value of position j on the target image according to the depth feature map of the target image;

[0128] Determine a first weight of position j in the dimension of query vector i based on the second depth value and the first depth value of position j in query vector i;

[0129] According to the query vector i, the key of position j on the s-th layer image feature map, the weight of position j and the value of position j on the s-th layer image feature map, the first matrix of the s-th layer corresponding to the target image is obtained.

[0130] In some implementations, the training module 720 is further configured to:

[0131] Determine the transformation matrix from the lidar coordinate system to the image coordinate system;

[0132] Based on the transformation matrix, determine the projection image positions of the T query vectors corresponding to the BEV feature map l onto the target image; where the target image is n images in the multi-view image, n is an integer, and 1≤n≤N, and T is a positive integer;

[0133] Sampling the area around the projection image position to obtain a sampling position;

[0134] A multi-head deformable attention mechanism is performed on the query vector, the key vector of each layer of the image feature map, and the sampling position to obtain the BEV feature map l+1.

[0135] In some implementations, the training module 720 is further configured to:

[0136] For query vector i among the T query vectors, obtain the fourth matrix with the sampling position at the sth layer according to the query vector i, the sampling position and the sth layer image feature map;

[0137] Add the fourth matrix of each layer to get the fifth matrix of the attention head;

[0138] According to the fifth matrix of the attention head and the weight matrix of the attention head, the sixth matrix of the attention head is obtained, and the sixth matrix of each attention head is added to obtain the deformable attention matrix of the query vector i;

[0139] The deformable attention matrices of T query vectors are added together to obtain the BEV feature map l+1.

[0140] In some implementations, the training module 720 is further configured to:

[0141] Determine a third depth value of the sampling position f according to the depth feature map of the target image;

[0142] Determining a second weight of the sampling position f in the dimension of the query vector i according to the fourth depth value and the third depth value of the sampling position f in the query vector i;

[0143] Stretch the first coordinate of the query vector i to obtain the second coordinate at the s-th level;

[0144] According to the second coordinate, the second weight of the sampling position f and the value on the image feature map of the s-th layer, a fourth matrix of the sampling position at the s-th layer is obtained.

[0145] In some implementations, the adjustment module 730 is further configured to:

[0146] Obtaining a first loss function between the depth feature map and the reference depth feature map;

[0147] Obtaining a second loss function between the target detection result and the reference detection result;

[0148] According to the first loss function and the second loss function, a total loss function of the image detection model is obtained;

[0149] Adjust the model parameters of the image detection model according to the total loss function.

[0150] The present disclosure can improve the efficiency and accuracy of image detection, enhance the accuracy of feature space conversion, introduce depth information in the process of mapping multi-view images from different image spaces to a unified BEV space, improve the accuracy of feature space conversion, and perform accurate and efficient 3D target detection in the BEV space.

[0151] Figure 8 is a structural diagram of a target detection device according to an embodiment of the present disclosure, such as Figure 8 As shown, the target detection device 800 includes:

[0152] The extraction module 810 is used to obtain N perspective images to be detected at the current moment and extract N multi-layer image feature maps of the N perspective images to be detected;

[0153] A depth estimation module 820 is configured to perform depth estimation based on the N multi-layer image feature maps to obtain N depth feature maps;

[0154] An acquisition module 830 is configured to input the N multi-layer image feature maps and the N depth feature maps into a target image detection model, and the target image detection model performs an attention mechanism based on the N multi-layer image feature maps and the N depth feature maps and the previous BEV feature map of the model to obtain a target BEV feature map;

[0155] The target detection module 840 is used to perform target detection on the target BEV characteristic map to obtain a target detection result;

[0156] The target image detection model is trained using a training device such as an image detection model.

[0157] The present disclosure can improve the efficiency and accuracy of image detection, enhance the accuracy of feature space conversion, introduce depth information in the process of mapping multi-view images from different image spaces to a unified BEV space, improve the accuracy of feature space conversion, and perform accurate and efficient 3D target detection in the BEV space.

[0158] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0159] Figure 9 is a block diagram of an electronic device used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0160] like Figure 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0161] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0162] The computing unit 901 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the training method of the image detection model or the target detection method. For example, in some embodiments, the training method of the image detection model or the target detection method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method of the image detection model or the target detection method described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the image detection model training method or the target detection method in any other appropriate manner (for example, by means of firmware).

[0163] In some implementations, the present disclosure also provides an autonomous driving vehicle comprising the electronic device as described above.

[0164] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0166] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0168] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0169] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0170] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0171] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A training method for an image detection model, wherein: The method comprises: right N Extract features from the perspective images and get N Multi-layer image feature maps, and based on the N Multi-layer image feature maps are used for depth estimation to obtain N A depth feature map, the N is a positive integer; Determine the transformation matrix from the lidar coordinate system to the image coordinate system; Based on the conversion matrix, the BEV characteristic map is determined l Corresponding T The query vector can be projected to the target image, wherein the target image is the target image in the multi-view image. n images, the l , n is a positive integer, and 1≤ n ≤ N , T is a positive integer; Determine the value vector and key vector of each layer of image feature map corresponding to the target image; A multi-head global attention mechanism is performed on the query vector and the value vector and key vector of each layer of the image feature map at each pixel position to obtain the BEV feature map l +1; The target detection network in the image detection model detects the BEV feature map l +1 performs target detection to obtain the target detection result; According to the depth feature map and the target detection result, the model parameters of the image detection model are adjusted, and the adjusted image detection model is continuously trained until the target image detection model is obtained.

2. The method according to claim 1, wherein The multi-head global attention mechanism is performed on the query vector and the value vector and key vector of each layer of the image feature map at each pixel position to obtain the BEV feature map l +1, including: Regarding the T query vectors in query vectors i , according to the query vector i Hedi s The value vector and key vector of the layer image feature map are obtained to obtain the first layer corresponding to the target image. s The first matrix of the layer, and add the first matrix of each layer to get the second matrix of the attention head; i and s are all integers, and 1≤ i ≤ T ; According to the second matrix of the attention head and the weight matrix of the attention head, the third matrix of the attention head is obtained, and the third matrix of each attention head is added to obtain the query vector i The global attention matrix of Regarding the T The global attention matrix of the query vectors is added to obtain the BEV feature map l +1.

3. The method according to claim 2, wherein: According to the query vector i Hedi s The value vector and key vector of the image feature map are obtained to obtain the first s The first matrix of the layer, including: Determine the position on the target image based on the depth feature map of the target image j The first depth value of According to the query vector i The location described j The second depth value and the first depth value determine the position j In the query vector i The first weight on the dimension; According to the query vector i , the location j In the s The key and position on the layer image feature map j The weight and position of j In the s The value on the layer image feature map is obtained to obtain the first layer corresponding to the target image. s The first matrix of the layer.

4. The method according to claim 1, wherein The method further comprises: Based on the conversion matrix, the BEV characteristic map is determined l Corresponding T The query vector is projected to the projected image position on the target image; Sampling the area around the projection image position to obtain a sampling position; A multi-head deformable attention mechanism is performed on the query vector, the key vector of each layer of the image feature map and the sampling position to obtain the BEV feature map l +1.

5. The method according to claim 4, wherein The multi-head deformable attention mechanism is performed on the query vector, the key vector of each layer of the image feature map and the sampling position to obtain the BEV feature map l +1, including: Regarding the T query vectors in query vectors i , according to the query vector i , the sampling position and the s Layer image feature map, get the sampling position in the s The fourth matrix of the layer; Add the fourth matrix of each layer to get the fifth matrix of the attention head; According to the fifth matrix of the attention head and the weight matrix of the attention head, the sixth matrix of the attention head is obtained, and the sixth matrix of each attention head is added to obtain the query vector i The deformable attention matrix of Regarding the T The deformable attention matrix of the query vector is added to obtain the BEV feature map l +1.

6. The method according to claim 5, wherein: According to the query vector i , the sampling position and the s Layer image feature map, get the sampling position in the s The fourth matrix of the layer includes: Determine the sampling position according to the depth feature map of the target image f The third depth value of According to the query vector i Sampling location as described in f The fourth depth value and the third depth value are used to determine the sampling position f In the query vector i Second weight on the dimension; For the query vector i The first coordinate of the coordinate is stretched to obtain the s The second coordinate under the layer size; According to the second coordinate, the sampling position f The second weight and the first s The value on the layer image feature map is obtained by s The fourth matrix of the layer.

7. The method according to any one of claims 1 to 6, wherein The adjusting model parameters of the image detection model according to the depth feature map and the target detection result includes: Obtaining a first loss function between the depth feature map and a reference depth feature map; Obtaining a second loss function between the target detection result and a reference detection result; Obtaining a total loss function of the image detection model according to the first loss function and the second loss function; The model parameters of the image detection model are adjusted according to the total loss function.

8. A target detection method, wherein: include: Get the current time to be detected N perspective images, and extract the N Perspective image N Multi-layer image feature maps; Based on the N Multi-layer image feature maps are used for depth estimation to obtain N A deep feature map; The N multi-layer image feature maps and the N The depth feature map is input into the target image detection model, and the target image detection model is based on the N multi-layer image feature maps and the N The depth feature map and the previous BEV feature map of the model are subjected to attention mechanism to obtain the target BEV feature map; Perform target detection on the target BEV feature map to obtain the target detection result; The target image detection model is trained using the training method described in any one of claims 1 to 7.

9. A training device for an image detection model, wherein: The device comprises: Feature map acquisition module, used to N Extract features from the perspective images and get N Multi-layer image feature maps, and based on the N Multi-layer image feature maps are used for depth estimation to obtain N A depth feature map, the N is a positive integer; A training module for determining the transformation matrix from the lidar coordinate system to the image coordinate system; Based on the conversion matrix, the BEV characteristic map is determined l Corresponding T The query vector can be projected to the target image, wherein the target image is the target image in the multi-view image. n images, the n is an integer, and 1≤ n ≤ N , T is a positive integer; Determine the value vector and key vector of each layer of image feature map corresponding to the target image; A multi-head global attention mechanism is performed on the query vector and the value vector and key vector of each layer of the image feature map at each pixel position to obtain the BEV feature map l +1; The target detection network in the image detection model detects the BEV feature map l +1 performs target detection to obtain the target detection result; An adjustment module is used to adjust the model parameters of the image detection model according to the depth feature map and the target detection result, and to continue training the adjusted image detection model until a target image detection model is obtained.

10. The device according to claim 9, wherein The training module is further used to: Regarding the T query vectors in query vectors i , according to the query vector i Hedi s The value vector and key vector of the layer image feature map are obtained to obtain the first layer corresponding to the target image. s The first matrix of the layer, and add the first matrix of each layer to get the second matrix of the attention head; in, i and s are all integers, and 1≤ i ≤ T ; According to the second matrix of the attention head and the weight matrix of the attention head, the third matrix of the attention head is obtained, and the third matrix of each attention head is added to obtain the query vector i The global attention matrix of Regarding the T The global attention matrix of the query vectors is added to obtain the BEV feature map l +1.

11. The device according to claim 10, wherein The training module is further used to: Determine the position on the target image based on the depth feature map of the target image j The first depth value of According to the query vector i The location described j The second depth value and the first depth value determine the position j In the query vector i The first weight on the dimension; According to the query vector i , the location j In the s The key and position on the layer image feature map j The weight and position of j In the s The value on the layer image feature map is obtained to obtain the first layer corresponding to the target image. s The first matrix of the layer.

12. The device according to claim 9, wherein The training module is further used to: Based on the conversion matrix, the BEV characteristic map is determined l Corresponding T The query vector is projected to the projected image position on the target image; Sampling the area around the projection image position to obtain a sampling position; A multi-head deformable attention mechanism is performed on the query vector, the key vector of each layer of the image feature map and the sampling position to obtain the BEV feature map l +1.

13. The device according to claim 12, wherein The training module is further used to: Regarding the T query vectors in query vectors i , according to the query vector i , the sampling position and the s Layer image feature map, get the sampling position in the s The fourth matrix of the layer; Add the fourth matrix of each layer to get the fifth matrix of the attention head; According to the fifth matrix of the attention head and the weight matrix of the attention head, the sixth matrix of the attention head is obtained, and the sixth matrix of each attention head is added to obtain the query vector i The deformable attention matrix of Regarding the T The deformable attention matrix of the query vector is added to obtain the BEV feature map l +1.

14. The device according to claim 13, wherein The training module is further used to: Determine the sampling position according to the depth feature map of the target image f The third depth value of According to the query vector i Sampling location as described in f The fourth depth value and the third depth value are used to determine the sampling position f In the query vector i Second weight on the dimension; For the query vector i The first coordinate of the coordinate is stretched to obtain the s The second coordinate under the layer size; According to the second coordinate, the sampling position f The second weight and the first s The value on the layer image feature map is obtained by s The fourth matrix of the layer.

15. The device according to any one of claims 9 to 14, wherein: The adjustment module is further used to: Obtaining a first loss function between the depth feature map and a reference depth feature map; Obtaining a second loss function between the target detection result and a reference detection result; Obtaining a total loss function of the image detection model according to the first loss function and the second loss function; The model parameters of the image detection model are adjusted according to the total loss function.

16. A target detection device, wherein: include: Extraction module, used to obtain the current moment to be detected N perspective images, and extract the N Perspective image N Multi-layer image feature maps; Depth estimation module, for N Multi-layer image feature maps are used for depth estimation to obtain N A deep feature map; Acquisition module, used to N multi-layer image feature maps and the N The depth feature map is input into the target image detection model, and the target image detection model is based on the N multi-layer image feature maps and the N The depth feature map and the previous BEV feature map of the model are subjected to attention mechanism to obtain the target BEV feature map; The target detection module is used to perform target detection on the target BEV feature map and obtain the target detection result; Wherein, the target image detection model is trained using the device described in any one of claims 9 to 15.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7 or the method according to claim 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the steps of the method according to any one of claims 1 to 7 or to execute the steps of the method according to claim 8.

19. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7 or the method according to claim 8.

20. An autonomous driving vehicle comprising the electronic device according to claim 17.

Images