Training method and device for image feature extraction module in BEV detection architecture

Abstract

The invention provides a training method and device for an image feature extraction module in a BEV detection architecture, and relates to the technical field of BEV detection architecture training, and the method comprises the steps: comparing supervision information outputted by a supervision information detection head with a ground real label, and determining a first loss value; performing forward propagation on the image features along a BEV detection architecture, and outputting a second loss value; constructing a loss function of the image feature extraction module based on the first loss value and the second loss value, and determining a third loss value; updating the target parameter in the image feature extraction module according to the third loss value, and executing the above steps again on the updated image feature extraction module until the second loss value and the third loss value converge; through a mode of directly supervising the image features, the image feature extraction module has better feature extraction capability, so that the technical problem that the accuracy of BEV detection results is limited is relieved.

Description

Technical Field

[0001] The present invention relates to the technical field of BEV detection architecture training, and in particular to a training method and device for an image feature extraction module in a BEV detection architecture. Background Art

[0002] With the rapid development of autonomous driving and intelligent transportation systems, bird's eye view (BEV)-based object detection technology has become a research hotspot. Traditional BEV detection methods usually rely on three-dimensional data provided by sensors such as lidar to construct BEV space and perform object detection on this basis. Purely visual BEV detection has attracted widespread attention due to its low cost and easy deployment.

[0003] The inventors have found that traditional BEV detection methods mostly perform supervised training in the BEV space. This method only provides indirect implicit supervision on the image feature extraction layer, which is not conducive to fully utilizing the potential of front-end image feature extraction, and limits the accuracy of BEV detection results. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a training method and device for an image feature extraction module in a BEV detection architecture, which enables the image feature extraction module to have better feature extraction capabilities by directly supervising the image features, thereby alleviating the technical problem in the prior art that the accuracy of BEV detection results is limited.

[0005] In a first aspect, an embodiment provides a training method for an image feature extraction module in a BEV detection architecture, comprising:

[0006] Extracting image features from training sample data under the action of an image feature extraction module in the BEV detection architecture; wherein the training sample data includes image data and corresponding ground truth labels of the image data;

[0007] Input the image feature into a supervision information detection head, and compare the supervision information corresponding to the image feature output by the supervision information detection head with the ground truth label to determine a first loss value of the image feature; wherein the supervision information includes three-dimensional information and depth information;

[0008] Forward propagating the image feature along the BEV detection architecture, and outputting a second loss value of the image feature;

[0009] constructing a loss function of the image feature extraction module based on the first loss value and the second loss value, and determining a third loss value of the image feature;

[0010] The target parameters in the image feature extraction module are updated according to the third loss value, and the updated image feature extraction module is used as a new image feature extraction module to again execute the step of extracting image features from the sample data under the action of the image feature extraction module, until the second loss value and the third loss value output by the image feature extraction module under the action of the training sample data converge.

[0011] In an optional implementation, the step of inputting the image feature into a supervision information detection head, and comparing the supervision information corresponding to the image feature output by the supervision information detection head with the ground truth label to determine the first loss value of the image feature includes:

[0012] Inputting the image feature into a three-dimensional information detection head, so that the three-dimensional information detection head outputs the number of objects in the image feature and the three-dimensional information of each object;

[0013] Based on the ground truth label and the three-dimensional information of each of the objects, a three-dimensional information loss value of each object is determined.

[0014] In an optional implementation, the step of inputting the image feature into a supervision information detection head, and comparing the supervision information corresponding to the image feature output by the supervision information detection head with the ground truth label to determine the first loss value of the image feature further includes:

[0015] Inputting the image feature into a depth information detection head so that the depth information detection head outputs depth information of each point corresponding to the image feature;

[0016] Based on the ground truth label and the depth information of each point, a depth information loss value of each point corresponding to the image feature is determined.

[0017] In an optional embodiment, under the action of the image feature extraction module in the BEV detection architecture, before the step of extracting image features from the training sample data, the method further includes:

[0018] A large-scale 2D data set is used to pre-train an image feature extraction module of a BEV detection architecture, and a target parameter corresponding to the image feature extraction module is initialized; wherein the target parameter is associated with the image feature.

[0019] In an optional embodiment, the BEV detection architecture includes the image feature extraction module, the view conversion module, the BEV space feature extraction module and the 3D object detection head connected in sequence; the step of forward propagating the image feature along the BEV detection architecture and outputting the second loss value of the image feature includes:

[0020] The image features are converted into BEV space by the view conversion module, the BEV space features are extracted by the BEV space feature extraction module, and the target information is obtained by the 3D target detection head;

[0021] Outputting a second loss value of the image feature based on the target information and the ground truth label.

[0022] In an optional embodiment, the step of constructing a loss function of the image feature extraction module based on the first loss value and the second loss value, and determining a third loss value of the image feature includes:

[0023] Assigning preset weight coefficients to the second loss value, and the three-dimensional information loss value and the depth information loss value in the first loss value respectively, and then adding them up to construct a loss function of the image feature extraction module;

[0024] When the first loss value and the second loss value are back-propagated along the BEV detection architecture, a loss function is input to obtain a third loss value of the image feature.

[0025] In an optional embodiment, under the action of the image feature extraction module in the BEV detection architecture, the step of extracting image features from the training sample data includes:

[0026] The training sample data is input into the image feature extraction module of the BEV detection architecture, and the image features are output after being processed by the image feature extraction module.

[0027] In a second aspect, an embodiment provides a training device for an image feature extraction module in a BEV detection architecture, comprising:

[0028] An extraction module, under the action of the image feature extraction module in the BEV detection architecture, extracts image features from training sample data; wherein the training sample data includes image data and corresponding ground truth labels of the image data;

[0029] A first determination module inputs the image feature into a supervision information detection head, and compares the supervision information corresponding to the image feature output by the supervision information detection head with the ground truth label to determine a first loss value of the image feature; wherein the supervision information includes three-dimensional information and depth information;

[0030] An output module forward-propagates the image feature along the BEV detection architecture and outputs a second loss value of the image feature;

[0031] a second determination module, which constructs a loss function of the image feature extraction module based on the first loss value and the second loss value, and determines a third loss value of the image feature;

[0032] An updating module updates the target parameters in the image feature extraction module according to the third loss value, and uses the updated image feature extraction module as a new image feature extraction module to again execute the step of extracting image features from the sample data under the action of the image feature extraction module, until the second loss value and the third loss value output by the image feature extraction module under the action of the training sample data converge.

[0033] In a third aspect, an embodiment provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method described in any of the aforementioned implementation methods are implemented.

[0034] In a fourth aspect, an embodiment provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the steps of the method described in any one of the aforementioned implementations.

[0035] An embodiment of the present invention provides a training method for an image feature extraction module in a BEV detection architecture, wherein the image feature extraction module extracts image features from training sample data and inputs the image features into a detection head of supervisory information so that the detection head of the supervisory information outputs a first loss value; the image features output a second loss value under the action of the remaining modules of the forward propagation of the BEV detection architecture; a third loss value is jointly constructed based on the first loss value and the second loss value, and the third loss value is applied to the image feature extraction module to update the target parameters; the updated image feature extraction module repeats the above steps until the second loss value and the third loss value output by the updated image feature extraction module converge; the image feature extraction module trained in this way has better feature extraction capability, thereby further improving the accuracy of the detection results of the BEV detection architecture.

[0036] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.

[0037] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flow chart of a training method for an image feature extraction module in a BEV detection architecture provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a training architecture of a BEV detection architecture provided by an embodiment of the present invention;

[0041] Figure 3 A functional module diagram of a training device for an image feature extraction module in a BEV detection architecture provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the hardware architecture of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The inventors have found that current BEV detection architectures are mostly supervised and trained in the BEV space. This supervision method is indirect, which is not conducive to effectively improving the feature extraction effect of the image feature extraction module, thereby affecting the accuracy of the detection results of the BEV detection architecture.

[0045] Based on this, an embodiment of the present invention provides a training method and device for an image feature extraction module in a BEV detection architecture, which adopts multiple means to enhance the ability of image feature extraction in BEV detection tasks, so that it can play a greater role in BEV detection tasks.

[0046] To facilitate understanding of this embodiment, a training method for an image feature extraction module in a BEV detection architecture disclosed in an embodiment of the present invention is first introduced in detail. This method can be applied to intelligent control devices such as controllers, servers, and host computers; for example, it is implemented in a scenario where a BEV detection architecture is trained.

[0047] Figure 1 A flow chart of a training method for an image feature extraction module in a BEV detection architecture provided in an embodiment of the present invention.

[0048] like Figure 1 As shown, the method comprises the following steps:

[0049] Step S102 : extracting image features from training sample data under the action of the image feature extraction module in the BEV detection framework.

[0050] Among them, the BEV detection architecture is used to identify targets and their corresponding feature information from image data; the training sample data includes image data and the corresponding ground truth label GT of the image data; the ground truth label includes the 2D position of the center point of the target in the image data, the three-dimensional information corresponding to the length, width, height and other size information of the target, the number of targets in the image data, the specific category corresponding to each target, and the depth information of each point in the image data; the point can be understood as a pixel point.

[0051] In machine learning, GT is short for ground truth, which refers to the real labels or data used by the model in training and testing. This data is usually derived from manual annotation or physical measurements in the real world and is used to evaluate the accuracy and performance of the model. For example, when training an image classification model, GT is the label of each category of the manually annotated image, which is used to train and test the model. In deep learning, GT is also called target or label output, which refers to the output that represents the category or regression value of the training sample. These labels usually correspond to each instance in the training data set, and the model updates the weights and biases by comparing its output with the actual label, thereby improving the accuracy of the model. During the testing process, GT is used to evaluate the performance of the model and the correctness of the predicted results.

[0052] Step S104: input the image feature into the supervision information detection head, and compare the supervision information corresponding to the image feature output by the supervision information detection head with the ground truth label to determine the first loss value of the image feature.

[0053] The supervision information includes three-dimensional information and depth information. In the embodiment of the present invention, when the image feature extraction module outputs the image feature, the image feature is input into the supervision information detection head to output the corresponding supervision information, and the two kinds of supervision information including the three-dimensional information and the depth information are directly applied to the image feature to obtain the first loss value under the influence of the supervision information. It can be understood that the first loss value includes the three-dimensional information loss value L img And the depth information loss value L depth .

[0054] Step S106, forward propagating the image features along the BEV detection architecture, and outputting a second loss value of the image features.

[0055] Here, during the training process of the BEV detection architecture, the image features output by the image feature extraction module will also be processed by the remaining architecture structures along the forward propagation direction to output the second loss value L of the image features. BEV .

[0056] Step S108: construct a loss function of the image feature extraction module based on the first loss value and the second loss value, and determine a third loss value of the image feature.

[0057] Among them, the three-dimensional information loss value L is used img And the depth information loss value L depth The first loss value, and the second loss value L BEV A loss function of the image feature extraction module is constructed, and based on the numerical input of the first loss value and the second loss value, an output of the loss function, i.e., a third loss value, is obtained.

[0058] Step S110, update the target parameters in the image feature extraction module according to the third loss value, and use the updated image feature extraction module as a new image feature extraction module to execute step S102 again until the second loss value and the third loss value output by the image feature extraction module under the action of the training sample data converge.

[0059] It should be noted that when the above-mentioned loss value is back-propagated, the third loss value is used to update the target parameter in the image feature extraction module. The target parameter is associated with the image features extracted by the image feature extraction module. In other words, by modifying the target parameter, the feature extraction capability of the image feature extraction module can be adjusted accordingly. After the structure is updated, the aforementioned step S102 is executed again, wherein after step S102 is executed, the execution subject of the method of the embodiment of the present invention will naturally execute steps S104-S110 in sequence. Repeat this process until the second loss value and the third loss value determined by the image features output by the updated image feature extraction module converge, and terminate the training process.

[0060] In a preferred embodiment of practical application, the image feature extraction module extracts image features from training sample data and inputs them into the detection head of the supervisory information so that the detection head of the supervisory information will output a first loss value; the image feature outputs a second loss value under the action of the remaining modules of the forward propagation of the BEV detection architecture; a third loss value is jointly constructed based on the first loss value and the second loss value, and the third loss value is applied to the image feature extraction module to update the target parameters; the updated image feature extraction module repeats the above steps until the second loss value and the third loss value output by the updated image feature extraction module converge; the image feature extraction module trained in this way has better feature extraction capabilities, thereby further improving the accuracy of the detection results of the BEV detection architecture.

[0061] In some embodiments, in order to establish a target point, that is, to initialize the image feature extraction module and limit it to a relatively accurate smaller spatial range, before step S102, the method further includes:

[0062] Step 1.1), using a large-scale 2D dataset, pre-train the image feature extraction module of the BEV detection architecture and initialize the target parameters corresponding to the image feature extraction module.

[0063] In order to enable the image feature extraction module to quickly extract high-quality image features at the beginning of training, this solution uses a large-scale 2D detection dataset to pre-train the backbone network of the image feature extraction module (image encoder) to provide strong initialization target parameters for the extraction of 2D image features.

[0064] It should be noted that only the image feature extraction module in the BEV detection architecture is initialized using the target parameters obtained by pre-training based on a large-scale 2D detection dataset, while the remaining module layers in the BEV detection architecture are randomly initialized.

[0065] On the basis of the above steps, the initialized image feature extraction module, under the application of this pre-training strategy, significantly improves the convergence speed of subsequent learning and the richness and accuracy of image feature representation during the learning process; at this time, step S102 can input the training sample data into the image feature extraction module of the BEV detection architecture, and output the image features after processing by the image feature extraction module, and the output efficiency of the image features is significantly improved.

[0066] In practical applications, some BEV supervision signals are often directly applied to BEV features in BEV space. This supervision method is indirect and sparse for image feature extraction. In terms of indirectness, since the supervision signal (loss) acts directly on the BEV features, it becomes indirect after being projected from the BEV space to the 2D image view control and then sampled for image features during the back propagation process. In terms of sparsity, only a few BEV grids that are concerned by the object query will contribute to the loss. Therefore, only the sparse pixels near the 2D reference points corresponding to these few BEV grids will receive supervision signals. These result in the loss on BEV to supervise image features sparsely and indirectly, which will ultimately reduce the recall and precision of the BEV detection architecture.

[0067] The embodiment of the present invention introduces a dense 3D supervision mechanism that directly acts on image features. In this way, richer and more direct three-dimensional information about the target can be fed back to the image feature extraction module, thereby helping it to more effectively learn and extract key features in three-dimensional space. This method not only enhances the model's ability to capture three-dimensional structures, but also improves the robustness and accuracy of the entire BEV detection architecture in complex environments; for example, Figure 2 As shown, step S104 includes:

[0068] Step 2.1), input the image features into the three-dimensional information detection head, so that the three-dimensional information detection head outputs the number of targets in the image features and the three-dimensional information of each target.

[0069] The three-dimensional information detection head Image-based 3D head can be understood as a virtual processing module, which can perform calculations on the input image features to output the number of targets in the image features and the three-dimensional information of each target.

[0070] In step 2.2), based on the ground truth label and the three-dimensional information of each object, the three-dimensional information loss value of each object is determined.

[0071] Here, the ground truth label can know the number of targets corresponding to the image feature and the 3D information of each target. The 3D information of each target output by the 3D information detection head is compared with the 3D information of each target in the ground truth label to obtain the 3D information loss value L of each target. img .

[0072] The embodiment of the present invention adds a three-dimensional information detection head to the training architecture of the image feature extraction module to enhance the image feature's ability to capture three-dimensional spatial information. This design can significantly improve the effect of image features in extracting three-dimensional spatial semantic information.

[0073] In actual applications, the inventors have found that since the BEV detection architecture uses depth information when performing perspective conversion, inaccurate depth will lead to errors in feature conversion, ultimately affecting the final detection effect of the model.

[0074] In the traditional BEV detection architecture training process, if the depth loss is not introduced, the result is usually that the implicitly learned depth information is not accurate and consistent. This is mainly manifested in: 1) In the detection architecture without dense depth supervision, the gradient of its depth module comes from the target detection loss of the BEV detection architecture, that is, the second loss value L BEV ; When a BEV detection architecture is trained without depth loss, it detects objects by learning only part of the depth. This implicitly learned depth is inaccurate; 2) Most pixels are not trained to predict reasonable depth values, resulting in limited generalization of the depth module, especially for specific target categories with limited sample size and ground truth labels (GT) that are difficult to cover at different viewing distances, such as the inability to accurately know the exact distance between the vehicle and the target pedestrian, resulting in a high risk of overfitting; 3) The lack of effective depth guidance will make it impossible to accurately project image features into 3D space, thereby affecting the accuracy of 3D semantic information and hindering BEV detection. Inaccurate and inconsistent depth information learning will ultimately lead to insufficient accuracy in the 3D position and orientation of the target detected by the BEV detection architecture.

[0075] Based on this, a dense deep supervision mechanism is introduced. On the basis of the above-mentioned embodiment, step S104 further includes:

[0076] Step 3.1), inputting the image feature into the depth information detection head, so that the depth information detection head outputs the depth information of each point corresponding to the image feature;

[0077] Among them, the depth information detection head Depth head can be understood as a virtual processing module, which can perform calculations on the input image features to output the depth information of each point corresponding to the image features; wherein the image features may include multiple dimensions, such as 100*100*256.

[0078] Step 3.2), based on the ground truth label and the depth information of each point, determine the depth information loss value of each point corresponding to the image feature.

[0079] Here, the depth information of each point corresponding to the image feature in the ground truth label is first dimensionally aligned according to the image feature, and then the depth information of each aligned point is compared with the depth information of each point corresponding to the image feature output by the depth information detection head to obtain the depth information loss value L of each point.depth .

[0080] The embodiment of the present invention combines the depth information obtained by adding the depth information detection head during the training process with the high-precision depth label to directly guide the extraction of depth features of the image feature extraction module, thereby giving the image feature extraction module stronger depth perception capabilities. Through dense depth supervision, the image feature extraction module can not only accurately learn the depth information of the target, but also grasp the depth distribution of the entire image scene, enhance the robustness of depth estimation, and ultimately improve the accuracy of the BEV detection architecture in detecting the three-dimensional information and orientation of the target.

[0081] Based on the above embodiments, illustratively, Figure 2 As shown, the image feature extraction module extracts image features from the training sample data in the image view space (2D space), and the image features are processed along three paths at the same time; including forward propagation along the BEV detection architecture, and also including inputting the image features into the three-dimensional information detection head and the depth information detection head for processing; the three-dimensional information detection head processes the image features and outputs a three-dimensional information loss value L img The depth information detection head processes the image features and outputs the depth information loss value L depth .

[0082] On this basis, the BEV detection architecture includes an image feature extraction module, a view conversion module, a BEV spatial feature extraction module and a 3D object detection head connected in sequence;

[0083] Among them, the image feature extraction module Image-view Encoder is responsible for extracting two-dimensional image features from the training sample data, that is, the original image; the view transformation module View Transformer maps these image features to the BEV space to obtain BEV features; then the BEV space feature extraction module BEV Encoder further extracts the BEV space features, and finally the 3D target detection head part is used to detect the target location and related target information; illustratively, under the action of the forward propagation function of the BEV detection architecture, the step S106 of outputting the second loss value of the image feature includes:

[0084] Step 4.1), convert the image features into BEV space through the view conversion module, extract the BEV space features through the BEV space feature extraction module, and then obtain the target information through the 3D target detection head.

[0085] like Figure 2 As shown, with the joint action of other modules in the BEV detection architecture, the target information corresponding to the image features can be obtained.

[0086] Step 4.2), output the second loss value of the image feature based on the target information and the ground truth label.

[0087] Here, the target information in the ground truth label is compared with the target information output by the 3D target detection head to obtain the second loss value L of the image feature. BEV .

[0088] In some embodiments, a loss function of the image feature extraction module may be established according to each loss value determined in the above embodiments; illustratively, step S108 includes:

[0089] Step 5.1), respectively assign preset weight coefficients to the second loss value, and the three-dimensional information loss value and the depth information loss value in the first loss value, and then add them up to construct a loss function of the image feature extraction module.

[0090] As an optional embodiment, the loss function may be:

[0091] L all =α*L bev +β*L img +γ*L depth

[0092] Among them, L all is the third loss value, α, β, and γ are weight coefficients for balancing the three loss values. In practical applications, the weight coefficients can be pre-set according to actual conditions. For example, α is selected as 0.5, β is selected as 0.3, and γ is selected as 0.2.

[0093] Step 5.2), when the first loss value and the second loss value are back-propagated along the BEV detection architecture, the loss function is input to obtain the third loss value of the image feature.

[0094] It should be noted that the first loss value is generated when the image feature extraction module outputs the image feature. At this time, the image feature is forward propagated along the BEV detection architecture at the same time, so that the BEV detection architecture outputs the second loss value; when the second loss value is reversely propagated along the BEV detection architecture, the corresponding parameters of the view conversion module, the BEV spatial feature extraction module and the 3D target detection head of the BEV detection architecture are modified based on the second loss value. At this time, the second loss value and the first loss value are merged between the image feature extraction module and the view conversion module, and work together to generate a third loss value which is then applied to the image feature extraction module; the BEV detection architecture is optimized in the above manner, so that the image feature extraction module in the BEV detection can work more accurately and robustly, thereby improving the final detection rate and the accuracy of the three-dimensional information of the detected target.

[0095] It should also be noted that the embodiment of the present invention only adds a three-dimensional information detection head image-based 3D head and a depth information detection head Depth head in the training stage of the BEV detection architecture to enhance the feature extraction capability of the image feature extraction module, while these two detection heads are removed in the application stage of the trained BEV detection architecture (these two detection head structures are not required in the trained BEV detection architecture), that is, the overall training improvement of the BEV detection architecture will not affect the actual reasoning application speed of the BEV detection architecture.

[0096] The embodiment of the present invention first uses a large-scale 2D detection data set for pre-training, and adopts a pre-training method to initialize and assign values ​​to the target parameters of the image feature extraction module, thereby providing a powerful initialization for image feature extraction; then, a three-dimensional information detection head based on image features is added to the image feature extraction module, thereby further enhancing the image feature extraction module's ability to capture three-dimensional spatial information; finally, a depth information detection head for dense depth supervision is added to the image feature extraction module, and additional depth information is added to the training of the image feature extraction module for supervision, and high-precision depth labels are used to directly guide the image feature extraction module to enable it to have the ability to extract deep features.

[0097] In some embodiments, the BEV detection architecture based on the BEV det method mainly includes four parts: an image feature extraction module Image-view Encoder, a view conversion module View Transformer, a BEV spatial feature extraction module BEV Encoder, and a 3D target detection head Head. Here, as an optional embodiment, the overall training process of the BEV detection architecture may include the following steps:

[0098] The input training sample data of the image feature extraction module is the original image data 6×W collected by 6 cameras in ×H in , the 2D image features of each camera are obtained through the CNN convolutional neural network Among them, W in and H in are the width and height of the original image captured by the camera, and C is the dimension of extracting image features;

[0099] At this time, the image feature extraction module is followed by the three-dimensional information detection head Image-based 3d head and the depth information detection head Depth head, and these two detection heads respectively express the image features to obtain two loss values ​​loss: L img and L depth .

[0100] At the same time, the image features are forward propagated along the BEV detection architecture; the view conversion module converts the image features from 2D to 3D space by combining the image features with the depth information, so that the image features are converted to three-dimensional space, and further projected to the BEV space to obtain the feature X×Y×C. Here X represents the horizontal distance range under BEV, Y represents the vertical distance range under BEV, and C represents the image feature dimension;

[0101] The BEV space feature extraction module is a module that extracts features in the BEV space. It uses a convolutional neural network (CNN) method and finally further extracts the feature X×Y×C row and column dimensions to generate BEV features X×Y×N; where N is the feature dimension after extraction.

[0102] The 3D target detection head receives the BEV feature X×Y×N, outputs the target position and related target information, and outputs the loss value L bev Then the loss is returned;

[0103] At this time, based on the loss value L bev Perform back propagation to update the corresponding parameters of the 3D object detection head, BEV spatial feature extraction module and view conversion module in turn;

[0104] At the loss value L bev , L img , L depth The loss function of the image feature extraction module is obtained through the joint action, and the target parameters of the image feature extraction module are updated;

[0105] Repeat the above training steps using the updated BEV detection architecture until the loss value L bev , L bev , L img , L depth The process terminates when the loss values ​​generated by the joint action converge.

[0106] The application of the embodiments of the present invention will improve the accuracy of the image feature extraction module, enable the BEV detection architecture to obtain richer and more detailed environmental information, and enhance the image feature extraction module's ability to capture depth information, thereby improving the representation capability of three-dimensional spatial information; at the same time, the overall performance of the BEV detection architecture is improved, especially in complex scenarios, providing new ideas and methods for the subsequent research and development of related technologies; it has brought substantial improvements to camera-based BEV detection technology, and is expected to be widely used in autonomous driving and related fields.

[0107] In some embodiments, Figure 3 As shown, the embodiment of the present invention further provides a training device 200 for an image feature extraction module in a BEV detection architecture, comprising:

[0108] An extraction module 201 extracts image features from training sample data under the action of an image feature extraction module in the BEV detection architecture; wherein the training sample data includes image data and corresponding ground truth labels of the image data;

[0109] The first determination module 202 inputs the image feature into a supervision information detection head, and compares the supervision information corresponding to the image feature output by the supervision information detection head with the ground truth label to determine a first loss value of the image feature; wherein the supervision information includes three-dimensional information and depth information;

[0110] An output module 203 forward-propagates the image feature along the BEV detection architecture and outputs a second loss value of the image feature;

[0111] A second determination module 204 constructs a loss function of the image feature extraction module based on the first loss value and the second loss value, and determines a third loss value of the image feature;

[0112] The updating module 205 updates the target parameters in the image feature extraction module according to the third loss value, and uses the updated image feature extraction module as a new image feature extraction module to again execute the step of extracting image features from the sample data under the action of the image feature extraction module, until the second loss value and the third loss value output by the image feature extraction module under the action of the training sample data converge.

[0113] Furthermore, the first determination module 202 is also specifically used to input the image features into a three-dimensional information detection head so that the three-dimensional information detection head outputs the number of targets in the image features and the three-dimensional information of each target; based on the ground truth label and the three-dimensional information of each target, determine the three-dimensional information loss value of each target.

[0114] Furthermore, the first determination module 202 is also specifically used to input the image feature into the depth information detection head so that the depth information detection head outputs the depth information of each point corresponding to the image feature; based on the ground truth label and the depth information of each point, determine the depth information loss value of each point corresponding to the image feature.

[0115] Furthermore, under the action of the image feature extraction module in the BEV detection architecture, before the step of extracting image features from training sample data, the device is also used to use a large-scale 2D data set to pre-train the image feature extraction module of the BEV detection architecture and initialize the target parameters corresponding to the image feature extraction module; wherein the target parameters are associated with the image features.

[0116] Furthermore, the output module 203 is also specifically used to convert the image features into BEV space through the view conversion module, extract BEV space features through the BEV space feature extraction module, and then obtain target information through the 3D target detection head; output the second loss value of the image feature according to the target information and the ground truth label.

[0117] Furthermore, the second determination module 204 is also specifically used to assign preset weight coefficients to the second loss value, and the three-dimensional information loss value and the depth information loss value in the first loss value, respectively, and then add them up to construct a loss function of the image feature extraction module; when the first loss value and the second loss value are back-propagated along the BEV detection architecture, the loss function is input to obtain a third loss value of the image feature.

[0118] Furthermore, the extraction module 201 is also specifically used to input the training sample data into the image feature extraction module of the BEV detection architecture, and output the image features after being processed by the image feature extraction module.

[0119] Figure 4 Schematic diagram of the hardware architecture of the electronic device 300 provided in an embodiment of the present invention. Figure 4 As shown, the electronic device 300 includes: a machine-readable storage medium 301 and a processor 302, and may also include a non-volatile storage medium 303, a communication interface 304 and a bus 305; wherein the machine-readable storage medium 301, the processor 302, the non-volatile storage medium 303 and the communication interface 304 communicate with each other through the bus 305. The processor 302 can execute the image feature extraction method in the BEV detection architecture described in the above embodiment by reading and executing the machine executable instructions for image feature extraction in the BEV detection architecture in the machine-readable storage medium 301.

[0120] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0121] The non-volatile medium may be a non-volatile memory, a flash memory, a storage drive (such as a hard drive), any type of storage disk (such as a CD, DVD, etc.), or a similar non-volatile storage medium, or a combination thereof.

[0122] It can be understood that the specific operation methods of each functional module in this embodiment can refer to the detailed description of the corresponding steps in the above method embodiment, and will not be repeated here.

[0123] The computer-readable storage medium provided in the embodiment of the present invention stores a computer program. When the computer program code is executed, the image feature extraction method in the BEV detection architecture described in any of the above embodiments can be implemented. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0124] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0125] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0126] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0127] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the aforementioned embodiments, those of ordinary skill in the art should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed by the present invention, or can easily conceive of changes, or make equivalent replacements for some of the technical features therein. Such modifications, changes or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the protection scope of the present invention.

Claims

1. A training method for an image feature extraction module in a BEV detection architecture, characterized in that: include: Extracting image features from training sample data under the action of an image feature extraction module in the BEV detection architecture; wherein the training sample data includes image data and corresponding ground truth labels of the image data; Input the image feature into a supervision information detection head, and compare the supervision information corresponding to the image feature output by the supervision information detection head with the ground truth label to determine a first loss value of the image feature; wherein the supervision information includes three-dimensional information and depth information; Forward propagating the image feature along the BEV detection architecture, and outputting a second loss value of the image feature; constructing a loss function of the image feature extraction module based on the first loss value and the second loss value, and determining a third loss value of the image feature; The target parameters in the image feature extraction module are updated according to the third loss value, and the updated image feature extraction module is used as a new image feature extraction module to again execute the step of extracting image features from the sample data under the action of the image feature extraction module, until the second loss value and the third loss value output by the image feature extraction module under the action of the training sample data converge.

2. The method according to claim 1, characterized in that The step of inputting the image feature into a supervision information detection head, and comparing the supervision information corresponding to the image feature output by the supervision information detection head with the ground truth label to determine a first loss value of the image feature includes: Inputting the image feature into a three-dimensional information detection head, so that the three-dimensional information detection head outputs the number of objects in the image feature and the three-dimensional information of each object; Based on the ground truth label and the three-dimensional information of each of the objects, a three-dimensional information loss value of each object is determined.

3. The method according to claim 2, characterized in that The step of inputting the image feature into a supervision information detection head, and comparing the supervision information corresponding to the image feature output by the supervision information detection head with the ground truth label to determine a first loss value of the image feature also includes: Inputting the image feature into a depth information detection head so that the depth information detection head outputs depth information of each point corresponding to the image feature; Based on the ground truth label and the depth information of each point, a depth information loss value of each point corresponding to the image feature is determined.

4. The method according to claim 1, characterized in that Before the step of extracting image features from training sample data under the action of the image feature extraction module in the BEV detection architecture, the method further includes: A large-scale 2D data set is used to pre-train an image feature extraction module of a BEV detection architecture, and a target parameter corresponding to the image feature extraction module is initialized; wherein the target parameter is associated with the image feature.

5. The method according to claim 1, characterized in that: The BEV detection architecture includes the image feature extraction module, the view conversion module, the BEV space feature extraction module and the 3D object detection head connected in sequence; the step of forward propagating the image feature along the BEV detection architecture and outputting the second loss value of the image feature includes: The image features are converted into BEV space by the view conversion module, the BEV space features are extracted by the BEV space feature extraction module, and the target information is obtained by the 3D target detection head; Outputting a second loss value of the image feature based on the target information and the ground truth label.

6. The method according to claim 3, characterized in that The step of constructing a loss function of the image feature extraction module based on the first loss value and the second loss value, and determining a third loss value of the image feature comprises: Assigning preset weight coefficients to the second loss value, and the three-dimensional information loss value and the depth information loss value in the first loss value respectively, and then adding them up to construct a loss function of the image feature extraction module; When the first loss value and the second loss value are back-propagated along the BEV detection architecture, a loss function is input to obtain a third loss value of the image feature.

7. The method according to claim 1, characterized in that Under the action of the image feature extraction module in the BEV detection architecture, the step of extracting image features from the training sample data includes: The training sample data is input into the image feature extraction module of the BEV detection architecture, and the image features are output after being processed by the image feature extraction module.

8. A training device for an image feature extraction module in a BEV detection architecture, characterized in that: include: An extraction module, under the action of the image feature extraction module in the BEV detection architecture, extracts image features from training sample data; wherein the training sample data includes image data and corresponding ground truth labels of the image data; A first determination module inputs the image feature into a supervision information detection head, and compares the supervision information corresponding to the image feature output by the supervision information detection head with the ground truth label to determine a first loss value of the image feature; wherein the supervision information includes three-dimensional information and depth information; An output module forward-propagates the image feature along the BEV detection architecture and outputs a second loss value of the image feature; a second determination module, which constructs a loss function of the image feature extraction module based on the first loss value and the second loss value, and determines a third loss value of the image feature; An updating module updates the target parameters in the image feature extraction module according to the third loss value, and uses the updated image feature extraction module as a new image feature extraction module to again execute the step of extracting image features from the sample data under the action of the image feature extraction module, until the second loss value and the third loss value output by the image feature extraction module under the action of the training sample data converge.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the steps of the method described in any one of claims 1 to 7.

Images