BEV perception model training method supporting multiple layouts of camera and related device

Abstract

The invention discloses a BEV perception model training method supporting multiple layouts of cameras and a related device. The method comprises the following steps: acquiring a plurality of pieces of sensing key data in different camera layouts, and respectively executing the following steps for each piece of sensing key data: extracting image features based on image data of n visual angles at the current moment, and constructing voxel query parameters based on a current camera layout scene; on the basis of an internal parameter matrix and an external parameter matrix in the current camera layout, camera parameters are coded into image features and voxel query parameters respectively; and according to the image features fusing the camera parameters and the voxel query parameters, carrying out space conversion based on a voxel query mechanism to obtain BEV space time sequence features, taking the determined multiple BEV space time sequence features and corresponding labels as training samples, optimizing parameters of the initialization model by using a gradient descent algorithm, and training to obtain a BEV perception model. According to the method, the data labeling cost can be effectively reduced, and the generalization ability and the detection precision of the perception model are improved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, specifically to the technical fields of BEV environmental perception and joint training of multi-source heterogeneous data, and particularly to a BEV perception model training method and related device that supports multiple camera layouts. Background Technology

[0002] Perception is a core component of autonomous driving systems. It uses sensor data to model the surrounding environment and outputs key information such as the location, size, and speed of obstacles, as well as the location and type of lane lines. The algorithms of the perception module are mainly based on AI neural networks. Data, algorithms, and computing power are the three core elements of neural networks, with data playing a central role in the development of the perception module.

[0003] In industrial production environments, data annotation requires investments ranging from millions to billions of dollars, making high-quality data resources extremely valuable. Currently, sensor layouts vary significantly across different vehicle models, while the mainstream Bird's Eye View (BEV) architecture, as a data-driven architecture, boasts strong scalability and extensibility, making it the mainstream perception solution in the autonomous driving industry. Its workflow involves: extracting features from multi-view images; transforming image features into a unified BEV space using camera intrinsic and extrinsic parameters; fusing these features with historical temporal features; and then completing various perception tasks through a multi-task head.

[0004] However, existing BEV architectures have a key flaw: they assume that the training data is completely identical to the sensor layout deployed in the vehicle, meaning that the camera extrinsic parameters remain fixed. This prevents the joint utilization of a large amount of high-quality BEV data with different camera layouts, limiting further improvements in model performance and resulting in a serious waste of data resources. Summary of the Invention

[0005] This application provides a training method and related apparatus for BEV perception models that support multiple camera layouts, in order to solve the problems of low data utilization and insufficient model generalization ability caused by the existing BEV perception architecture relying on fixed camera layouts and being unable to train datasets with different sensor layouts.

[0006] The technical solution is as follows: Firstly, a method for training a BEV perception model that supports multiple camera layouts is provided, including: Acquire multiple key perception data in different camera layouts. Each key perception data includes image data from n viewpoints at the current moment in its corresponding camera layout, as well as the intrinsic and extrinsic parameter matrices of the corresponding camera. The number of camera viewpoints or viewpoints varies in each camera layout. For each key perception data point, the following steps are performed: Image features are extracted from image data from n viewpoints at the current time, and voxel query parameters are constructed based on the current camera layout scene; the intrinsic and extrinsic parameter matrices in the current camera layout are concatenated and converted into camera parameter encodings of the same dimension as the image features; based on the camera parameter encoding, the intrinsic and extrinsic parameter matrices are encoded into the image features to obtain image features fused with camera parameters; and based on the camera parameter encoding, the intrinsic and extrinsic parameter matrices are encoded into the voxel query parameters to obtain voxel query parameters fused with camera parameters; based on the image features and voxel query parameters of the fused camera parameters, spatial transformation is performed using a voxel query mechanism to obtain vehicle-related BEV spatial temporal features, where n is a positive integer. The BEV perception model is trained by using multiple determined BEV spatial temporal features and labels that are annotated for each key perception data based on the selected perception task as training samples, and optimizing the parameters of the initialization model using the gradient descent algorithm.

[0007] In one possible implementation, the extrinsic parameter matrix is ​​{M}. k} 1≤k≤n M∈R 3×4 The intrinsic parameter matrix is ​​{K} k} 1≤k≤n , K∈R 3×3 ; The intrinsic and extrinsic parameter matrices in the current camera layout are then concatenated and converted into camera parameter encodings of the same dimension as the image features. Specifically, this includes: The extrinsic parameter matrix M∈R is respectively... 3×4 and the intrinsic parameter matrix K∈R 3×3 Straighten it into a one-dimensional matrix according to the set arrangement rules; The straightened one-dimensional extrinsic and intrinsic parameter matrices are concatenated and converted into camera parameter codes ξ∈R. 21×H×W The dimensions of the camera parameter encoding are the same as the dimensions of the image features; H and W are the height and width of the image features, respectively.

[0008] In one possible implementation, the intrinsic and extrinsic parameter matrices are encoded into the image features based on the camera encoding parameters to obtain image features fused with camera parameters, specifically including: Based on the height and width of the image features, an image location code P is generated. c ∈R 2×H×W , where P c (·,u,v)= [u / W,v / H] T The u and v are the pixel coordinates of the image features; The camera parameter encoding and the image location encoding are fused through a fully connected layer, and the fused first vector is fused with the image features to obtain image features with fused camera parameters: in, This represents a concatenation operation of vectors, where F is an image feature, and... This represents a fully connected layer, used to map the fused first vector to the same dimension as the image features; Furthermore, based on the camera encoding parameters, the intrinsic and extrinsic parameter matrices are encoded into the voxel query parameters to obtain the voxel query parameters for fused camera parameters, specifically including: Based on the voxel query three-dimensional dimensions, a voxel position code P is generated. v ∈R 3×Z×Y×X , where P v (·,d,h,w)= [d / Z, h / Y, w / X] T d, h, and w are the pixel coordinates of the voxel; The camera parameter encoding and the voxel position encoding are fused through a fully connected layer, and the fused second vector is fused with the voxel query to obtain the voxel query parameters of the fused camera parameters: Among them, the For image features, the This represents a fully connected layer used to map the fused second vector to the same dimension as the image features.

[0009] In one possible implementation, based on the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters, a spatial transformation is performed using a voxel query mechanism to obtain vehicle-related BEV spatial-temporal features, specifically including: Based on the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters, the image features of the fused camera parameters at the current moment are converted into the vehicle-related BEV spatial features at the current moment based on the voxel query mechanism. The BEV spatial features at the current moment are fused with the BEV spatial features at multiple previous moments to obtain the vehicle-related BEV spatial temporal features.

[0010] In one possible implementation, based on the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters, the image features of the fused camera parameters at the current moment are transformed into vehicle-related BEV spatial features at the current moment based on a voxel query mechanism, specifically including: Calculate the projected coordinates on the images from each viewpoint based on the three-dimensional coordinates of each voxel; Based on the deformation attention mechanism, the viewpoint image features corresponding to the projection coordinates of the image features of the fused camera parameters are extracted from the image features of the fused camera parameters using the query parameters in the voxel query parameters of the fused camera parameters. The BEV spatial features extracted from images from different viewpoints are aggregated to obtain the BEV spatial features at the current time.

[0011] Secondly, a BEV perception prediction method supporting multiple camera layouts is provided, including: Acquire multiple key perception data to be predicted and convert them into BEV spatial features; The BEV spatial features are input into the BEV perception model trained based on the method described in the first aspect, and the perception results are output. The camera layout of the multiple key perception data to be predicted is different from the camera layout during the training of the BEV perception model.

[0012] Thirdly, a BEV perception model training device supporting multiple camera layouts is provided, including: The acquisition module is used to acquire multiple key perception data in different camera layouts. Each key perception data includes image data from n perspectives in its corresponding camera layout at the current moment, as well as the intrinsic and extrinsic parameter matrices of the corresponding camera. The number of camera perspectives or perspectives varies in each camera layout. The conversion module is used to perform the following operations for each key perception data: extracting image features based on image data from n perspectives at the current moment, and constructing voxel query parameters based on the current camera layout scene; concatenating the intrinsic and extrinsic parameter matrices in the current camera layout to convert them into camera encoding parameters of the same dimension as the image features; encoding the intrinsic and extrinsic parameter matrices into the image features based on the camera encoding parameters to obtain image features of fused camera parameters; and encoding the intrinsic and extrinsic parameter matrices into the voxel query parameters based on the camera encoding parameters to obtain voxel query parameters of fused camera parameters; and performing spatial conversion based on the image features of fused camera parameters and voxel query parameters of fused camera parameters to obtain vehicle-related BEV spatial features based on the voxel query mechanism. The training module is used to take multiple determined BEV spatial features and labels that are annotated for each key perception data based on the selected perception task as training samples, and use the gradient descent algorithm to optimize the parameters of the initial model to train the BEV perception model.

[0013] Fourthly, a BEV perception and prediction device supporting multiple camera layouts is provided, comprising: The extraction module is used to acquire multiple key perception data to be predicted and extract and convert them into BEV spatial features; The prediction module is used to input the BEV spatial features into the BEV perception model trained based on the method described in the first aspect, and output the perception result. The camera layout of the multiple key perception data to be predicted is different from the camera layout during the training of the BEV perception model.

[0014] Fifthly, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described above and any possible implementations.

[0015] In a sixth aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the aspects and any possible implementations described above.

[0016] The beneficial effects of the technical solution provided in this application include at least the following: As can be seen from the above technical solution, the embodiments of this application acquire multiple key perception data from different camera layouts, and perform the following for each key perception data: extract image features based on image data from n perspectives at the current time, and construct voxel query parameters based on the current camera layout scene; based on the intrinsic and extrinsic parameter matrices in the current camera layout, encode camera parameters into image features and voxel query parameters respectively; based on the image features fused with camera parameters and the voxel query parameters fused with camera parameters, perform spatial transformation based on the voxel query mechanism to obtain vehicle-related BEV spatial temporal features; use the determined multiple BEV spatial temporal features and the labels annotated for each key perception data based on the selected perception task as training samples, use the gradient descent algorithm to optimize the parameters of the initialization model, and train to obtain the BEV perception model, realizing unified training of multiple camera layout data. This application can be seamlessly integrated into mainstream BEV solutions, effectively reducing data annotation costs and improving the generalization ability and detection accuracy of the perception model.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating the steps of the BEV perception model training method supporting multiple camera layouts provided in this application embodiment. Figure 2 This is a schematic diagram illustrating the steps of the BEV perception prediction method supporting multiple camera layouts provided in this application embodiment.

[0020] Figure 3 This is a schematic flowchart of a BEV perception prediction method supporting multiple camera layouts provided in one embodiment of this application.

[0021] Figure 4 This is a structural block diagram of a BEV perception model training device supporting multiple camera layouts provided in another embodiment of this application.

[0022] Figure 5 This is a structural block diagram of a BEV perception and prediction device supporting multiple camera layouts provided in another embodiment of this application.

[0023] Figure 6 This is a block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0026] It should be noted that the terminal devices involved in the embodiments of this application may include, but are not limited to, smart devices such as mobile phones, personal digital assistants (PDAs), wireless handheld devices, and tablet computers; the display devices may include, but are not limited to, personal computers, televisions, and other devices with display functions.

[0027] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0028] Autonomous driving systems typically comprise five core modules: perception, localization, prediction, planning, and control. Among these, the perception module is responsible for accurately modeling the vehicle's surrounding environment and serves as the "eyes" of the entire system. The perception module takes various sensor data (such as camera images, LiDAR point clouds, and millimeter-wave radar signals) as input and outputs information such as the position, size, orientation, and speed of obstacles (vehicles, pedestrians, cyclists, etc.), the position, type, and curvature of lane lines, as well as environmental information such as drivable areas and traffic light status.

[0029] In the technological evolution of perception modules, deep learning-based neural networks have become the mainstream solution. For neural networks, data, algorithms, and computing power are the three core elements: data determines the learning ceiling of the model, algorithms determine the learning efficiency of the model, and computing power provides support for large-scale data processing and model training. Among these, the importance of data is particularly prominent. In industrial practice, training a high-performance perception model often requires an investment of millions to billions of dollars in data collection, cleaning, and labeling.

[0030] With the rapid development of autonomous driving technology, environmental perception, as the foundation of vehicle decision-making and control, directly determines the safety of autonomous driving systems through its accuracy and robustness. Current mainstream perception solutions have upgraded from traditional post-fusion architectures to BEV architectures. This architecture achieves efficient data fusion and unified task modeling by transforming multi-sensor data into a unified bird's-eye view space within a vehicle coordinate system. However, existing BEV architectures are highly dependent on sensor layout, resulting in the ineffective utilization of massive amounts of data from different vehicle models and sensor configurations, severely restricting model performance improvement. This application addresses this technical challenge by proposing a cross-sensor layout camera parameter encoding scheme, breaking through the limitations of sensor layout and providing a feasible path for joint training of multi-source heterogeneous data.

[0031] The main inventive concept of the cross-sensor layout camera encoding scheme provided in this application is as follows: Multiple key perception data points from different camera layouts are acquired, and for each key perception data point, the following steps are performed: image features are extracted from image data from n viewpoints at the current time, and voxel query parameters are constructed based on the current camera layout scene; based on the intrinsic and extrinsic parameter matrices in the current camera layout, camera parameters are encoded into the image features and voxel query parameters respectively; based on the image features fused with the camera parameters and the voxel query parameters fused with the camera parameters, spatial transformation is performed using a voxel query mechanism to obtain vehicle-related BEV spatial-temporal features; the determined multiple BEV spatial-temporal features and the labels annotated for each key perception data point based on the selected perception task are used as training samples; the gradient descent algorithm is used to optimize the parameters of the initialization model, and a BEV perception model is trained, achieving unified training of data from multiple camera layouts. This application can be seamlessly integrated into mainstream BEV solutions, effectively reducing data annotation costs and improving the generalization ability and detection accuracy of the perception model.

[0032] It should be understood that this method can be widely applied to core environmental perception tasks of autonomous vehicles, such as 3D object detection, lane line recognition, free space segmentation, and occupancy grid prediction, and is suitable for the research and development and deployment of L2 to L4 level autonomous driving systems.

[0033] Reference Figure 1 The diagram illustrates the steps of a BEV perception model training method supporting multiple camera layouts, as provided in this embodiment. The method is executed by a BEV perception model training device, which can be a computer hardware device and software module with computing, processing, and storage capabilities. For example, it can be an electronic device such as a computer, tablet computer, smartphone, or smart wearable device, or a software module or component integrated or installed in an electronic device.

[0034] The BEV perception model training method that supports multiple camera layouts may specifically include the following steps: Step 102: Obtain multiple key perception data in different camera layouts. Each key perception data includes image data from n perspectives in its corresponding camera layout at the current moment, as well as the intrinsic and extrinsic parameter matrices of the corresponding camera. The number of camera perspectives or perspectives may vary in different camera layouts.

[0035] In this application, the data required for training the perception model comes from different camera layouts. This can have two meanings: First, the multiple key perception data come from different camera layout scenarios. For example, some key perception data comes from scenario 1, while other key perception data comes from scenarios 2 and 3 respectively. Second, the multiple key perception data come from different camera layout scenarios. For example, scenario 1 has 5 cameras, scenario 2 has 3 cameras, and scenario 3 has 4 cameras. Moreover, the layout positions (i.e., camera viewpoints) of each camera differ in scenarios 1, 2, and 3.

[0036] Step 104: For each key perception data point, perform the following steps: extract image features from image data of n perspectives at the current time, and construct voxel query parameters based on the current camera layout scene; connect the intrinsic and extrinsic parameter matrices in the current camera layout to convert them into camera parameter encodings of the same dimension as the image features; based on the camera parameter encoding, encode the intrinsic and extrinsic parameter matrices into the image features to obtain image features fused with camera parameters; and based on the camera parameter encoding, encode the intrinsic and extrinsic parameter matrices into the voxel query parameters to obtain voxel query parameters fused with camera parameters; based on the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters, perform spatial transformation based on the voxel query mechanism to obtain vehicle-related BEV spatial temporal features, where n is a positive integer.

[0037] Optionally, the extrinsic parameter matrix is ​​{M} k} 1≤k≤n M∈R 3×4 The intrinsic parameter matrix is ​​{K} k} 1≤k≤n , K∈R 3×3 When converting the intrinsic and extrinsic parameter matrices in the current camera layout into camera parameter encodings of the same dimension as the image features, specifically, the extrinsic parameter matrix M∈R can be converted into... 3×4 and the intrinsic parameter matrix K∈R 3×3 Straighten the matrix into a one-dimensional matrix according to the set arrangement rules; then concatenate the straightened one-dimensional extrinsic and intrinsic parameter matrices to convert them into camera parameter codes ξ∈R. 21×H×W The dimensions of the camera parameter encoding are the same as the dimensions of the image features; H and W are the height and width of the image features, respectively.

[0038] Furthermore, when encoding the intrinsic and extrinsic parameter matrices into the image features based on the camera encoding parameters to obtain image features fused with camera parameters, specifically, image position codes P can be generated based on the height and width of the image features. c ∈R 2×H×W , where Pc (·,u,v) = [u / W,v / H] T The u and v are the pixel coordinates of the image features; the camera parameter encoding and the image position encoding are fused through a fully connected layer, and the fused first vector is fused with the image features to obtain the image features with fused camera parameters: in, This represents a concatenation operation of vectors, where F is an image feature, and... This represents a fully connected layer, used to map the fused first vector to the same dimension as the image features; Simultaneously, when encoding the intrinsic and extrinsic parameter matrices into the voxel query parameters based on the camera encoding parameters to obtain the voxel query parameters fused with camera parameters, specifically, voxel position codes P can be generated based on the three-dimensional dimensions of the voxel query. v ∈R 3×Z×Y×X , where P v (·,d,h,w) = [d / Z, h / Y, w / X] T The d, h, and w are the pixel coordinates of the voxel; the camera parameter encoding and the voxel position encoding are fused through a fully connected layer, and the fused second vector is fused with the voxel query to obtain the voxel query parameters of the fused camera parameters: Among them, the For voxel lookup, the This represents a fully connected layer used to map the fused second vector to the same dimension as the image features.

[0039] Optionally, when obtaining vehicle-related BEV spatial temporal features by spatial transformation based on the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters, specifically, the image features of the fused camera parameters at the current moment can be transformed into vehicle-related BEV spatial features at the current moment based on the voxel query mechanism, according to the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters; the BEV spatial features at the current moment are then temporally fused with the BEV spatial features at multiple previous moments to obtain vehicle-related BEV spatial temporal features.

[0040] In order to incorporate image features {F k} 1≤k≤n Transforming to a unified BEV space related to the vehicle body typically involves discretizing the space near the vehicle body into a Z×Y×X voxel grid and assigning it a set of query parameters Q∈R. C×Z×Y×XWhere X, Y, and Z represent the size of the discretized 3D space near the vehicle body, and C is the dimension of the feature of each voxel. Therefore, when converting the image features of the fused camera parameters at the current moment into the vehicle-related BEV space features at the current moment based on the voxel query mechanism, according to the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters, specifically, the projection coordinates on each viewpoint image can be calculated based on the three-dimensional coordinates of each voxel; based on the deformation attention mechanism, the viewpoint image features corresponding to the projection coordinates are extracted from the image features of the fused camera parameters using the query parameters in the voxel query parameters of the fused camera parameters; the viewpoint image features extracted from each viewpoint image are aggregated to obtain the BEV space features at the current moment.

[0041] Step 106: Using the determined multiple BEV spatial temporal features and the labels annotated for each key perception data based on the selected perception task as training samples, the gradient descent algorithm is used to optimize the parameters of the initialization model, and the BEV perception model is trained.

[0042] The above technical solution breaks through the dependence of existing BEV architecture on fixed camera layout, and realizes joint training of data from different sensor layouts; it makes full use of massive high-quality heterogeneous data, reduces data annotation costs, and significantly improves the model's generalization ability and perception accuracy; it can be seamlessly integrated into mainstream BEV solutions such as BEVFormer, with strong compatibility and no need to reconstruct the existing model architecture; it is applicable to a variety of core perception tasks such as 3D object detection and lane line recognition, and has a wide range of application scenarios.

[0043] Reference Figure 2 The diagram shown illustrates the steps of a BEV perception prediction method supporting multiple camera layouts provided in this application embodiment. The execution entity of this perception prediction method can be the same as or different from the execution entity of the perception model training method; that is, the execution entity of the perception prediction method can also be a computing hardware device or software module with computing processing, storage, and other functions.

[0044] like Figure 2 As shown, the BEV perception prediction method supporting multiple camera layouts may specifically include the following steps: Step 202: Obtain multiple key perception data to be predicted and convert them into BEV spatial features.

[0045] The specific implementation of step 202 can be found in the details of the data processing and conversion process in steps 102 and 104, which will not be elaborated here.

[0046] It should be understood that the sources of the multiple key perception data to be predicted obtained here, namely the multi-camera layout scene, can be the same as or different from the scene from which the data samples come during model training.

[0047] Step 204: Input the BEV spatial features into the BEV perception model trained based on the method described in steps 102-106, and output the perception results; wherein the camera layout of the multiple key perception data to be predicted is different from the camera layout during the training of the BEV perception model.

[0048] This prediction scheme enables the perception and prediction of data from various camera layouts, improving the accuracy and generalization ability of prediction and perception.

[0049] Reference Figure 3 The diagram shown is a flowchart of a BEV perception prediction method supporting multiple camera layouts provided in an embodiment of this application.

[0050] The perception prediction process mainly includes: the training process of the perception model and the perception prediction process of the data to be processed. Training process: Data input: multi-view images Camera internal parameters Camera external parameters .

[0051] Image feature extraction: Obtaining image features through the backbone. .

[0052] Camera parameter encoding: Generates camera encoded parameters that include camera intrinsic and extrinsic parameters. ; Intrinsic parameter matrix It is a 3×3 matrix containing parameters such as focal length and principal point coordinates, which, when flattened by rows, becomes a 9-dimensional vector; the extrinsic parameter matrix... It is a 3×4 matrix, including the rotation matrix. Translation vector The vector is straightened out to a 12-dimensional vector; the intrinsic and extrinsic parameter vectors are concatenated to obtain a 21-dimensional camera parameter encoding; this 21-dimensional vector is then broadcast to the same spatial dimension (H×W) as the image features, resulting in... , where H and W are the height and width of the image feature.

[0053] Camera encoding of image features: To enable image features to carry camera parameter information, the following fusion method is used: Generate image location encoding ,in , , These are the pixel coordinates of image features; this encoding is used to identify the relative position of pixels in the image plane; concatenated camera parameter encoding. Image location encoding This yields a 23-dimensional (21+2) fusion vector; through a fully connected layer... : → Map the 23-dimensional vector to the same dimension as the image features. ,in It includes two layers of linear transformation and the ReLU activation function; it uses the Hadmar product to... Output and Image Features By fusing, image features with fused camera parameters are obtained: .

[0054] Camera encoding for voxel queries: To enable voxel queries to adapt to different camera layouts, a similar fusion method is used: Generate voxel position encoding ,in , , , Using voxel coordinates, this encoding is used to identify the relative position of the voxel in BEV space; the generated camera parameter encoding is also used. and broadcast it to the same spatial dimension as the voxel query. ; series and This yields a 24-dimensional (21+3) fusion vector; through a fully connected layer... : → Mapping a 24-dimensional vector to a dimension ,in Structure and Consistent; using the Hadema product to... Output and voxel query Fusion, obtaining voxel lookups of fused camera parameters: .

[0055] BEV Space Transformation: The encoded features are transformed to the BEV space to obtain BEV space. t .

[0056] Temporal fusion: B t Compared with historical BEV characteristics {B t-1 B t-2 ,···} are fused in time sequence to obtain B.

[0057] Model training: Construct a training dataset containing various camera layouts, with each sample containing images. Internal Reference External reference and label the model (e.g., 3D bounding box, lane line coordinates, etc.); initialize model parameters, including the main network, , The parameters of modules such as attention mechanism and task head are defined; for each training sample, the model output is calculated through the above steps; the loss function is defined, including regression loss and classification loss for object detection, segmentation loss for lane line recognition, etc.; the parameters are optimized using stochastic gradient descent (SGD) or Adam algorithm, and the loss function is minimized through backpropagation to train the prediction model.

[0058] Prediction process: Acquire the sensor data to be processed.

[0059] Processing of sensory data. The processing method is similar to that used during model training, and will not be elaborated here.

[0060] Use predictive models to make predictions.

[0061] Output the prediction results.

[0062] This application overcomes sensor layout limitations through the aforementioned technical solutions: by explicitly encoding camera intrinsic and extrinsic parameters, the model can automatically adapt to different camera layouts, achieving joint training of multi-layout datasets for the first time; improves data utilization: massive amounts of data from different vehicle models and configurations can be incorporated into training without repeated labeling, significantly reducing data costs (calculated to reduce labeling costs by 30%-50%); enhances model generalization ability: when deployed on new vehicle models, there is no need to retrain the model; only new camera parameters need to be input to maintain high accuracy (actual measurements show that cross-vehicle deployment accuracy decreases by no more than 5%); has strong compatibility: it can be seamlessly integrated into mainstream BEV architectures such as BEVFormer and PETR without reconstructing existing models, facilitating rapid industrial deployment; and comprehensively improves performance: with the same amount of data, the model trained using the method of this invention can improve 3D object detection AP (Average Precision) by 8%-15% and lane line recognition F1 score by 5%-10%.

[0063] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0064] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0065] Figure 4This paper illustrates a structural block diagram of a BEV perception model training device supporting multiple camera layouts according to an embodiment of this application, as shown below. Figure 4 As shown. The BEV perception model training device 400 supporting multiple camera layouts in this embodiment may include an acquisition module 401, a conversion module 402, and a training module 403. The acquisition module 401 is used to acquire multiple perception key data in different camera layouts. Each perception key data includes image data from n viewpoints at the current time in its corresponding camera layout, as well as the intrinsic and extrinsic parameter matrices of the corresponding camera; the number or number of camera viewpoints differs in each camera layout. The conversion module 402 is used to perform the following operations for each key perception data point: extracting image features based on image data from n viewpoints at the current time, and constructing voxel query parameters based on the current camera layout scene; concatenating the intrinsic and extrinsic parameter matrices in the current camera layout to convert them into camera encoding parameters of the same dimension as the image features; encoding the intrinsic and extrinsic parameter matrices into the image features based on the camera encoding parameters to obtain image features of fused camera parameters; and encoding the intrinsic and extrinsic parameter matrices into the voxel query parameters based on the camera encoding parameters to obtain voxel query parameters of fused camera parameters; and performing spatial transformation based on the image features and voxel query parameters of fused camera parameters to obtain vehicle-related BEV spatial features. The training module 403 is used to use multiple determined BEV spatial features and labels annotated for each key perception data point based on the selected perception task as training samples, and to optimize the parameters of the initialization model using the gradient descent algorithm to train the BEV perception model.

[0066] It should be noted that some or all of the device supporting BEV perception model training with multiple camera layouts in this embodiment may be an application located on a local terminal, or it may be a plugin or software development kit (SDK) or other functional unit set in an application located on a local terminal, or it may be a processing engine located on a network-side server, or it may be a distributed system located on the network side, such as a processing engine or distributed system in a network-side autonomous driving platform, etc. This embodiment does not impose any particular limitations on this.

[0067] It is understood that the application can be a native program installed on the local terminal, or it can be a web application of a browser on the local terminal. This embodiment does not limit this.

[0068] Optionally, in one possible implementation of this embodiment, the extrinsic parameter matrix is ​​{M}. k} 1≤k≤n M∈R3 ×4 The intrinsic parameter matrix is ​​{K} k} 1≤k≤n , K∈R 3×3 The conversion module 402, when converting the concatenation of the intrinsic and extrinsic parameter matrices in the current camera layout into camera parameter encodings of the same dimension as the image features, can specifically be used to convert the extrinsic parameter matrix M∈R... 3×4 and the intrinsic parameter matrix K∈R 3×3 Straighten the matrix into a one-dimensional matrix according to the set arrangement rules; then concatenate the straightened one-dimensional extrinsic and intrinsic parameter matrices to convert them into camera parameter codes ξ∈R. 21×H×W The dimensions of the camera parameter encoding are the same as the dimensions of the image features; H and W are the height and width of the image features, respectively.

[0069] Optionally, in one possible implementation of this embodiment, when the conversion module 402 encodes the intrinsic and extrinsic parameter matrices into the image features based on the camera encoding parameters to obtain image features fused with camera parameters, it is specifically used to generate image position codes P based on the height and width of the image features. c ∈R 2×H×W , where P c (·,u,v) = [u / W,v / H] T The u and v are the pixel coordinates of the image features; the camera parameter encoding and the image position encoding are fused through a fully connected layer, and the fused first vector is fused with the image features to obtain the image features with fused camera parameters: in, This represents a concatenation operation of vectors, where F is an image feature, and... This represents a fully connected layer, used to map the fused first vector to the same dimension as the image features; Furthermore, when encoding the intrinsic and extrinsic parameter matrices into the voxel query parameters based on the camera encoding parameters to obtain the voxel query parameters fused with camera parameters, this is specifically used to generate voxel position codes P based on the voxel query 3D dimensions. v ∈R 3×Z×Y×X , where P v (·,d,h,w) = [d / Z, h / Y, w / X] T The d, h, and w are the pixel coordinates of the voxel; the camera parameter encoding and the voxel position encoding are fused through a fully connected layer, and the fused second vector is fused with the voxel query to obtain the voxel query parameters of the fused camera parameters: Among them, the For image features, the This represents a fully connected layer used to map the fused second vector to the same dimension as the image features.

[0070] Optionally, in one possible implementation of this embodiment, when the conversion module 402 performs spatial transformation based on the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters to obtain vehicle-related BEV spatial temporal features using a voxel query mechanism, it specifically converts the image features of the fused camera parameters at the current moment to vehicle-related BEV spatial features at the current moment based on the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters; and performs temporal fusion of the current BEV spatial features with the BEV spatial features from multiple previous moments to obtain vehicle-related BEV spatial temporal features.

[0071] Optionally, in one possible implementation of this embodiment, when the conversion module 402 converts the image features of the fused camera parameters at the current moment to the vehicle-related BEV spatial features at the current moment based on the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters using a voxel query mechanism, it specifically calculates the projection coordinates on each viewpoint image based on the three-dimensional coordinates of each voxel; extracts the viewpoint image features corresponding to the projection coordinates from the image features of the fused camera parameters using the query parameters in the voxel query parameters of the fused camera parameters based on the deformation attention mechanism; and aggregates the viewpoint image features extracted from each viewpoint image to obtain the BEV spatial features at the current moment.

[0072] Reference Figure 5 The diagram shown is a structural schematic of a BEV perception and prediction device supporting multiple camera layouts provided in an embodiment of this application, comprising: Extraction module 501 is used to acquire multiple key perception data to be predicted and extract and convert them into BEV spatial features; prediction module 502 is used to input the BEV spatial features into the BEV perception model trained based on the method described in the first aspect and output the perception result; wherein, the camera layout of the multiple key perception data to be predicted is different from the camera layout during the training of the BEV perception model.

[0073] In this embodiment, multiple key perception data points from different camera layouts can be acquired. For each key perception data point, the following steps are performed: image features are extracted from image data from n viewpoints at the current time, and voxel query parameters are constructed based on the current camera layout scene; based on the intrinsic and extrinsic parameter matrices in the current camera layout, camera parameters are encoded into the image features and voxel query parameters respectively; based on the image features fused with the camera parameters and the voxel query parameters fused with the camera parameters, spatial transformation is performed using a voxel query mechanism to obtain vehicle-related BEV spatial-temporal features; the determined multiple BEV spatial-temporal features and the labels annotated for each key perception data point based on the selected perception task are used as training samples; the gradient descent algorithm is used to optimize the parameters of the initialization model, and a BEV perception model is trained, achieving unified training of multiple camera layout data. This application can be seamlessly integrated into mainstream BEV solutions, effectively reducing data annotation costs and improving the generalization ability and detection accuracy of the perception model.

[0074] One embodiment of this application provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the BEV perception model training method supporting multiple camera layouts and the BEV perception prediction method supporting multiple camera layouts as described above.

[0075] One embodiment of this application provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, which is loaded and executed by the processor to implement the BEV perception model training method supporting multiple camera layouts and the BEV perception prediction method supporting multiple camera layouts as described above.

[0076] One embodiment of this application provides an autonomous driving vehicle, including the electronic devices described above. Specifically, the autonomous driving vehicle can be a Level 2 or higher vehicle.

[0077] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0078] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0079] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0080] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0081] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the BEV perception model training method supporting multiple camera layouts and the BEV perception prediction method supporting multiple camera layouts. For example, in some embodiments, the BEV perception model training method supporting multiple camera layouts and the BEV perception prediction method supporting multiple camera layouts can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by computing unit 601, one or more steps of the BEV perception model training method and the BEV perception prediction method supporting multiple camera layouts described above can be performed. Alternatively, in other embodiments, computing unit 601 can be configured to perform the BEV perception model training method and the BEV perception prediction method supporting multiple camera layouts by any other suitable means (e.g., by means of firmware).

[0082] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, at least one input device, and at least one output device.

[0083] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0084] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0085] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0086] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0087] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0088] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0089] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A training method for a BEV perception model supporting multiple camera layouts, characterized in that, include: Acquire multiple key perception data in different camera layouts. Each key perception data includes image data from n viewpoints at the current moment in its corresponding camera layout, as well as the intrinsic and extrinsic parameter matrices of the corresponding camera. The number of camera viewpoints or viewpoints varies in each camera layout. For each key perception data point, the following steps are performed: Image features are extracted from image data from n viewpoints at the current time, and voxel query parameters are constructed based on the current camera layout scene; the intrinsic and extrinsic parameter matrices in the current camera layout are concatenated and converted into camera parameter encodings of the same dimension as the image features; based on the camera parameter encoding, the intrinsic and extrinsic parameter matrices are encoded into the image features to obtain image features fused with camera parameters; and based on the camera parameter encoding, the intrinsic and extrinsic parameter matrices are encoded into the voxel query parameters to obtain voxel query parameters fused with camera parameters; based on the image features and voxel query parameters of the fused camera parameters, spatial transformation is performed using a voxel query mechanism to obtain vehicle-related BEV spatial temporal features, where n is a positive integer. The BEV perception model is trained by using multiple determined BEV spatial temporal features and labels that are annotated for each key perception data based on the selected perception task as training samples, and optimizing the parameters of the initialization model using the gradient descent algorithm.

2. The method as described in claim 1, characterized in that, The extrinsic parameter matrix is ​​{M} k } 1≤k≤n M∈R 3×4 The intrinsic parameter matrix is ​​{K} k } 1≤k≤n , K∈R 3×3 ; The intrinsic and extrinsic parameter matrices in the current camera layout are then concatenated and converted into camera parameter encodings of the same dimension as the image features. Specifically, this includes: The extrinsic parameter matrix M∈R is respectively... 3×4 and the intrinsic parameter matrix K∈R 3×3 Straighten it into a one-dimensional matrix according to the set arrangement rules; The straightened one-dimensional extrinsic and intrinsic parameter matrices are concatenated and converted into camera parameter codes ξ∈R. 21×H×W The dimensions of the camera parameter encoding are the same as the dimensions of the image features; H and W are the height and width of the image features, respectively.

3. The method as described in claim 2, characterized in that, Based on the camera encoding parameters, the intrinsic and extrinsic parameter matrices are encoded into the image features to obtain image features fused with camera parameters, specifically including: Based on the height and width of the image features, an image location code P is generated. c ∈R 2×H×W , where P c (·,u,v) =[u / W,v / H] T u and v are the pixel coordinates of the image feature; The camera parameter encoding and the image location encoding are fused through a fully connected layer, and the fused first vector is fused with the image features to obtain image features with fused camera parameters: in, This represents a concatenation operation of vectors, where F is an image feature, and... This represents a fully connected layer, used to map the fused first vector to the same dimension as the image features; Furthermore, based on the camera encoding parameters, the intrinsic and extrinsic parameter matrices are encoded into the voxel query parameters to obtain the voxel query parameters for fused camera parameters, specifically including: Based on the voxel query three-dimensional dimensions, a voxel position code P is generated. v ∈R 3×Z×Y×X , where P v (·,d,h,w) =[d / Z, h / Y, w / X] T d, h, and w are the pixel coordinates of the voxel; The camera parameter encoding and the voxel position encoding are fused through a fully connected layer, and the fused second vector is fused with the voxel query to obtain the voxel query parameters of the fused camera parameters: Among them, the For image features, the This represents a fully connected layer used to map the fused second vector to the same dimension as the image features.

4. The method according to any one of claims 1-3, characterized in that, Based on the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters, spatial transformation is performed using a voxel query mechanism to obtain vehicle-related BEV spatial temporal features, specifically including: Based on the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters, the image features of the fused camera parameters at the current moment are converted into the vehicle-related BEV spatial features at the current moment based on the voxel query mechanism. The BEV spatial features at the current moment are fused with the BEV spatial features at multiple previous moments to obtain the vehicle-related BEV spatial temporal features.

5. The method as described in claim 4, characterized in that, Based on the image features of the fused camera parameters and the voxel query parameters of the fused camera parameters, the image features of the fused camera parameters at the current moment are transformed into vehicle-related BEV spatial features at the current moment based on the voxel query mechanism, specifically including: Calculate the projected coordinates on the images from each viewpoint based on the three-dimensional coordinates of each voxel; Based on the deformation attention mechanism, the viewpoint image features corresponding to the projection coordinates of the image features of the fused camera parameters are extracted from the image features of the fused camera parameters using the query parameters in the voxel query parameters of the fused camera parameters. The BEV spatial features extracted from images from different viewpoints are aggregated to obtain the BEV spatial features at the current time.

6. A BEV perception prediction method supporting multiple camera layouts, characterized in that, include: Acquire multiple key perception data to be predicted and convert them into BEV spatial features; The BEV spatial features are input into the BEV perception model trained based on the method described in any one of claims 1-5, and the perception results are output. The camera layout of the multiple key perception data to be predicted is different from the camera layout during the training of the BEV perception model.

7. A BEV perception model training device supporting multiple camera layouts, characterized in that, include: The acquisition module is used to acquire multiple key perception data in different camera layouts. Each key perception data includes image data from n perspectives in its corresponding camera layout at the current moment, as well as the intrinsic and extrinsic parameter matrices of the corresponding camera. The number of camera perspectives or perspectives varies in each camera layout. The conversion module is used to perform the following operations for each key perception data: extracting image features based on image data from n perspectives at the current moment, and constructing voxel query parameters based on the current camera layout scene; concatenating the intrinsic and extrinsic parameter matrices in the current camera layout to convert them into camera encoding parameters of the same dimension as the image features; encoding the intrinsic and extrinsic parameter matrices into the image features based on the camera encoding parameters to obtain image features of fused camera parameters; and encoding the intrinsic and extrinsic parameter matrices into the voxel query parameters based on the camera encoding parameters to obtain voxel query parameters of fused camera parameters; and performing spatial conversion based on the image features of fused camera parameters and voxel query parameters of fused camera parameters to obtain vehicle-related BEV spatial features based on the voxel query mechanism. The training module is used to train a BEV perception model by using multiple determined BEV spatial features and labels that are annotated for each key perception data based on the selected perception task as training samples, and optimizing the parameters of the initialization model using the gradient descent algorithm.

8. A BEV perception and prediction device supporting multiple camera layouts, characterized in that, include: The extraction module is used to acquire multiple key perception data to be predicted and extract and convert them into BEV spatial features; The prediction module is used to input the BEV spatial features into the BEV perception model trained based on the method of any one of claims 1-5, and output the perception result; The camera layout of the multiple key perception data to be predicted is different from the camera layout during the training of the BEV perception model.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-5.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images