Model training method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a model training method and device, electronic equipment and storage medium. The model training method of the present disclosure comprises: obtaining a surround view image of a vehicle and a corresponding laser radar point cloud; obtaining strong enhanced BEV features and weak enhanced BEV features by using the surround view image through a teacher-student model, the teacher-student model comprising a teacher model and a student model; obtaining point cloud 3D features by using the laser radar point cloud through a point cloud model; determining a migration loss according to the point cloud 3D features and the strong enhanced BEV features; determining an image modal intra-loss according to the strong enhanced BEV features and the weak enhanced BEV features; obtaining an overall loss according to the migration loss and the image modal intra-loss; updating parameters of the point cloud model and the student model by using the overall loss, and updating parameters of the teacher model by using the updated parameters of the student model; wherein the student model is used to extract BEV features of the surround view image. The BEV features of the surround view image obtained by the model trained by the present disclosure can be directly used for 3D target detection.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a model training method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Currently, according to different data sources, 3D target detection can be divided into 3D target detection based on point cloud data and 3D target detection based on image data. Since images cannot preserve the spatial structure information of 3D data, the traditional 3D target detection based on a single 2D image has always lagged behind the 3D target detection algorithm based on point cloud in terms of accuracy.

[0003] In recent years, with the increase in the number of vehicle-mounted cameras, many studies have attempted to fuse the image data of multiple vehicle-mounted cameras to extract the BEV features within the 360° range of the vehicle. Since some cameras have overlapping fields of view, the influence of the depth missing of 2D images is alleviated to some extent. Therefore, the technology of detecting 3D targets within the driving range of the vehicle based on BEV features has become the mainstream of 3D target detection technology based on image data.

[0004] Based on the related technology of detecting 3D targets within the driving range of the vehicle based on BEV features, according to whether laser radar data is used to correct the depth in the model training stage, it can be roughly divided into two kinds, BEVDepth as the representative of 3D target detection and BEVDet, PETR as the representative of 3D target detection. In the BEVDepth as the representative of 3D target detection technology, the laser radar data is used to supervise and correct the depth prediction of the image, which can greatly improve the prediction accuracy of the 3D target detection model. The PETR algorithm using Transformer to implicitly obtain BEV features, compared with BEVDepth, only lags behind 1% in the NuScenes detection score (NDS) index, which shows that using multiple vehicle-mounted cameras can largely make up for the inherent defects of the depth missing of 2D images. However, based on the surround view BEV features generated by multiple vehicle-mounted cameras for 3D target detection, the NDS accuracy reached only 0.664; while the NDS accuracy based on laser radar point cloud reached 0.734. Therefore, even if there is an overlapping field of view of the vehicle-mounted camera, there is still an essential representation defect in the 2D image data. Therefore, using multi-modal data containing laser radar data and image data to train the 3D target detection model has become the main means to improve the accuracy of the 3D target detection model.

[0005] In the technical solution disclosed in the paper UVTR: Unifying Voxel-based Representation with Transformer for 3D Object Detection, the accuracy of the 3D object detection model is improved by aligning the BEV features from image data with the point cloud features from lidar data, but the result is not ideal. Therefore, how to retain the representation advantages of 2D image features in texture and color in the BEV features, and fully exploit the 3D structural features and include them in the BEV features, is the key to improving the accuracy of the 3D object detection model using multi-modal data.

[0006] Currently, the supervised method is mainly used when training the 3D object detection model using multi-modal data. Under the pressure of geometrically growing data volume and complex scenes, the supervised training method greatly limits the generalization ability of the 3D object detection model, resulting in low accuracy of the 3D object detection model, high training cost, and difficulty in applying to various scenes. SUMMARY

[0007] To solve at least one of the above technical problems, the present disclosure provides a model training method, device, electronic equipment and storage medium.

[0008] According to one aspect of the present disclosure, a model training method is provided, comprising:

[0009] Obtaining a surround view image of a vehicle and a lidar point cloud corresponding to the surround view image;

[0010] Obtaining a strong enhanced BEV feature and a weak enhanced BEV feature using the surround view image through a teacher-student model, the teacher-student model comprising a teacher model and a student model;

[0011] Obtaining a point cloud 3D feature using the lidar point cloud through a point cloud model;

[0012] Determining a migration loss according to the point cloud 3D feature and the strong enhanced BEV feature;

[0013] Determining an image modal intra-loss according to the strong enhanced BEV feature and the weak enhanced BEV feature;

[0014] Obtaining an overall loss according to the migration loss and the image modal intra-loss;

[0015] Updating parameters of the point cloud model and the student model using the overall loss, and updating parameters of the teacher model using the updated parameters of the student model;

[0016] The student model is used to extract a BEV feature of the surround view image.

[0017] In some implementations of the model training method of the embodiments of the present disclosure, the parameters of the point cloud model and the student model are updated using the overall loss, and the parameters of the teacher model are updated using the updated parameters of the student model, including: in the process of back propagation of the overall loss, the gradient of the teacher model is blocked from being returned back, and the gradient is returned back only in the student model and the point cloud model to update the parameters of the student model and the point cloud model; and the parameters of the teacher model are updated using the updated parameters of the student model in an EMA manner.

[0018] In some implementations of the model training method of the embodiments of the present disclosure, the parameters of the teacher model are updated using the updated parameters of the student model in an EMA manner, including: updating the corresponding parameters ξ in the teacher model based on the parameters ω in the updated student model according to the following formula:

[0019] ξ = τξ + (1 - τ)ω

[0020] wherein τ is a decay rate, τ ∈ [0, 1].

[0021] In some implementations of the model training method of the embodiments of the present disclosure, the strong enhanced BEV feature includes a strong enhanced 3D BEV feature and a strong enhanced 2D BEV feature, and the weak enhanced 3D feature includes a weak enhanced 3D BEV feature and a weak enhanced 2D BEV feature; the image intra-modal loss is determined according to the strong enhanced BEV feature and the weak enhanced BEV feature, including: calculating a contrast loss between the strong enhanced 3D BEV feature and the weak enhanced 3D BEV feature to obtain a 3D feature contrast loss; calculating a contrast loss between the strong enhanced 2D BEV feature and the weak enhanced 2D BEV feature to obtain a 2D feature contrast loss; and obtaining the image intra-modal loss between the strong enhanced BEV feature and the weak enhanced BEV feature according to the 3D feature contrast loss and the 2D feature contrast loss.

[0022] In some implementations of the model training method of the embodiments of the present disclosure, the strong enhanced BEV feature includes a strong enhanced 3D BEV feature; and the transfer loss is determined according to the point cloud 3D feature and the strong enhanced BEV feature, including: calculating a contrast loss between the strong enhanced 3D BEV feature and the point cloud 3D feature.

[0023] In some implementations of the model training method in the embodiments of the present disclosure, the obtaining of the strong enhanced BEV feature and the weak enhanced BEV feature from the surround view image by the teacher-student model comprises: performing different degrees of data enhancement processing on each perspective image in the surround view image to obtain weak enhanced image data and strong enhanced image data of each perspective image in the surround view image; processing the weak enhanced image data of each perspective image in the surround view image by the teacher model to obtain a weak enhanced BEV feature of each perspective image; and processing the strong enhanced image data of each perspective image in the surround view image by the student model to obtain a strong enhanced BEV feature of each perspective image.

[0024] In some implementations of the model training method in the embodiments of the present disclosure, the different degrees of data enhancement processing on each perspective image in the surround view image comprises: respectively performing scaling and cropping on each perspective image to obtain scaled and cropped perspective images of a uniform size; and performing random flipping and different degrees of rotation operations on the scaled and cropped perspective images to obtain weak enhanced images and strong enhanced images of each perspective image.

[0025] In some implementations of the model training method in the embodiments of the present disclosure, the different degrees of rotation operations comprise weak operations, and the weak operations comprise: selecting one camera as a reference camera from a plurality of cameras used to collect each perspective image in the surround view image, and determining a rotation matrix of the reference camera according to a predetermined image rotation angle; converting the rotation matrix of the reference camera to the image coordinate system of each of the other cameras to obtain the rotation matrix of each of the other cameras; and after performing random flipping on each scaled and cropped perspective image, processing the flipped perspective image based on the rotation matrix of the corresponding camera to obtain a weak enhanced image of each perspective image.

[0026] In some implementations of the model training method in the embodiments of the present disclosure, the processing of the teacher model and the processing of the student model respectively comprise: extracting 2D image features of each perspective image; predicting depth information by using the 2D image features of each perspective image to obtain 3D image features of each perspective image; obtaining 3D BEV features of the surround view image by using the 3D image features of each perspective image; and collapsing the 3D BEV features of the surround view image into 2D BEV features.

[0027] In some implementations of the model training method in the embodiments of the present disclosure, the BEV features of the surround view image are used for 3D target detection.

[0028] According to a second aspect of the present disclosure, a model training apparatus is provided, comprising:

[0029] An acquisition unit is configured to acquire a surround view image of a vehicle and a laser radar point cloud corresponding to the surround view image.

[0030] an image processing unit configured to obtain strong enhanced BEV features and weak enhanced BEV features by a teacher-student model using the surround view images, the teacher-student model comprising a teacher model and a student model;

[0031] a point cloud processing unit configured to obtain point cloud 3D features by a point cloud model using the lidar point cloud;

[0032] a transfer loss determination unit configured to determine a transfer loss according to the point cloud 3D features and the strong enhanced BEV features;

[0033] an intra-modal loss determination unit configured to determine an image intra-modal loss according to the strong enhanced BEV features and the weak enhanced BEV features;

[0034] a total loss determination unit configured to obtain a total loss according to the transfer loss and the image intra-modal loss;

[0035] a parameter updating unit configured to update parameters of the point cloud model and the student model using the total loss, and update parameters of the teacher model using the updated parameters of the student model;

[0036] The student model is configured to extract BEV features of the surround view images.

[0037] In a third aspect, the present disclosure provides an electronic device, comprising:

[0038] a memory storing execution instructions; and

[0039] a processor executing the execution instructions stored in the memory, so that the processor executes the model training method described above.

[0040] In a fourth aspect, the present disclosure provides a readable storage medium, wherein the readable storage medium stores execution instructions, and the execution instructions are executed by a processor to implement the model training method described above.

[0041] The present embodiment combines the intra-modal contrast loss of 2D images and the cross-modal contrast loss of 2D images and point clouds to realize the pre-training of the BEV feature extraction model (i.e., the student model herein), which not only accelerates the model convergence and effectively improves the model precision, but also improves the representation ability of the BEV features. In addition, the present embodiment only uses lidar point cloud data in the training stage, and only image data such as surround view images are required in the application stage to realize the extraction of BEV features, thereby realizing the purpose of 3D target detection and other purposes, which can greatly reduce the computing resources required by applications such as 3D target detection in the autonomous driving scene, and at the same time improve the precision of applications such as 3D target detection. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the general description given above, and the detailed description given below, serve to explain the principles of the present disclosure. These drawings should be understood are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0043] Figure 1 is a flowchart of a model training method according to an embodiment of the present disclosure.

[0044] Figure 2 is an architecture diagram of a deep learning network according to an embodiment of the present disclosure.

[0045] Figure 3 is a process diagram of a teacher-student model obtaining 3D image features according to an embodiment of the present disclosure.

[0046] Figure 4 is a diagram of an overall loss calculation process and an EMA algorithm updating teacher model parameters process according to an embodiment of the present disclosure.

[0047] Figure 5 is a structural schematic block diagram of a 3D object detection device using a hardware implementation of a processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] The present disclosure will be described in further detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related content, and not to limit the present disclosure. In addition, it should be noted that only parts related to the present disclosure are shown in the drawings for ease of description.

[0049] It should be noted that the embodiments and features in the present disclosure can be combined with each other without conflict. The technical solutions of the present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0050] Unless otherwise specified, the exemplary embodiments / examples shown will be understood to provide exemplary features of various details that can implement the technical concepts of the present disclosure in practice. Therefore, unless otherwise specified, the features of various embodiments / examples can be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of the present disclosure.

[0051] The use of cross-hatching and / or shading in the drawings is generally used to make the boundaries and regions of adjacent components more clearly understood. As such, unless specifically stated otherwise, the presence of cross-hatching or shading in one part of a figure does not necessarily constitute a preference or requirement for a specific material, material property, dimension, ratio, commonality between illustrated components, or any other characteristic, attribute, property, or the like of the components in that figure. Moreover, the size and relative sizes of components in the figures can be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be practiced differently, a specific sequence of processes can be performed in an order other than described. For example, two sequentially described processes can be performed at substantially the same time or in the reverse order described. Further, like reference numerals indicate like components.

[0052] When a component is referred to as being "on" or "over" another component, "connected to" or "coupled to" another component, it can be directly on, directly connected to, or directly coupled to the other component, or one or more intervening components can be present. In contrast, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components present. To that extent, the term "connected" can refer to physical or electrical connection, with or without intervening components.

[0053] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, when the terms "comprises," "comprising," "includes," "including," and / or "contains," "containing," are used, it is understood that "comprising," "including," and / or "contains" and / or "containing," specify the presence of stated features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It is also noted that, as used herein, the terms "substantially," "approximately," and other similar terms are used as terms of approximation and not as terms of degree unless the context clearly indicates otherwise. Accordingly, they are employed herein to account for inherent deviations in measurements, calculations, and / or other processes and / or their results.

[0054] Currently, the sensor data that can be directly used for 3D target detection in the field of automatic driving mainly includes image data collected by a vehicle-mounted camera, laser point cloud data, radar point cloud data, etc. Any single data source has corresponding advantages:

[0055] Image data belongs to dense data and can retain rich textures, colors, lighting, etc. in real scenes. Laser point cloud data can accurately calculate and detect the distance of a spatial object.

[0056] Currently, the technology of image BEV feature extraction is relatively mature, and the mainstream direction is to directly fuse point cloud BEV features and image BEV features in the BEV feature stage. However, this method requires simultaneous processing of laser point cloud data and vehicle-mounted camera data in the 3D target detection process, which requires high computing power of the vehicle-mounted device and cannot meet the real-time requirements at present.

[0057] Secondly, existing multi-modal 3D target detection is limited to supervised learning. The paper BYOL: Bootstrap your own latent: A new approach to self-supervised Learning has proved that using only positive examples to calculate contrastive loss and combining EMA algorithm for pre-training can significantly improve the representation ability of 3D target detection model. However, directly using the surround view BEV features of the vehicle-mounted camera for model pre-training is still a blank, and the main reasons are as follows: first, BEV feature extraction requires a large amount of computing resources; second, BEV features lack an effective loss evaluation algorithm.

[0058] The following is a brief description of the unsupervised 3D target detection model training technology in the field of autonomous driving.

[0059] The paper BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View published at the end of 2021 as the pioneer of BEV, inherited the paradigm of Lift-splat, first extracted the 2D features of the image, then used the visual Transformer to convert the 2D features from the image view to the BEV view, finally, based on the BVE features, constructed a task-specific head, and predicted the 3D target. BEVDepth appeared in 2022, which integrated the idea of CaDDN based on the 2D features of the image to predict the depth information, and used the laser radar point cloud data to supervise the predicted depth, the experimental results proved that this method improved the NDS index of BEVDet on the nuScenes dataset by nearly 10%. However, although BEVDepth introduced point cloud in the 3D target detection model training stage for depth supervision, it did not effectively utilize the structural information of the foreground objects in the point cloud, therefore, the results were not ideal.

[0060] To address the shortcomings of BEVDet and BEVDepth, the paper UVTR: Unifying Voxel-based Representation with Transformer for 3D Object Detection published in 2022 aims to unify multi-modal representation in voxel space, achieving accurate and robust single-modal or cross-modal 3D detection. The paper proposes cross-modal interaction, utilizing the inherent features of different sensors to achieve knowledge transfer and modal fusion. BEVDistill published in November 2022 further proposes the concept of BEV feature distillation, using 3D BEV features extracted from point clouds as the teacher and 2D BEV features extracted from multi-view images as the student. It calculates BEV feature distillation loss and instance target prediction distillation loss to transfer knowledge from point cloud features to image features. BEVDistill improves the NDS index on the nuScenes dataset by nearly 1% compared to UVTR, setting a research paradigm for subsequent BEV knowledge distillation. The paper TiGBEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning published at the end of 2022 analyzes the shortcomings of BEVDepth and UVTR, proposes depth constraints for the internal structure of foreground targets, and improves the accuracy of 2D image feature depth prediction, resulting in a 1% improvement in NDS index. In addition, it proposes channel distillation and target key point distillation based on BEV features, effectively transferring point cloud structure information to 2D BEV features. However, the techniques disclosed in the above papers mainly apply knowledge distillation to BEV features. Since BEV features can only represent the features of objects at a certain height, i.e., the foreground objects (vehicles, pedestrians, etc.), they lack the ability to represent the entire 3D space, resulting in the loss of spatial structure information, which limits the accuracy improvement of 3D object detection models.

[0061] The paper CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding published at CVPR in 2022 combines intra-modal contrastive learning for laser point clouds with laser point cloud and 2D rendered image contrastive learning, proposing intra-modal and cross-modal contrastive learning paradigms. However, this method is only applicable to 3D point cloud single-object detection and cannot achieve multi-3D target detection in complex scenarios such as autonomous driving. Moreover, the method has limited effectiveness in improving the accuracy of 3D object detection models.

[0062] The present disclosure proposes a new model training method, which migrates the features of laser point cloud to the surround view BEV features of vehicle-mounted camera, thereby improving the representation ability of surround view image BEV features, not only improving the structure and processing process of teacher-student model, but also improving the calculation method of its loss, mining deep features, and realizing unsupervised pre-training of surround view image BEV feature extraction model based on multi-modal. It has been verified that the embodiment of the present disclosure can effectively improve the representation ability of surround view image BEV features, and then improve the precision and generalization ability of applications such as 3D target detection. The model trained by the embodiment of the present disclosure is suitable for various complex scenes of autonomous driving, and can be applied to applications such as multi-3D target detection based on 2D multi-view image, while improving the precision of applications such as 3D target detection. In addition, the present disclosure also uses EMA algorithm to update the parameters of the teacher model in the teacher-student model, which can further improve the representation ability of the model.

[0063] Figure 1 is a flowchart of a model training method according to an embodiment of the present disclosure. As shown in Figure 1 , the model training method of the embodiment of the present disclosure can include the following steps S102-S110:

[0064] Step S102, obtaining surround view images of a vehicle and corresponding laser radar point clouds of the surround view images;

[0065] Step S104, obtaining strong enhanced BEV features and weak enhanced BEV features by a teacher-student model using the surround view images, wherein the teacher-student model includes a teacher model and a student model;

[0066] Step S106, obtaining point cloud 3D features by a point cloud model using the laser radar point clouds;

[0067] Step S108, determining a migration loss according to the point cloud 3D features and the strong enhanced BEV features;

[0068] Step S110, determining an image modal intra-loss according to the strong enhanced BEV features and the weak enhanced BEV features;

[0069] Step S112, obtaining a total loss according to the migration loss and the image modal intra-loss;

[0070] Step S114, updating the parameters of the point cloud model and the student model using the total loss, and updating the parameters of the teacher model using the updated parameters of the student model.

[0071] Among them, the student model is used to obtain the BEV features of the surround view images. Here, the BEV features of the surround view images obtained by the student model can be, but are not limited to, 2D BEV features and / or 3D BEV features of the surround view images.

[0072] In a specific application, steps S102-S114 of the embodiments of the present disclosure can be iteratively performed until the model converges. During the iteration, the parameters of the teacher model obtained by step S114 in the current round can be applied to the training process of the student model in the next round.

[0073] In a specific application, the model trained by the embodiments of the present disclosure can be used for various applications such as 3D target detection, etc. For example, after the pre-training of the student model is completed, i.e., after the model converges and the training is completed, a detection head or the like can be connected to the output end of the student model, and 3D target information can be obtained by processing the BEV feature of the surround view image through the detection head. The 3D target information can be, but is not limited to, single 3D target information, multiple 3D target information, and the 3D target information can be represented by key point coordinates, size, etc. of a 3D bounding box. In this application, the detection head can be trained by the pre-trained student model. The detection head can be, but is not limited to, a semantic segmentation network or a target detection network. The architecture, training method, etc. of the detection head are not limited by the embodiments of the present disclosure.

[0074] The embodiments of the present disclosure propose a deep learning network for training the student model described above. The deep learning network can include a point cloud model, a teacher-student model, the point cloud model is used to extract point cloud 3D features of a laser radar point cloud, the teacher-student model includes a teacher model and a student model, the teacher model can be used to extract weakly augmented 3D features of a surround view image, and the student model can be used to extract strongly augmented 3D features of the surround view image. The point cloud model and the teacher model are used to realize unsupervised multi-modal training of the student model. The trained student model can be deployed as part of a 3D target detection model on an application end, which can be used to extract BEV features of a surround view image, so that a detection head or the like of the 3D target detection model can directly obtain 3D target information around the vehicle through the BEV features of the surround view image.

[0075] Figure 2 The overall structure of the deep learning network of the embodiments of the present disclosure is shown. As shown in Figure 2 The teacher-student model in the deep learning network can include a strongly data-augmented branch network and a weakly data-augmented branch network, the strongly data-augmented branch network is used as the student model in the teacher-student model, the weakly data-augmented branch network is used as the teacher model, and the point cloud model in the deep learning network can be implemented by using a point cloud feature extraction network such as SECOND FeatureEncoder, DGCNN, and PointPillars, etc. The strongly data-augmented branch network and the weakly data-augmented branch network, i.e., the student model and the teacher model, can use the same network structure, for example, can use, but are not limited to, BEVDepth, BEVDet, and other feature extraction networks.

[0076] As shown in Figure 2 , the network structures of the teacher model and the student model can respectively include: a 2D feature extraction network, a depth prediction network, a BEV network module, and a collapse model, the 2D feature extraction network is used to extract 2D image features of each perspective image, the depth prediction network can be used to estimate depth information of each perspective image based on the 2D image features of each perspective image to obtain 3D features of each perspective image, the BEV network module is used to obtain 3D BEV features of the surround view image by using the 3D features of each perspective image, and the collapse model can be used to collapse the 3D BEV features of the surround view image into 2D BEV features. The 2D feature extraction network can be, but is not limited to, ResNet50, ResNet101, etc.; the depth prediction network can be, but is not limited to, a multi-layer perceptron (MLP), a DepthNet, etc.; the BEV network module can be, but is not limited to, a voxel pooling network, and the collapse module can be, but is not limited to, a convolutional network, etc.

[0077] In step S102, the surround view image contains 2D images of multiple perspectives. Taking the NuScenes dataset as an example, each frame of the surround view image contains perspective images collected by 6 vehicle-mounted cameras at the same time, that is, each frame of the surround view image contains 6 perspective images, which are: a front left perspective image I front_left , a front perspective image I front , a front right perspective image I front_right , a rear right perspective image I back_right , a rear perspective image I back , and a rear left perspective image I back_left . At this time, a single frame of surround view image data can be represented as , wherein C represents the number of image channels, H represents the image height, and W represents the image width.

[0078] In step S102, the laser radar point cloud corresponding to the surround view image refers to the point cloud data of the environment around the vehicle collected by the vehicle-mounted laser radar, the collection time of the point cloud data is the same as or within a predetermined range of the collection time of the surround view image, and the coverage range of the laser radar partially or entirely overlaps with the field of view of the vehicle-mounted camera used to collect the surround view image.

[0079] Step S104 can include: performing different degrees of data enhancement processing on each perspective image in the surround view image to obtain weakly enhanced image data and strongly enhanced image data of each perspective image in the surround view image; processing the weakly enhanced image data of each perspective image in the surround view image by using the teacher model to obtain weakly enhanced 3D features of each perspective image; and processing the strongly enhanced image data of each perspective image in the surround view image by using the student model to obtain strongly enhanced 3D features of each perspective image.

[0080] In step S104, the surround view image can be subjected to data enhancement of different degrees by rotation, grayscale, etc. to obtain strong enhancement image data I strong and weak enhancement image data I weak .

[0081] In some embodiments, the data enhancement in step S104 can include, but is not limited to, scaling, random cropping, flipping, rotation, etc. of the image.

[0082] In some embodiments, the data enhancement in step S104 can include the following steps a1 and a2:

[0083] Step a1: scaling and cropping each view image to obtain a scaled and cropped view image of a uniform size;

[0084] Specifically, the image can be scaled according to a pre-set image scaling ratio range, and then the scaled image I r is randomly cropped to obtain a scaled and cropped view image of a uniform size.

[0085] In some embodiments, assuming that the scaling ratio range of the image is (0.386, 0.55), as shown in the following formula (1), the scaled image of each view image can be obtained by using the resize algorithm of OpenCV.

[0086] I r = R(I) (1)

[0087] wherein I is a view image (for example, I front_left ), R represents the resize operation, and I r is the scaled image of I.

[0088] In some embodiments, as shown in the following formula (2), the image scale can also be scaled by using bilinear interpolation to obtain the scaled image of each view image.

[0089] f(i+u, j+v) = (1-u)*(1-v)*f(i, j) + (1-u)*v*f(i, j+1) + u*(1-v*f(i+1, j) + u*v*f(i+1, j+1) (2)

[0090] wherein (i, j) is the integer part of the scaled pixel coordinates, (u, v) is the decimal part of the scaled pixel coordinates, and f(i, j) represents the pixel value of the image coordinates i, j.

[0091] Step a2: Randomly flip and rotate the scaled and cropped images obtained in step a1 to obtain the enhanced images I for each viewpoint. strong and weakly enhanced image I weak .

[0092] Forward-facing image For example, the process of randomly flipping the view image can be shown in the following equation (3):

[0093]

[0094] Randomflip represents a random left-right flip operation. This is a frontal view image after random flipping. This represents the frontal view image before random flipping.

[0095] Specifically, a rotation angle range can be preset, and weak operations (OPs) can be performed on the randomly flipped image within this range. w To obtain weakly enhanced image I weak Perform a strong operation (OP) on the randomly flipped image within this rotation angle range. s To obtain a strongly enhanced image I weak .

[0096] For example, the rotation angle range can be set to (-5.4°, 5.4°) or other values, and this disclosure does not limit the specific values.

[0097] In some implementations, the strong operation OP s It can be the coordinate system of the image center (C) with the top left corner of the image as the origin and an angle θ around the image center. x C y The rotation operation. Specifically, the pixel coordinates after the strong operation can be obtained by the following equation (4).

[0098]

[0099] Where (x0, y0) represents the original pixel coordinates, i.e., the coordinates of the pixel before the strong operation, and (x, y) represents the coordinates of the pixel after the strong operation. C represents the original pixel coordinates after homogenization. x For W0 / 2, C y H0 / 2, (x, y, 1) represents the homogeneous pixel coordinates after the rotation operation.

[0100] In some implementations, weak operation OP wThis can include: selecting one camera as the reference camera from among multiple cameras used to acquire images from various viewpoints in the panoramic view (6 cameras in the nuScenes dataset); determining the rotation matrix of the reference camera based on a predetermined image rotation angle; transforming the rotation matrix of the reference camera into the image coordinate system of the other cameras to obtain the rotation matrices of the other cameras; randomly flipping the scaled and cropped images from each viewpoint; and then processing the randomly flipped images based on the rotation matrices of the corresponding cameras to obtain a weakly enhanced image I for each viewpoint. weak This allows for the maintenance of consistent data enhancement across images from various perspectives.

[0101] Still from the front-view image For example, the process of obtaining weakly enhanced images from various viewpoints can be as follows:

[0102] Assuming camera A is a pre-selected reference camera, and camera B is the camera that acquires the front view image, the rotation matrix of the image rotation angle θ of camera A can be expressed as the following equation (5):

[0103]

[0104] First, determine the rotation matrix R′ of camera B according to the following formula (6). z Then, the weakly enhanced image of the randomly flipped front view image is obtained by using the following formula (7):

[0105]

[0106] P′ B =R′ z P B (7)

[0107] Among them, R t This represents the image captured by camera B. The transformation matrix R from the pixels on camera A to the image coordinate system in camera A. z Let R′ represent the image rotation matrix of camera A. z R represents the image rotation matrix of camera B. A and R B Let A and B be the extrinsic parameter matrices, respectively. K A and K B Let A and B be the intrinsic parameter matrices of cameras A and B, respectively, denoted as... As can be seen from equation (6), the rotation matrix R′ of camera B is... z It is the projection of the image transformation matrix of camera A onto the image coordinate system of camera B.

[0108] Among them, P B P′ represents the coordinates of a pixel in the randomly flipped front-view image.B The weakly enhanced image data representing the forward-view image corresponds to P. B The coordinates of the pixel.

[0109] Equation (6) can be derived from equations (8) to (11):

[0110] The coordinates of points in the image coordinate systems of camera A and camera B satisfy the following equations (8) to (9):

[0111] P A =K A R A P W (8)

[0112] P B =K B R B P W (9)

[0113] Among them, P w Let P be a point in the world coordinate system. A and P B P respectively W The projection points in the image coordinate systems of camera A and camera B.

[0114] Assume the image rotation matrix in camera A is R. z The rotation matrix corresponding to camera B is R′. z Then, the following equations (10) to (11) are satisfied:

[0115] P′ A =R z P A =R z K A R A P W (10)

[0116] P′ B =R′ z P B =R′ z K B R B P W (11)

[0117] Among them, P′ B The weakly enhanced image representing the frontal view corresponds to P. B The coordinates of the pixel, P′ A This represents the weakly enhanced image of the viewpoint image of camera A corresponding to P. A The coordinates of the pixel. P A This represents the coordinates of a pixel in the view image of camera A after random flipping.

[0118] Since the 2D image transformation does not affect the depth, the depth-invariant assumption can be made in the above derivation process.

[0119] In step S104, the processing of the teacher model and the processing of the student model are the same, and can respectively include steps b1-b4: step b1, extracting 2D image features of each view image Step b2, predicting depth information D using 2D image features of each view image pred to obtain 3D image features of each view image Step b3, obtaining 3D BEV features of the surround view image using 3D image features of each view image; and step b4, collapsing the 3D BEV features of the surround view image into 2D BEV features.

[0120] Specifically, in step S104, the teacher model is for the weakly augmented image I weak Steps b1-b4 are sequentially executed to obtain weakly augmented 3D features of each view image, which include weakly augmented 3D BEV features and weakly augmented 2D BEV features The student model is for the strongly augmented image I strong Steps b1-b4 are sequentially executed to obtain strongly augmented 3D features of each view image, which include strongly augmented 3D BEV features and strongly augmented 2D BEV features

[0121] Figure 3 An exemplary processing procedure of steps b1-b2 is shown. As Figure 3 shown, the process of steps b1-b2 can be implemented by the following formulas (12)-(13):

[0122]

[0123]

[0124] wherein DepthNet1 represents the depth extraction network in the student model, DepthNet2 represents the depth extraction network in the teacher model, represents 3D image features of the strongly augmented image of the surround view image (hereinafter referred to as strongly augmented 3D image features), represents 3D image features of the weakly augmented image of the surround view image, represents 2D image features of the weakly augmented image of the surround view image (hereinafter referred to as weakly augmented 3D image features), represents 2D image features of the strongly augmented image of the surround view image.

[0125] In step b3, 3D image features of each view image in the same frame of surround view image are unified to the 3D BEV features of the frame of surround view image under the BEV plane. For example, the strongly enhanced 3D image features of each view image in the same frame of surround view image can be unified to the strongly enhanced 3D BEV features of the frame of surround view image under the BEV plane. The voxel pooling network of the student model is inputted to extract the strongly enhanced 3D BEV features The weakly enhanced 3D image features of each view image in the same frame of surround view image are unified to the weakly enhanced 3D BEV features of the frame of surround view image under the BEV plane. The voxel pooling network of the teacher model is inputted to extract the weakly enhanced 3D BEV features

[0126] For example, step b3 can be implemented by the following formulas (14)~(15):

[0127]

[0128]

[0129] Wherein, Voxel Pooling1 and Voxel Pooling2 are 3D to BEV feature conversion operators with the same structure. Voxel Pooling1 represents the operation in the voxel pooling network of the student model, and Voxel Pooling2 represents the operation in the voxel pooling network of the teacher model.

[0130] In step b4, the D dimension of the 3D BEV features can be compressed using a convolution network, so as to collapse the 3D BEV features into 2D BEV features. The size of the BEV features obtained by the convolution network can be (1, H0, W0).

[0131] Specifically, the strongly enhanced 3D BEV features of the surround view image can be collapsed into strongly enhanced 2D BEV features by compressing the D dimension of the strongly enhanced 3D BEV features. The convolution network of the student model is inputted to compress the D dimension thereof, so as to collapse the strongly enhanced 3D BEV features of the surround view image into strongly enhanced 2D BEV features The weakly enhanced 3D BEV features of the surround view image can be collapsed into weakly enhanced 2D BEV features by compressing the D dimension of the weakly enhanced 3D BEV features. The convolution network of the teacher model is inputted to compress the D dimension thereof, so as to collapse the weakly enhanced 3D BEV features of the surround view image into weakly enhanced 2D BEV features

[0132] For example, the convolution network used to implement the collapse can be Conv2d or Conv3d. In this case, the process of step b4 can be represented by the following formulas (16)~(17):

[0133]

[0134]

[0135] where Collapse1 and Collapse2 represent collapse operations.

[0136] In step S106, the laser radar point cloud can be pre-processed to obtain voxel data P of the point cloud, and then a point cloud model such as SECOND Feature Encoder is used to extract 3D features F of the voxel data P p ∈R C×D×h×w .

[0137] In step S108, a contrast loss CLLoss between the point cloud 3D features and the strongly enhanced 3D BEV features is calculated, which is a migration loss, i.e., a cross-modal migration loss. By adding this loss, cross-modal feature migration from laser point cloud data to 2D image data can be achieved.

[0138] For example, the contrast loss L between the point cloud 3D features F p and the strongly enhanced 3D BEV features can be calculated by the L1 norm loss function shown in the following formula (18): inter :

[0139]

[0140] In some embodiments, the intra-image modality loss in step S110 can include two parts: 1) a contrast loss between the 3D features calculated directly from the strongly enhanced 3D BEV features and the weakly enhanced 3D BEV features; and 2) a contrast loss between the 2D features calculated from the strongly enhanced 2D BEV features and the weakly enhanced 2D BEV features.

[0141] In this embodiment, step S110 can include the following steps c1-c3:

[0142] Step c1, calculate the contrast loss between the strongly enhanced 3D BEV features and the weakly enhanced 3D BEV features to obtain a 3D feature contrast loss;

[0143] Step c2, calculate the contrast loss between the strongly enhanced 2D BEV features and the weakly enhanced 2D BEV features to obtain a 2D feature contrast loss;

[0144] Step c3, obtain the intra-image modality loss between the strongly enhanced BEV features and the weakly enhanced BEV features according to the 3D feature contrast loss and the 2D feature contrast loss.

[0145] For example, the contrast loss between the strongly enhanced 3D BEV features and the weakly enhanced 3D BEV features can be calculated by the L1 norm loss function of the following formula (19): The contrast loss between the strong enhanced 2D BEV features and the weak enhanced 2D BEV features is calculated by an L1 norm loss function of formula (20)

[0146]

[0147]

[0148] In step c3, the 3D feature contrast loss and the 2D feature contrast loss can be combined to obtain the intra-image modality loss.

[0149] For example, the intra-image modality loss L between the strong enhanced BEV features and the weak enhanced BEV features can be calculated by formula (21) intra :

[0150]

[0151] wherein γ is a balance factor, and γ takes an empirical value. For example, γ = 2 can be taken.

[0152] In step S112, the overall loss L can be calculated by formula (22) all :

[0153] L all = αL inter + βL intra (22)

[0154] wherein α represents a weight value of the transfer loss L inter , and β represents a weight value of the intra-image modality loss L intra , both of which can take empirical values. For example, α = 2 and β = 1 can be taken.

[0155] In step S112, the gradient descent method can be used to update the model parameters based on the overall loss in a backpropagation manner. Specifically, in the backpropagation process of the overall loss, the gradient of the teacher model is blocked from being returned, and the gradient of only the student model and the point cloud model is returned to update the parameters of the student model and the point cloud model, and then the parameters of the updated student model are used to update the parameters of the teacher model using the EMA algorithm.

[0156] Figure 4 The calculation of the overall loss and the EMA updating process of the parameters of the teacher model in the embodiments of the present disclosure are shown. Figure 4 ξ ω is a 3D feature extraction network, q ξ ω is an MLP network for BEV feature compression. f η is a point cloud model for extracting point cloud 3D features. Figure 4 ​​​The solid arrows represent the entire calculation process of the overall loss, and " / / " represents blocking.

[0157] In some embodiments, the parameters of the teacher model can be updated by the following formula (23):

[0158] ξ = τξ + (1 - τ)ω (23)

[0159] wherein ξ is a parameter in the teacher model corresponding to ω, ω represents a certain parameter of the updated student model, τ is a decay rate, τ ∈ [0, 1]. For example, τ can be 0.9.

[0160] The embodiments of the present disclosure adopt strong and weak data enhancement methods to process the surround view image, obtain its BEV feature and calculate the loss, and explore 3D information from the BEV perspective. At the same time, the transfer learning method is used to align the 2D image feature and the point cloud feature, and assist the model to converge. Moreover, the embodiments of the present disclosure combine the intra-modal contrast loss of the 2D image and the cross-modal contrast loss of the 2D image and the point cloud to pre-train the model, which not only accelerates the model convergence, but also effectively improves the model precision.

[0161] The embodiments of the present disclosure improve the feature representation ability of the student model by migrating the features of the laser point cloud to the BEV features of the surround view image, so that the BEV features extracted from the surround view image by the student model not only contain the texture, color, illumination and other information of the 2D image, but also implicitly contain the spatial structure information of the point cloud. Therefore, the BEV features extracted from the surround view image by the student model trained by the embodiments of the present disclosure can effectively improve the accuracy of 3D target information and improve the precision of 3D target detection.

[0162] The embodiments of the present disclosure improve the precision of the BEV feature extraction part in the 3D target detection model by migrating the features of the laser point cloud data to the BEV features of the image data in the training stage, and only image data is needed in the detection stage, which can greatly reduce the need for computing resources of the model and improve the detection precision of the image-based surround BEV features.

[0163] Figure 5 A model training device example diagram using a hardware implementation of a processing system is shown.

[0164] The device can include corresponding modules that perform one or several steps in the above flowcharts. Therefore, each step or several steps in the above flowcharts can be performed by corresponding modules, and the device can include one or more of these modules. The modules can be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer readable medium for implementation by a processor, or implemented by some combination.

[0165] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 600 connects various circuits, including one or more processors 700, memory 800, and / or hardware modules. Bus 600 can also connect various other circuits 900, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0166] Bus 600 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0167] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0168] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0169] For the purposes of this specification, a "read only memory" can be any apparatus that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus or device. More specific examples (a non-exhaustive list) of the read only memory include the following: an electrical connection having one or more wires (electrical apparatus), a portable computer diskette (magnetic apparatus), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or Flash memory), an optical fiber device, and a portable compact disc read only memory (CD ROM). In addition, the read only memory can even be paper or another suitable medium upon which the program can be printed, because the program can be electronically obtained, for example by optical scanning of the paper or other medium, followed by electronic means to produce the program, and then stored in the memory.

[0170] It should be understood that portions of the present disclosure can be realized with hardware, software, or a combination of them. In the above-described embodiments, a plurality of steps or methods can be realized with software stored in a memory and executed by a suitable instruction execution system. For example, if realized with hardware, and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, application specific integrated circuit having a suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA), etc.

[0171] Those skilled in the art of the present technology can understand that all or part of the steps of the above-described embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a readable storage medium. When executed, the programs include one or a combination of the steps of the method embodiments.

[0172] In addition, each functional unit in each embodiment of the present disclosure can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read only memory, a magnetic disk or an optical disk, etc.

[0173] Figure 5 is a structural schematic diagram of a model training device according to an embodiment of the present disclosure. As shown in Figure 5 The model training device 500 according to the present disclosure can include:

[0174] The acquisition unit 502 is configured to acquire a surround view image of a vehicle and a laser radar point cloud corresponding to the surround view image.

[0175] The image processing unit 504 is configured to obtain strong enhanced BEV features and weak enhanced BEV features by using the surround view image through a teacher-student model, the teacher-student model including a teacher model and a student model.

[0176] The point cloud processing unit 506 is configured to obtain point cloud 3D features by using the laser radar point cloud through a point cloud model.

[0177] The migration loss determination unit 508 is configured to determine a migration loss according to the point cloud 3D features and the strong enhanced BEV features.

[0178] The intra-modal loss determination unit 510 is configured to determine an image intra-modal loss according to the strong enhanced BEV features and the weak enhanced BEV features.

[0179] The overall loss determination unit 512 is configured to obtain an overall loss according to the migration loss and the image intra-modal loss.

[0180] The parameter updating unit 514 is configured to update parameters of the point cloud model and the student model by using the overall loss, and update parameters of the teacher model by using the updated parameters of the student model.

[0181] In specific applications, the BEV features of the surround view image can be directly used for 3D target detection.

[0182] The specific technical details of each unit in the model training apparatus 500 can be found in the foregoing method part, and will not be described here again.

[0183] The present disclosure also provides an electronic device, including a memory storing execution instructions, and a processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module executes the above-mentioned model training method.

[0184] The present disclosure also provides a readable storage medium, which stores execution instructions, and the execution instructions are executed by a processor to implement the above-mentioned model training method.

[0185] In the description of the present specification, the description of the term "one embodiment / way", "some embodiments / ways", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present application. It is not necessarily the same embodiment / way or example in the illustrative description of the above terms in the present specification. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments / ways or examples. In addition, the person skilled in the art can combine and combine the different embodiments / ways or examples described in the present specification and the features of the different embodiments / ways or examples without contradiction.

[0186] In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0187] The person skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. Other changes or modifications can be made on the basis of the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A model training method, characterized in that, The method comprises the following steps: obtaining a surround view image of a vehicle and a laser radar point cloud corresponding to the surround view image; obtaining strong enhanced BEV features and weak enhanced BEV features from the surround view image by using a teacher-student model, wherein the teacher-student model comprises a teacher model and a student model; obtaining point cloud 3D features from the laser radar point cloud by using a point cloud model; determining a migration loss according to the point cloud 3D features and the strong enhanced BEV features; determining an image modal intra-loss according to the strong enhanced BEV features and the weak enhanced BEV features; obtaining an overall loss according to the migration loss and the image modal intra-loss; updating parameters of the point cloud model and the student model by using the overall loss, and updating parameters of the teacher model by using the updated parameters of the student model; wherein the student model is used to extract BEV features of the surround view image; the step of obtaining the strong enhanced BEV features and the weak enhanced BEV features from the surround view image by using the teacher-student model comprises the following steps: performing different degrees of data enhancement processing on each perspective image in the surround view image to obtain weak enhanced image data and strong enhanced image data of each perspective image in the surround view image; obtaining weak enhanced BEV features of each perspective image by processing the weak enhanced image data of each perspective image in the surround view image through the teacher model; and obtaining strong enhanced BEV features of each perspective image by processing the strong enhanced image data of each perspective image in the surround view image through the student model; the strong enhanced BEV features comprise strong enhanced 3D BEV features and strong enhanced 2D BEV features, and the weak enhanced BEV features comprise weak enhanced 3D BEV features and weak enhanced 2D BEV features; the step of determining the image modal intra-loss according to the strong enhanced BEV features and the weak enhanced BEV features comprises the following steps: calculating a contrast loss between the strong enhanced 3D BEV features and the weak enhanced 3D BEV features to obtain a 3D feature contrast loss; calculating a contrast loss between the strong enhanced 2D BEV features and the weak enhanced 2D BEV features to obtain a 2D feature contrast loss; and obtaining the image modal intra-loss between the strong enhanced BEV features and the weak enhanced BEV features according to the 3D feature contrast loss and the 2D feature contrast loss.

2. The model training method of claim 1, wherein, the step of updating the parameters of the point cloud model and the student model by using the overall loss, and updating the parameters of the teacher model by using the updated parameters of the student model comprises the following steps: in the process of back propagation of the overall loss, the gradient of the teacher model is blocked from being returned back, and the gradient is returned back only in the student model and the point cloud model to update the parameters of the student model and the point cloud model; and the parameters of the teacher model are updated in an EMA manner by using the updated parameters of the student model.

3. The model training method of claim 2, wherein, the step of updating the parameters of the teacher model in an EMA manner by using the updated parameters of the student model comprises the following steps: updating the student model based on the updated parameters in the teacher model updating the corresponding parameters in the teacher model : wherein is the decay rate, .

4. The model training method of claim 1, wherein, the step of determining the migration loss according to the point cloud 3D features and the strong enhanced BEV features comprises the following step: calculating a contrast loss between the strong enhanced 3D BEV features and the point cloud 3D features.

5. The model training method of claim 1, wherein, the step of performing different degrees of data enhancement processing on each perspective image in the surround view image comprises the following steps: The scaling and cropping are respectively performed on each perspective image to obtain scaled and cropped perspective images with uniform size; Random flipping and different degrees of rotation operations are performed on the scaled and cropped perspective images to obtain weak augmented images and weak augmented images of each perspective image.

6. The model training method of claim 5, wherein, The different degrees of rotation operations include weak operations, and the weak operations include: A camera is selected as a reference camera from a plurality of cameras used to collect each perspective image of the surround view image, and a rotation matrix of the reference camera is determined according to a predetermined image rotation angle; The rotation matrix of the reference camera is converted to the image coordinate system of each of the other cameras to obtain the rotation matrix of each of the other cameras; After random flipping is performed on each of the scaled and cropped perspective images, the perspective image after the random flipping is processed based on the rotation matrix of the corresponding camera to obtain a weak augmented image of each perspective image.

7. The model training method of claim 1, wherein The processing of the teacher model and the processing of the student model respectively include: extracting 2D image features of each perspective image; predicting depth information by using the 2D image features of each perspective image to obtain 3D image features of each perspective image; obtaining 3D BEV features of the surround view image by using the 3D image features of each perspective image; collapsing the 3D BEV features of the surround view image into 2D BEV features.

8. A model training apparatus, comprising: comprises: an acquisition unit configured to acquire a surround view image of a vehicle and a laser radar point cloud corresponding to the surround view image; an image processing unit configured to obtain strong augmented BEV features and weak augmented BEV features by using the surround view image through a teacher-student model, the teacher-student model comprising a teacher model and a student model; a point cloud processing unit configured to obtain point cloud 3D features by using the laser radar point cloud through a point cloud model; a migration loss determination unit configured to determine a migration loss according to the point cloud 3D features and the strong augmented BEV features; an intra-modal loss determination unit configured to determine an image intra-modal loss according to the strong augmented BEV features and the weak augmented BEV features; an overall loss determination unit configured to obtain an overall loss according to the migration loss and the image intra-modal loss; a parameter updating unit configured to update parameters of the point cloud model and the student model by using the overall loss, and update parameters of the teacher model by using the updated parameters of the student model; wherein the student model is configured to extract BEV features of the surround view image; the teacher-student model is configured to perform different degrees of data augmentation processing on each perspective image in the surround view image to obtain weak augmented image data and strong augmented image data of each perspective image in the surround view image, process the weak augmented image data of each perspective image in the surround view image through the teacher model to obtain weak augmented BEV features of each perspective image, and process the strong augmented image data of each perspective image in the surround view image through the student model to obtain strong augmented BEV features of each perspective image. The strong enhancement BEV features include strong enhancement 3D BEV features and strong enhancement 2D BEV features, and the weak enhancement BEV features include weak enhancement 3D BEV features and weak enhancement 2D BEV features. The determining the intra-image modality loss according to the strong enhancement BEV features and the weak enhancement BEV features comprises: calculating a contrast loss between the strong enhancement 3D BEV features and the weak enhancement 3D BEV features to obtain a 3D feature contrast loss; calculating a contrast loss between the strong enhancement 2D BEV features and the weak enhancement 2D BEV features to obtain a 2D feature contrast loss; and obtaining the intra-image modality loss between the strong enhancement BEV features and the weak enhancement BEV features according to the 3D feature contrast loss and the 2D feature contrast loss.

9. An electronic device, comprising: comprise: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the model training method in any one of claims 1 to 7.

10. A readable storage medium, characterized by, The readable storage medium stores execution instructions, and the execution instructions are executed by the processor to implement the model training method in any one of claims 1 to 7.

Images