Model training method, object recognition method, device, vehicle and storage medium

Abstract

The present disclosure relates to the field of autonomous driving, and relates to a model training method, an object recognition method, an apparatus, a vehicle, and a storage medium. The method comprises: obtaining a first image acquired by a camera of a first object and first point cloud data acquired by a radar of the first object; extracting features of the first point cloud data through a radar feature extraction network to obtain a first feature; extracting features of the first image through a camera feature extraction network to obtain a second feature; converting the first feature to a bird's-eye view representation space to obtain a first bird's-eye view feature corresponding to the first point cloud data; converting the second feature to a bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image; and training the radar feature extraction network and the camera feature extraction network based on the first and second bird's-eye view features. This enables model training using unlabeled data acquired by the camera and radar, thereby reducing the model training process's reliance on labeled samples.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to a model training method, an object recognition method, a device, a vehicle, and a storage medium. Background Art

[0002] In the field of autonomous driving, trained models can be used to implement a variety of functions. For example, an image recognition model can be trained using image data to identify image information about the vehicle's environment. Alternatively, a laser model can be trained using laser data to process the relevant laser data.

[0003] However, in related model training methods, more labeled samples may be required. However, due to the small number of labeled samples, it is easy to encounter problems such as insufficient training data and poor model performance. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides a model training method, an object recognition method, an apparatus, a vehicle and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a model training method is provided, including:

[0006] Acquire a first image of a first object acquired by a camera and first point cloud data of the first object acquired by a radar;

[0007] Extracting features of the first point cloud data through a radar feature extraction network to obtain a first feature;

[0008] Extracting features of the first image through a camera feature extraction network to obtain second features;

[0009] Converting the first feature into a bird's-eye view representation space to obtain a first bird's-eye view feature corresponding to the first point cloud data;

[0010] converting the second feature into the bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image;

[0011] The radar feature extraction network and the camera feature extraction network are trained according to the first bird's-eye view features and the second bird's-eye view features.

[0012] Optionally, the training the radar feature extraction network and the camera feature extraction network according to the first bird's-eye view feature and the second bird's-eye view feature includes:

[0013] Calculating a target loss value according to the first bird's-eye view feature and the second bird's-eye view feature;

[0014] The radar feature extraction network and the camera feature extraction network are trained according to the target loss value.

[0015] Optionally, calculating the target loss value according to the first bird's-eye view feature and the second bird's-eye view feature includes:

[0016] Calculating a central core loss value according to the first bird's-eye view feature and the second bird's-eye view feature;

[0017] Calculating a contrastive learning loss value between the first bird's-eye view feature and the second bird's-eye view feature;

[0018] The target loss value includes the central kernel loss value and the contrastive learning loss value.

[0019] Optionally, calculating the target loss value according to the first bird's-eye view feature and the second bird's-eye view feature includes:

[0020] The target loss value loss is calculated by the following formula:

[0021]

[0022]

[0023]

[0024] Among them, loss1 is the center kernel loss value, L is the kernel function of the first bird's-eye view feature, K is the kernel function of the second bird's-eye view feature, HSIC is the Hilbert-Schmidt independence index, loss2 is the contrastive learning loss value, CE is the cross entropy loss function, L e For the first bird's-eye view feature, I e For the second bird's-eye view feature,

[0025] R=0,1,2,⋯,n; n is the batch size value of model training, t is the hyperparameter, and α is the weight coefficient.

[0026] Optionally, extracting the feature of the first point cloud data by using a radar feature extraction network to obtain the first feature includes:

[0027] generating a radar image based on the first point cloud data;

[0028] The features of the radar image are extracted through a radar feature extraction network to obtain the first features.

[0029] Optionally, converting the second feature into the bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image includes:

[0030] Converting the second feature to the bird's-eye view representation space through a feature conversion network to obtain a second bird's-eye view feature corresponding to the first image;

[0031] The training of the radar feature extraction network and the camera feature extraction network according to the first bird's-eye view feature and the second bird's-eye view feature includes:

[0032] The radar feature extraction network, the camera feature extraction network, and the feature conversion network are trained according to the first bird's-eye view features and the second bird's-eye view features.

[0033] According to a second aspect of an embodiment of the present disclosure, there is provided an object recognition method, including:

[0034] Acquire a second image of a second object acquired by the camera and second point cloud data of the second object acquired by the radar;

[0035] Extracting features of the second point cloud data through a radar feature extraction network to obtain a third feature;

[0036] Extracting features of the second image using a camera feature extraction network to obtain a fourth feature;

[0037] Inputting the third feature and the fourth feature into a recognition network to obtain a recognition result of the second object;

[0038] The radar feature extraction network and the camera feature extraction network are trained by the model training method described in any one of the first aspects above.

[0039] According to a third aspect of an embodiment of the present disclosure, a model training device is provided, comprising:

[0040] A first acquisition module is configured to acquire a first image acquired by a camera of a first object and first point cloud data acquired by a radar of the first object;

[0041] A first feature extraction module is configured to extract features of the first point cloud data through a radar feature extraction network to obtain a first feature;

[0042] A second feature extraction module is configured to extract features of the first image through a camera feature extraction network to obtain second features;

[0043] a first feature conversion module configured to convert the first feature into a bird's-eye view representation space to obtain a first bird's-eye view feature corresponding to the first point cloud data;

[0044] a second feature conversion module configured to convert the second feature into the bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image;

[0045] A training module is configured to train the radar feature extraction network and the camera feature extraction network according to the first bird's-eye view feature and the second bird's-eye view feature.

[0046] Optionally, the training module includes:

[0047] A first calculation submodule is configured to calculate a target loss value according to the first bird's-eye view feature and the second bird's-eye view feature;

[0048] The first training submodule is configured to train the radar feature extraction network and the camera feature extraction network according to the target loss value.

[0049] Optionally, the first calculation submodule includes:

[0050] a first calculation subunit, configured to calculate a central core loss value according to the first bird's-eye view feature and the second bird's-eye view feature;

[0051] a second calculation subunit, configured to calculate a contrastive learning loss value between the first bird's-eye view feature and the second bird's-eye view feature;

[0052] The target loss value includes the central kernel loss value and the contrastive learning loss value.

[0053] Optionally, the first calculation submodule is configured to calculate the target loss value loss by the following calculation formula:

[0054]

[0055]

[0056]

[0057] Among them, loss1 is the center kernel loss value, L is the kernel function of the first bird's-eye view feature, K is the kernel function of the second bird's-eye view feature, HSIC is the Hilbert-Schmidt independence index, loss2 is the contrastive learning loss value, CE is the cross entropy loss function, L e For the first bird's-eye view feature, I e For the second bird's-eye view feature,

[0058] R=0,1,2,⋯,n; n is the batch size value of model training, t is the hyperparameter, and α is the weight coefficient.

[0059] Optionally, the first feature extraction module includes:

[0060] a radar image generation submodule, configured to generate a radar image based on the first point cloud data;

[0061] The feature extraction submodule is configured to extract the features of the radar image through a radar feature extraction network to obtain the first feature.

[0062] Optionally, the second feature conversion module includes:

[0063] a feature conversion submodule, configured to convert the second feature into the bird's-eye view representation space through a feature conversion network to obtain a second bird's-eye view feature corresponding to the first image;

[0064] The training module includes:

[0065] The second training submodule is configured to train the radar feature extraction network, the camera feature extraction network, and the feature conversion network according to the first bird's-eye view feature and the second bird's-eye view feature.

[0066] According to a fourth aspect of an embodiment of the present disclosure, a model training device is provided, comprising:

[0067] processor;

[0068] a memory for storing processor-executable instructions;

[0069] The processor is configured to execute the steps of the model training method described in any one of the first aspects above.

[0070] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the model training method described in any one of the first aspects above are implemented.

[0071] According to a sixth aspect of an embodiment of the present disclosure, there is provided an object recognition device, including:

[0072] A second acquisition module is configured to acquire a second image acquired by the camera from a second object and second point cloud data acquired by the radar from the second object;

[0073] a third feature extraction module, configured to extract features of the second point cloud data through a radar feature extraction network to obtain a third feature;

[0074] a fourth feature extraction module, configured to extract features of the second image through a camera feature extraction network to obtain a fourth feature;

[0075] a first input module, configured to input the third feature and the fourth feature into a recognition network to obtain a recognition result of the second object;

[0076] The radar feature extraction network and the camera feature extraction network are trained by the model training method described in any one of the first aspects above.

[0077] According to a seventh aspect of an embodiment of the present disclosure, there is provided a vehicle, comprising:

[0078] processor;

[0079] a memory for storing processor-executable instructions;

[0080] Wherein, the processor is configured to:

[0081] Acquire a second image of a second object acquired by the camera and second point cloud data of the second object acquired by the radar;

[0082] Extracting features of the second point cloud data through a radar feature extraction network to obtain a third feature;

[0083] Extracting features of the second image using a camera feature extraction network to obtain a fourth feature;

[0084] Inputting the third feature and the fourth feature into a recognition network to obtain a recognition result of the second object;

[0085] The radar feature extraction network and the camera feature extraction network are trained by the model training method described in any one of the first aspects above.

[0086] According to an eighth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the method described in the second aspect are implemented.

[0087] In the above technical solution, the first feature of the first point cloud data can be extracted by the radar feature extraction network, and the second feature of the first image can be extracted by the camera feature extraction network. Afterwards, the first feature and the second feature can be converted to the bird's-eye view representation space to obtain the first bird's-eye view feature corresponding to the first point cloud data and the second bird's-eye view feature corresponding to the first image. Since the first point cloud data and the first image are collected for the same object, and the first bird's-eye view feature and the second bird's-eye view feature are in the same representation space. Therefore, the first bird's-eye view feature and the second bird's-eye view feature can be labels for each other, thereby realizing unsupervised training. In this way, the unlabeled data collected by the camera and radar can be used for model training, thereby reducing the dependence of the model training process on labeled samples.

[0088] Furthermore, during training, the radar and camera feature extraction networks can exchange model information, enabling cross-modal joint training of the two networks. This approach helps improve the feature extraction capabilities of both networks and enhances model training results.

[0089] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0091] Figure 1 The figure is a flowchart of a model training method according to an exemplary embodiment.

[0092] Figure 2 The figure is a flowchart of obtaining a first feature according to an exemplary embodiment.

[0093] Figure 3 The figure is a flowchart of a model training method according to an exemplary embodiment.

[0094] Figure 4 The figure is a flowchart showing a calculation flow of a target loss value according to an exemplary embodiment.

[0095] Figure 5 The figure is a flowchart of an object recognition method according to an exemplary embodiment.

[0096] Figure 6 is a block diagram of a model training device 600 according to an exemplary embodiment.

[0097] Figure 7 is a block diagram of an object recognition device 700 according to an exemplary embodiment.

[0098] Figure 8 It is a block diagram of a device 800 for model training according to an exemplary embodiment.

[0099] Figure 9 It is a functional block diagram of a vehicle 900 according to an exemplary embodiment. DETAILED DESCRIPTION

[0100] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0101] Before introducing the model training method, object recognition method, device, vehicle, and storage medium disclosed herein, we first introduce the application scenarios of the present disclosure. In fields such as autonomous driving, trained models can be used to implement a variety of functions. For example, an image perception model and a laser perception model can be trained to perceive and recognize the environment around the vehicle, and the outputs of the two models can be combined to obtain the final perception results. In this way, the perception results can be used to assist in vehicle control decisions.

[0102] It's worth noting that this training method requires a large amount of labeled data. With less labeled data, it can be difficult to achieve effective model training results, resulting in poor model generalization. Furthermore, because the image perception model and laser perception model are trained independently, each model has its own flaws. Therefore, the trained model may exhibit poor performance. For example, a laser perception model trained in this way may struggle to accurately extract the semantic information of a scene, while an image perception model may struggle to accurately extract the spatial geometry of objects in the scene.

[0103] To this end, the present disclosure provides a model training method. Figure 1 This is a flow chart of a model training method shown in an exemplary embodiment of the present disclosure. The method can be applied to various stationary or mobile computing devices, such as servers, desktop computers, laptop computers, etc. Figure 1 As shown, the method includes the following steps.

[0104] In step S11 , a first image of a first object acquired by a camera and first point cloud data of the first object acquired by a radar are acquired.

[0105] For example, the camera is a vehicle-mounted camera, and the radar is a vehicle-mounted lidar. In this way, the camera and radar can be used to collect data about the vehicle's environment while the vehicle is driving, and the image data and point cloud data collected at the same time can be used as a set of matching data pairs. During training, a portion of the collected data pairs can be obtained, and the image data in the data pairs can be used as the first image, and the point cloud data in the data pairs can be used as the first point cloud data.

[0106] It should be noted that the number of cameras can be one or more, and this disclosure does not limit this.

[0107] In step S12, features of the first point cloud data are extracted through a radar feature extraction network to obtain a first feature.

[0108] Here, in a possible implementation, the first point cloud data may be processed using a pointnet architecture to obtain point cloud features. In this case, the first features may include the point cloud features.

[0109] In one possible implementation, refer to Figure 2 In the flowchart of obtaining the first feature shown, step S12 may also include:

[0110] S121, generating a radar image based on the first point cloud data;

[0111] S122: Extract features of the radar image through a radar feature extraction network to obtain a first feature.

[0112] Among them, the radar feature extraction network can be, for example, a backbone network (such as a residual network) obtained by pre-training with radar image samples, and the training method can be supervised training or self-supervised training, such as the MAE (Masked Autoencoders) training method.

[0113] As an example, when obtaining the first feature, the first point cloud data can be voxelized to convert the first point cloud data into a radar image. In this way, the converted radar image can be input into a radar feature extraction network to obtain the first feature.

[0114] In step S13, features of the first image are extracted through a camera feature extraction network to obtain second features.

[0115] Among them, the camera feature extraction network can be, for example, a backbone network (such as a residual network) obtained by pre-training with camera image samples. The training method can be supervised training or self-supervised training (such as the MAE method), and this disclosure does not impose any restrictions on this.

[0116] The process for obtaining the second feature is described using a 2D image as an example. In some embodiments, the 2D image can be input into a camera feature extraction network, and the second feature output by the camera feature extraction network is: b*c*h*w. Here, b is the batch size of the first image during model training, c is the number of image channels, h is the image height, and w is the image width.

[0117] In addition, when there are multiple cameras, the images captured by each camera can be used as the first image, and the camera feature extraction network can be used to extract image features to obtain the second feature: b*N*c*h*w, where N is the number of cameras.

[0118] In step S14 , the first feature is converted into a bird's-eye view representation space to obtain a first bird's-eye view feature corresponding to the first point cloud data.

[0119] For example, the first feature may be converted to a bird's eye view representation space (BEV) by embedding, and the converted vector feature is used as the first bird's eye view feature.

[0120] In step S15 , the second feature is converted into a bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image.

[0121] In some embodiments, depth estimation of the image feature may be performed on the second feature, thereby converting the second feature into a bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image.

[0122] In some embodiments, the second feature may also be transformed through deep learning. For example, the second feature may be transformed into the bird's-eye view representation space using a feature transformation network to obtain a second bird's-eye view feature corresponding to the first image. Here, the feature transformation network may include, for example, a transform network architecture, which is not limited in this disclosure.

[0123] In step S16, the radar feature extraction network and the camera feature extraction network are trained according to the first bird's-eye view feature and the second bird's-eye view feature.

[0124] In a possible implementation, converting the second feature into the bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image includes:

[0125] Converting the second feature to the bird's-eye view representation space through a feature conversion network to obtain a second bird's-eye view feature corresponding to the first image;

[0126] In this case, the training of the radar feature extraction network and the camera feature extraction network according to the first bird's-eye view feature and the second bird's-eye view feature (step S16) includes:

[0127] The radar feature extraction network, the camera feature extraction network, and the feature conversion network are trained according to the first bird's-eye view features and the second bird's-eye view features.

[0128] In the above technical solution, the first feature of the first point cloud data can be extracted by the radar feature extraction network, and the second feature of the first image can be extracted by the camera feature extraction network. Afterwards, the first feature and the second feature can be converted to the bird's-eye view representation space to obtain the first bird's-eye view feature corresponding to the first point cloud data and the second bird's-eye view feature corresponding to the first image. Since the first point cloud data and the first image are collected for the same object, and the first bird's-eye view feature and the second bird's-eye view feature are in the same representation space. Therefore, the first bird's-eye view feature and the second bird's-eye view feature can be labels for each other, thereby realizing unsupervised training. In this way, the unlabeled data collected by the camera and radar can be used for model training, thereby reducing the dependence of the model training process on labeled samples.

[0129] Furthermore, during training, the radar and camera feature extraction networks can exchange model information, enabling cross-modal joint training of the two networks. This approach helps improve the feature extraction capabilities of both networks and enhances model training results.

[0130] It is worth noting that the above embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit the same. Those skilled in the art will appreciate that, in specific implementations, there is no particular order in which the first and second features are acquired, and there is no particular order in which the first and second bird's-eye view features are acquired. This disclosure does not impose any limitations on this.

[0131] Figure 3 This is a flow chart of a model training method shown in an exemplary embodiment of the present disclosure, with reference to Figure 3 , the method comprising:

[0132] In step S11 , a first image of a first object acquired by a camera and first point cloud data of the first object acquired by a radar are acquired.

[0133] In step S12, features of the first point cloud data are extracted through a radar feature extraction network to obtain a first feature.

[0134] In step S13, features of the first image are extracted through a camera feature extraction network to obtain second features.

[0135] In step S14 , the first feature is converted into a bird's-eye view representation space to obtain a first bird's-eye view feature corresponding to the first point cloud data.

[0136] In step S15 , the second feature is converted into a bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image.

[0137] In step S161 , a target loss value is calculated according to the first bird's-eye view feature and the second bird's-eye view feature.

[0138] Since the first point cloud data and the first image are collected for the same object, and the first bird's-eye view feature and the second bird's-eye view feature are in the same representation space, the first bird's-eye view feature and the second bird's-eye view feature can be labels for each other, thereby calculating the loss value. Figure 4 This is a flow chart showing a target loss value calculation according to an exemplary embodiment of the present disclosure. Figure 4 , step S161 may include:

[0139] S41: Calculate a central core loss value according to the first bird's-eye view feature and the second bird's-eye view feature.

[0140] As an example, the central kernel loss value loss1 can be calculated using the following formula:

[0141]

[0142] HSIC (Hilbert-Schmidt Independence Criterion) is a Hilbert-Schmidt independence criterion that measures the independence between two variables. HSIC functions similarly to mutual information, but instead of estimating the probability density of the two variables, it directly converts the two variables into a sampling form to calculate the correlation between the two variables. L is the kernel function for the first bird's-eye view feature, and K is the kernel function for the second bird's-eye view feature. For example, kernel functions L and K can be linear kernel functions.

[0143] S42: Calculate a contrastive learning loss value between the first bird's-eye view feature and the second bird's-eye view feature, where the target loss value includes a central kernel loss value and a contrastive learning loss value.

[0144] As an example, the contrastive learning loss loss2 can be calculated as follows:

[0145]

[0146] Among them, CE is the cross entropy loss function, L e For the first bird's-eye view feature, I e For the second bird's-eye view feature,

[0147] R=0,1,2,⋯,n; n is the batch size value of model training, and t is a hyperparameter.

[0148] In this way, the target loss value loss can be expressed as:

[0149]

[0150] Wherein, α is a weight coefficient. In a specific implementation, the value of α can be set to 0.5, for example.

[0151] In other words, the target loss can be composed of two components: the central kernel loss and the contrastive learning loss. This approach helps the network learn the semantic information of features, thereby improving the radar feature extraction network's ability to extract scene semantic information. Furthermore, the contrastive learning loss helps the network learn the spatial geometric relationships of objects, thereby improving the image feature extraction network's ability to extract spatial geometric relationship features.

[0152] It is worth noting that in some possible implementation scenarios, a weight coefficient can also be set for the target loss value and / or the comparative learning loss value. The weight coefficient is used to adjust the proportion of the central kernel loss value and the comparative learning loss value in the target loss value. So that the difference between the first proportion of the central kernel loss value and the second proportion of the comparative learning loss value is less than a preset threshold, or so that the first proportion is equal to the second proportion. The first proportion is the ratio of the central kernel loss value to the target loss value; the second proportion is the ratio of the comparative learning loss value to the target loss value. In this way, the training focus of the model can be adjusted.

[0153] In step S162, the radar feature extraction network and the camera feature extraction network are trained according to the target loss value.

[0154] Using the above technical solution, the first bird's-eye view feature and the second bird's-eye view feature can be labeled with each other, thereby achieving unsupervised training. In this way, the model can be trained using unlabeled data collected by the camera and radar, thereby reducing the model training process's reliance on labeled samples. In addition, during the training process, the radar feature extraction network and the camera feature extraction network can transfer model information to each other, thereby achieving cross-modal joint training between the radar feature extraction network and the camera feature extraction network. Using this approach helps to improve the feature extraction capabilities of the radar feature extraction network and the camera feature extraction network, and improve the training effect.

[0155] The present disclosure also provides an object recognition method. Figure 5 This is a flow chart of an object recognition method shown in the present disclosure, referring to Figure 5 , the method comprising:

[0156] In step S51 , a second image of the second object acquired by the camera and second point cloud data of the second object acquired by the radar are acquired.

[0157] For example, the camera is a vehicle-mounted camera, and the radar is a vehicle-mounted lidar. In this way, the camera and radar can collect data about the vehicle's environment while the vehicle is driving, and the image data and point cloud data collected at the same time are used as a set of matching data pairs. During object recognition, the image data in the data pair to be recognized can be used as the second image, and the point cloud data in the data pair can be used as the second point cloud data.

[0158] In step S52, features of the second point cloud data are extracted through a radar feature extraction network to obtain third features.

[0159] In step S53, features of the second image are extracted through a camera feature extraction network to obtain a fourth feature.

[0160] In step S54 , the third feature and the fourth feature are input into the recognition network to obtain a recognition result of the second object.

[0161] The radar feature extraction network and the camera feature extraction network are trained using the model training method provided in this disclosure. The recognition network can be, for example, a classification network, a detection network, etc. For details, please refer to the description in the relevant art, which is not elaborated in this disclosure.

[0162] The above technical solution can utilize unlabeled data collected by cameras and radars for model training, thereby reducing the model training process's reliance on labeled samples. Furthermore, during training, the radar feature extraction network and the camera feature extraction network can transfer model information to each other, enabling cross-modal joint training of the radar and camera feature extraction networks. This approach helps improve the feature extraction capabilities of the radar and camera feature extraction networks, thereby helping to improve the accuracy of the features input to the recognition network, ultimately improving object recognition accuracy.

[0163] The present disclosure also provides a model training device. Figure 6 is a block diagram of a model training device 600 shown in an exemplary embodiment of the present disclosure, such as Figure 6 As shown, the model training device 600 includes:

[0164] A first acquisition module 601 is configured to acquire a first image acquired by a camera of a first object and first point cloud data acquired by a radar of the first object;

[0165] A first feature extraction module 602 is configured to extract features of the first point cloud data through a radar feature extraction network to obtain a first feature;

[0166] A second feature extraction module 603 is configured to extract features of the first image through a camera feature extraction network to obtain second features;

[0167] A first feature conversion module 604 is configured to convert the first feature into a bird's-eye view representation space to obtain a first bird's-eye view feature corresponding to the first point cloud data;

[0168] A second feature conversion module 605 is configured to convert the second feature into the bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image;

[0169] The training module 606 is configured to train the radar feature extraction network and the camera feature extraction network according to the first bird's-eye view feature and the second bird's-eye view feature.

[0170] In the above technical solution, the first feature of the first point cloud data can be extracted by the radar feature extraction network, and the second feature of the first image can be extracted by the camera feature extraction network. Afterwards, the first feature and the second feature can be converted to the bird's-eye view representation space to obtain the first bird's-eye view feature corresponding to the first point cloud data and the second bird's-eye view feature corresponding to the first image. Since the first point cloud data and the first image are collected for the same object, and the first bird's-eye view feature and the second bird's-eye view feature are in the same representation space. Therefore, the first bird's-eye view feature and the second bird's-eye view feature can be labels for each other, thereby realizing unsupervised training. In this way, the unlabeled data collected by the camera and radar can be used for model training, thereby reducing the dependence of the model training process on labeled samples.

[0171] Furthermore, during training, the radar feature extraction network and the camera feature extraction network can transfer model information to each other, enabling cross-modal joint training of the two networks. This approach helps improve the feature extraction capabilities of both networks, enhancing training effectiveness.

[0172] Optionally, the training module 606 includes:

[0173] A first calculation submodule is configured to calculate a target loss value according to the first bird's-eye view feature and the second bird's-eye view feature;

[0174] The first training submodule is configured to train the radar feature extraction network and the camera feature extraction network according to the target loss value.

[0175] Optionally, the first calculation submodule includes:

[0176] a first calculation subunit, configured to calculate a central core loss value according to the first bird's-eye view feature and the second bird's-eye view feature;

[0177] a second calculation subunit, configured to calculate a contrastive learning loss value between the first bird's-eye view feature and the second bird's-eye view feature;

[0178] The target loss value includes the central kernel loss value and the contrastive learning loss value.

[0179] Optionally, the first calculation submodule is configured to calculate the target loss value loss by the following calculation formula:

[0180]

[0181]

[0182]

[0183] Among them, loss1 is the center kernel loss value, L is the kernel function of the first bird's-eye view feature, K is the kernel function of the second bird's-eye view feature, HSIC is the Hilbert-Schmidt independence index, loss2 is the contrastive learning loss value, CE is the cross entropy loss function, L e For the first bird's-eye view feature, I e For the second bird's-eye view feature,

[0184] R=0,1,2,⋯,n; n is the batch size value of model training, t is the hyperparameter, and α is the weight coefficient.

[0185] Optionally, the first feature extraction module 602 includes:

[0186] a radar image generation submodule, configured to generate a radar image based on the first point cloud data;

[0187] The feature extraction submodule is configured to extract the features of the radar image through a radar feature extraction network to obtain the first feature.

[0188] Optionally, the second feature conversion module 605 includes:

[0189] a feature conversion submodule, configured to convert the second feature into the bird's-eye view representation space through a feature conversion network to obtain a second bird's-eye view feature corresponding to the first image;

[0190] The training module 606 includes:

[0191] The second training submodule is configured to train the radar feature extraction network, the camera feature extraction network, and the feature conversion network according to the first bird's-eye view feature and the second bird's-eye view feature.

[0192] The present disclosure also provides a model training device, comprising:

[0193] processor;

[0194] a memory for storing processor-executable instructions;

[0195] The processor is configured to execute the steps of the model training method provided in the present disclosure.

[0196] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the model training method provided by the present disclosure.

[0197] Figure 7 is a block diagram of an object recognition device 700 shown in an exemplary embodiment of the present disclosure, with reference to Figure 7 , the object recognition device 700 includes:

[0198] A second acquisition module 701 is configured to acquire a second image acquired by a camera of a second object and second point cloud data acquired by a radar of the second object;

[0199] A third feature extraction module 702 is configured to extract features of the second point cloud data through a radar feature extraction network to obtain a third feature;

[0200] A fourth feature extraction module 703 is configured to extract features of the second image through a camera feature extraction network to obtain a fourth feature;

[0201] A first input module 704 is configured to input the third feature and the fourth feature into a recognition network to obtain a recognition result of the second object;

[0202] The radar feature extraction network and the camera feature extraction network are trained by the model training method provided in the present disclosure.

[0203] The above technical solution can utilize unlabeled data collected by cameras and radars for model training, thereby reducing the model training process's reliance on labeled samples. Furthermore, during training, the radar feature extraction network and the camera feature extraction network can transfer model information to each other, enabling cross-modal joint training of the radar and camera feature extraction networks. This approach helps improve the feature extraction capabilities of the radar and camera feature extraction networks, thereby helping to improve the accuracy of the features input to the recognition network, ultimately improving object recognition accuracy.

[0204] The present disclosure also provides a vehicle, comprising:

[0205] processor;

[0206] a memory for storing processor-executable instructions;

[0207] Wherein, the processor is configured to:

[0208] Acquire a second image of a second object acquired by the camera and second point cloud data of the second object acquired by the radar;

[0209] Extracting features of the second point cloud data through a radar feature extraction network to obtain a third feature;

[0210] Extracting features of the second image using a camera feature extraction network to obtain a fourth feature;

[0211] Inputting the third feature and the fourth feature into a recognition network to obtain a recognition result of the second object;

[0212] The radar feature extraction network and the camera feature extraction network are trained by the model training method provided in the present disclosure.

[0213] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon. When the program instructions are executed by a processor, the steps of the object recognition method provided by the present disclosure are implemented.

[0214] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0215] Figure 8 FIG1 is a block diagram of an apparatus 800 for model training according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, etc.

[0216] Reference Figure 8The apparatus 800 may include one or more of the following components: a processing component 802 , a first memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output interface 812 , a sensor component 814 , and a communication component 816 .

[0217] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more first processors 820 to execute instructions to complete all or part of the steps of the above-mentioned model training method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0218] The first memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, training data, messages, pictures, videos, etc. The first memory 804 can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0219] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 800.

[0220] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a capture mode or a video mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and the rear-facing camera can have a fixed optical lens system or have focal length and optical zoom capabilities.

[0221] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the device 800 is in an operating mode, such as a call mode, a recording mode, or a voice recognition mode. The received audio signals may be further stored in the first memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0222] The input / output interface 812 provides an interface between the processing component 802 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a volume button, a start button, and a lock button.

[0223] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800. The sensor assembly 814 can also detect changes in the position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and temperature changes of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0224] The communication component 816 is configured to facilitate wired or wireless communication between the apparatus 800 and other devices. The apparatus 800 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0225] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described model training method.

[0226] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a first memory 804 including instructions. The instructions can be executed by the first processor 820 of the apparatus 800 to perform the above-described model training method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0227] See Figure 9 , Figure 9 FIG2 is a functional block diagram of a vehicle 900, illustrating an exemplary embodiment. Vehicle 900 can be configured for fully or partially autonomous driving. For example, vehicle 900 can obtain environmental information about its surroundings through perception system 920 and, based on analysis of the environmental information, derive an autonomous driving strategy to achieve fully autonomous driving, or present the analysis results to the user to achieve partially autonomous driving.

[0228] Vehicle 900 may include various subsystems, such as an infotainment system 910, a perception system 920, a decision control system 930, a drive system 940, and a computing platform 950. Alternatively, vehicle 900 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of vehicle 900 may be interconnected via wired or wireless means.

[0229] In some embodiments, infotainment system 910 may include a communication system 911 , an entertainment system 912 , and a navigation system 913 .

[0230] The communication system 911 may include a wireless communication system that can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system may use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE. Or 5G cellular communication. The wireless communication system may use WiFi to communicate with a wireless local area network (WLAN). In some embodiments, the wireless communication system may use an infrared link, Bluetooth, or ZigBee to communicate directly with the device. Other wireless protocols, such as various vehicle communication systems, for example, the wireless communication system may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0231] The entertainment system 912 may include a display device, a microphone and speakers. Users can listen to the radio and play music in the car based on the entertainment system; or connect the mobile phone to the vehicle and project the mobile phone screen on the display device. The display device can be touch-sensitive and the user can operate it by touching the screen.

[0232] In some cases, the user's voice signal can be obtained through a microphone, and based on the analysis of the user's voice signal, the user can control certain aspects of the vehicle 900, such as adjusting the temperature inside the vehicle, etc. In other cases, music can be played to the user through a speaker.

[0233] Navigation system 913 may include map services provided by a map provider, thereby providing navigation for vehicle 900. Navigation system 913 may be used in conjunction with the vehicle's global positioning system 921 and inertial measurement unit 922. The map services provided by the map provider may be two-dimensional maps or high-precision maps.

[0234] The perception system 920 may include several sensors that sense information about the environment surrounding the vehicle 900. For example, the perception system 920 may include a global positioning system 921 (the global positioning system may be a GPS system, a BeiDou system, or other positioning systems), an inertial measurement unit (IMU) 922, a lidar 923, a millimeter-wave radar 924, an ultrasonic radar 925, and a camera 926. The perception system 920 may also include sensors for the internal systems of the monitored vehicle 900 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). This detection and recognition is a key function for the safe operation of the vehicle 900.

[0235] The global positioning system 921 is used to estimate the geographic location of the vehicle 900 .

[0236] The inertial measurement unit 922 is used to sense the posture change of the vehicle 900 based on inertial acceleration. In some embodiments, the inertial measurement unit 922 can be a combination of an accelerometer and a gyroscope.

[0237] LiDAR 923 utilizes laser light to sense objects in the environment in which vehicle 900 is located. In some embodiments, LiDAR 923 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components.

[0238] The millimeter wave radar 924 uses radio signals to sense objects in the surrounding environment of the vehicle 900. In some embodiments, in addition to sensing objects, the millimeter wave radar 924 can also be used to sense the speed and / or heading of the objects.

[0239] The ultrasonic radar 925 may sense objects around the vehicle 900 using ultrasonic signals.

[0240] The camera device 926 is used to capture image information of the surrounding environment of the vehicle 900. The camera device 926 may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, etc. The image information obtained by the camera device 926 may include static images or video stream information.

[0241] The decision control system 930 includes a computing system 931 that analyzes and makes decisions based on the information obtained by the perception system 920. The decision control system 930 also includes a vehicle controller 932 that controls the power system of the vehicle 900, as well as a steering system 933, throttle 934 and braking system 935 for controlling the vehicle 900.

[0242] The computing system 931 can be operated to process and analyze various information obtained by the perception system 920 in order to identify targets, objects, and / or features in the environment surrounding the vehicle 900. Targets may include pedestrians or animals, and objects and / or features may include traffic signals, road boundaries, and obstacles. The computing system 931 may use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other technologies. In some embodiments, the computing system 931 can be used to map the environment, track objects, estimate the speed of objects, and so on. The computing system 931 can analyze the various information obtained and derive a control strategy for the vehicle.

[0243] The vehicle controller 932 can be used to coordinate and control the vehicle's power battery and engine 941 to improve the power performance of the vehicle 900.

[0244] The steering system 933 is operable to adjust the forward direction of the vehicle 900. For example, in one embodiment, it can be a steering wheel system.

[0245] The throttle 934 is used to control the operating speed of the engine 941 and thus the speed of the vehicle 900 .

[0246] Braking system 935 is used to control the deceleration of vehicle 900. Braking system 935 can use friction to slow down wheels 944. In some embodiments, braking system 935 can convert the kinetic energy of wheels 944 into electric current. Braking system 935 can also take other forms to slow the rotation speed of wheels 944 and thus control the speed of vehicle 900.

[0247] Drive system 940 may include components that provide powered motion for vehicle 900. In one embodiment, drive system 940 may include an engine 941, an energy source 942, a transmission system 943, and wheels 944. Engine 941 may be an internal combustion engine, an electric motor, an air compression engine, or another combination of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine. Engine 941 converts energy source 942 into mechanical energy.

[0248] Examples of energy source 942 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 942 can also provide energy to other systems of vehicle 900.

[0249] The transmission system 943 can transmit mechanical power from the engine 941 to the wheels 944. The transmission system 943 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission system 943 may also include other components, such as a clutch. The drive shaft may include one or more shafts that can be coupled to one or more wheels 944.

[0250] Some or all functions of the vehicle 900 are controlled by a computing platform 950. The computing platform 950 may include at least one second processor 951, which may execute instructions 953 stored in a non-transitory computer-readable medium such as a second memory 952. In some embodiments, the computing platform 950 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 900 in a distributed manner.

[0251] The second processor 951 may be any conventional processor, such as a commercially available CPU. Alternatively, the second processor 951 may also include a graphics processor (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof. Figure 9 Functionally illustrate the second processor 951, the second memory 952 and other elements of the computer in the same block, but those skilled in the art will appreciate that the second processor 951, the computer or the second memory 952 can actually include multiple processors, computers or memories that may or may not be stored in the same physical housing. For example, the second memory 952 can be a hard drive or other storage medium that is located in a housing that is different from the computer. Therefore, reference to the second processor 951 or the computer will be understood to include references to a collection of processors or computers or memories that may or may not operate in parallel. Different from using a single processor to perform the steps described herein, some assemblies such as the steering assembly and the deceleration assembly can each have their own processor that only performs the calculations relevant to the functions of the assembly.

[0252] In an embodiment of the present disclosure, the second processor 951 may execute the above-mentioned object recognition method.

[0253] In various aspects described herein, the second processor 951 can be located remotely from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle while others are performed by a remote processor, including taking the necessary steps to perform a single maneuver.

[0254] In some embodiments, the second memory 952 may contain instructions 953 (e.g., program logic) that may be executed by the second processor 951 to perform various functions of the vehicle 900. The second memory 952 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the infotainment system 910, the perception system 920, the decision control system 930, and the drive system 940.

[0255] In addition to instructions 953, second memory 952 may also store data such as road maps, route information, the vehicle's location, direction, speed, and other such vehicle data, as well as other information. This information may be used by vehicle 900 and computing platform 950 during operation of vehicle 900 in autonomous, semi-autonomous, and / or manual modes.

[0256] The computing platform 950 may control functions of the vehicle 900 based on input received from various subsystems, such as the drive system 940, the perception system 920, and the decision control system 930. For example, the computing platform 950 may utilize input from the decision control system 930 to control the steering system 933 to avoid an obstacle detected by the perception system 920. In some embodiments, the computing platform 950 may be operable to provide control over many aspects of the vehicle 900 and its subsystems.

[0257] Alternatively, one or more of the above components may be installed or associated separately from the vehicle 900. For example, the second memory 952 may be partially or completely separate from the vehicle 900. The above components may be communicatively coupled together in a wired and / or wireless manner.

[0258] Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 9 It should not be understood as limiting the embodiments of the present disclosure.

[0259] An autonomous vehicle traveling on a road, such as vehicle 900 above, can identify objects in its surroundings to determine adjustments to its current speed. Objects can be other vehicles, traffic control devices, or other types of objects. In some examples, each identified object can be considered independently, and the speed adjustment to be made to the autonomous vehicle can be determined based on its respective characteristics, such as its current speed, acceleration, and distance from the vehicle.

[0260] Optionally, vehicle 900 or sensing and computing devices associated with vehicle 900 (e.g., computing system 931, computing platform 950) can predict the behavior of identified objects based on their characteristics and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, because the behavior of each identified object depends on the behavior of others, the behavior of all identified objects can be considered together to predict the behavior of a single identified object. Vehicle 900 can adjust its speed based on the predicted behavior of the identified objects. In other words, the autonomous vehicle can determine the desired stable state (e.g., acceleration, deceleration, or stop) based on the predicted behavior of the objects. Other factors can also be considered in determining the speed of vehicle 900, such as the lateral position of vehicle 900 on the road, the curvature of the road, the proximity of static and dynamic objects, and so on.

[0261] In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device may also provide instructions to modify the steering angle of vehicle 900 so that the autonomous vehicle follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the autonomous vehicle (e.g., vehicles in adjacent lanes on the road).

[0262] The vehicle 900 may be any type of vehicle, such as a car, a truck, a motorcycle, a bus, etc., and is not particularly limited in the embodiment of the present disclosure.

[0263] In another exemplary embodiment, a computer program product is also provided, which includes a computer program that can be executed by a programmable device, and the computer program has a code portion for performing the above-mentioned model training method when executed by the programmable device.

[0264] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device, and has a code portion for performing the above-mentioned object recognition method when executed by the programmable device.

[0265] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0266] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A model training method, characterized in that: include: Acquire a first image of a first object acquired by a camera and first point cloud data of the first object acquired by a radar; Extracting features of the first point cloud data through a radar feature extraction network to obtain a first feature; Extracting features of the first image through a camera feature extraction network to obtain second features; Converting the first feature into a bird's-eye view representation space to obtain a first bird's-eye view feature corresponding to the first point cloud data; converting the second feature into the bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image; Training the radar feature extraction network and the camera feature extraction network according to the first bird's-eye view feature and the second bird's-eye view feature includes: A target loss value is calculated based on the first bird's-eye view feature and the second bird's-eye view feature. The target loss value includes a central kernel loss for measuring the correlation between the first bird's-eye view feature and the second bird's-eye view feature, so that the first bird's-eye view feature and the second bird's-eye view feature are labeled with each other for unsupervised training.

2. The method according to claim 1, characterized in that The training of the radar feature extraction network and the camera feature extraction network according to the first bird's-eye view feature and the second bird's-eye view feature includes: The radar feature extraction network and the camera feature extraction network are trained according to the target loss value.

3. The method according to claim 2, characterized in that The calculating the target loss value according to the first bird's-eye view feature and the second bird's-eye view feature includes: Calculating a contrastive learning loss value between the first bird's-eye view feature and the second bird's-eye view feature; The target loss value includes the central kernel loss value and the contrastive learning loss value.

4. The method according to claim 1, wherein The calculating the target loss value according to the first bird's-eye view feature and the second bird's-eye view feature includes: The target loss value loss is calculated by the following formula: Among them, loss1 is the center kernel loss value, L is the kernel function of the first bird's-eye view feature, K is the kernel function of the second bird's-eye view feature, HSIC is the Hilbert-Schmidt independence index, loss2 is the contrastive learning loss value, CE is the cross entropy loss function, L e For the first bird's-eye view feature, I e For the second bird's-eye view feature, R=0,1,2,⋯,n; n is the batch size value of model training, t is the hyperparameter, and α is the weight coefficient.

5. The method according to any one of claims 1 to 4, characterized in that The step of extracting the feature of the first point cloud data by using a radar feature extraction network to obtain the first feature includes: generating a radar image based on the first point cloud data; The features of the radar image are extracted through a radar feature extraction network to obtain the first features.

6. The method according to any one of claims 1 to 4, characterized in that The converting the second feature into the bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image includes: Converting the second feature to the bird's-eye view representation space through a feature conversion network to obtain a second bird's-eye view feature corresponding to the first image; The training of the radar feature extraction network and the camera feature extraction network according to the first bird's-eye view feature and the second bird's-eye view feature includes: The radar feature extraction network, the camera feature extraction network, and the feature conversion network are trained according to the first bird's-eye view features and the second bird's-eye view features.

7. A method for object recognition, characterized in that: include: Acquire a second image of a second object acquired by the camera and second point cloud data of the second object acquired by the radar; Extracting features of the second point cloud data through a radar feature extraction network to obtain a third feature; Extracting features of the second image using a camera feature extraction network to obtain a fourth feature; Inputting the third feature and the fourth feature into a recognition network to obtain a recognition result of the second object; The radar feature extraction network and the camera feature extraction network are trained by the model training method according to any one of claims 1 to 6.

8. A model training device, characterized in that: include: A first acquisition module is configured to acquire a first image acquired by a camera of a first object and first point cloud data acquired by a radar of the first object; A first feature extraction module is configured to extract features of the first point cloud data through a radar feature extraction network to obtain a first feature; A second feature extraction module is configured to extract features of the first image through a camera feature extraction network to obtain second features; a first feature conversion module configured to convert the first feature into a bird's-eye view representation space to obtain a first bird's-eye view feature corresponding to the first point cloud data; a second feature conversion module configured to convert the second feature into the bird's-eye view representation space to obtain a second bird's-eye view feature corresponding to the first image; A training module is configured to train the radar feature extraction network and the camera feature extraction network according to the first bird's-eye view feature and the second bird's-eye view feature, comprising: A target loss value is calculated based on the first bird's-eye view feature and the second bird's-eye view feature. The target loss value includes a central kernel loss for measuring the correlation between the first bird's-eye view feature and the second bird's-eye view feature, so that the first bird's-eye view feature and the second bird's-eye view feature are labeled with each other for unsupervised training.

9. A model training device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

11. An object recognition device, characterized in that: include: A second acquisition module is configured to acquire a second image acquired by the camera from a second object and second point cloud data acquired by the radar from the second object; a third feature extraction module, configured to extract features of the second point cloud data through a radar feature extraction network to obtain a third feature; a fourth feature extraction module, configured to extract features of the second image through a camera feature extraction network to obtain a fourth feature; a first input module, configured to input the third feature and the fourth feature into a recognition network to obtain a recognition result of the second object; The radar feature extraction network and the camera feature extraction network are trained by the model training method according to any one of claims 1 to 6.

12. A vehicle, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Acquire a second image of a second object acquired by the camera and second point cloud data of the second object acquired by the radar; Extracting features of the second point cloud data through a radar feature extraction network to obtain a third feature; Extracting features of the second image using a camera feature extraction network to obtain a fourth feature; Inputting the third feature and the fourth feature into a recognition network to obtain a recognition result of the second object; The radar feature extraction network and the camera feature extraction network are trained by the model training method according to any one of claims 1 to 6.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in claim 7 are implemented.

Images