3D object recognition method and 3D object recognition apparatus

Abstract

The application provides a 3D target recognition method and device, and the method comprises the following steps: acquiring a to-be-detected road end image, wherein the to-be-detected road end image is a to-be-detected image collected by an image collector arranged at a road end; inputting the to-be-detected road end image into a pre-trained 3D target detection model to obtain a 3D target detection result corresponding to the to-be-detected road end image output by the 3D target detection model, wherein the 3D target detection model is trained by combining road end image training samples and vehicle end image training samples with cross-view contrast learning. The 3D target recognition cost is reduced under the premise of ensuring the accuracy of 3D target recognition.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a 3D target recognition method and a 3D target recognition device. Background Technology

[0002] With technological advancements, autonomous driving is becoming a trend. However, to ensure the safety of autonomous driving processes, effective and accurate 3D target recognition of roadside images is necessary.

[0003] According to relevant technologies, current 3D target recognition in roadside images requires training a detection model using a large number of labeled roadside image samples, and then using the trained model to achieve 3D target recognition. However, obtaining a large number of labeled roadside image samples is costly, resulting in high costs for accurately identifying 3D targets.

[0004] Therefore, finding a 3D target recognition method that reduces the cost of 3D target recognition while ensuring the accuracy of 3D target recognition has become a research hotspot. Summary of the Invention

[0005] This invention provides a 3D target recognition method and a 3D target recognition device, which reduces the cost of 3D target recognition while ensuring the accuracy of 3D target recognition.

[0006] This invention provides a 3D target recognition method, the method comprising: acquiring a roadside image to be detected, wherein the roadside image to be detected is an image to be detected acquired by an image acquisition device set at the roadside; inputting the roadside image to be detected into a pre-trained 3D target detection model to obtain a 3D target detection result output by the 3D target detection model corresponding to the roadside image to be detected, wherein the 3D target detection model is trained by combining roadside image training samples and vehicle-side image training samples through cross-view comparative learning.

[0007] According to a 3D target recognition method provided by the present invention, the 3D target detection model is trained in the following manner: Based on multiple roadside image training samples, a first loss function and a first foreground depth map loss function are constructed in the roadside image domain; based on multiple vehicle-side image training samples, a second loss function and a second foreground depth map loss function are constructed in the vehicle-side image domain; based on multiple vehicle-side image training samples and multiple roadside image training samples, a cross-view comparison learning loss function is constructed; based on the first loss function, the second loss function, the cross-view comparison learning loss function, the first foreground depth map loss function, and the second foreground depth map loss function, a total loss function is determined during the training of the 3D target detection model; based on the total loss function, the 3D target detection model is trained to obtain the trained 3D target detection model.

[0008] According to a 3D target recognition method provided by the present invention, the step of constructing a first loss function in the roadside image domain based on a plurality of roadside image training samples specifically includes: extracting first content features from each of the roadside image training samples; performing spatial dimension alignment processing on the plurality of first content features to obtain a plurality of aligned first content features; performing element-wise averaging processing on the plurality of aligned first content features to obtain a first target content feature; determining a first depth feature corresponding to the first target content feature based on the first target content feature; performing embedding processing on the first target content feature and the first depth feature respectively to obtain a first content embedding feature corresponding to the first target content feature and a first depth embedding feature corresponding to the first depth feature; and performing a method for recognizing a target based on the first content embedding feature and the first depth embedding feature. The initial target object query features corresponding to the roadside image training samples are updated to obtain the first updated query features. Based on the first updated query features, the detection head network predicts the first initial 3D target detection result in the roadside image domain. According to the 3D target detection result label corresponding to the roadside image training samples and the first initial 3D target detection result, the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, the first 2D bounding box loss function, and the first 3D center point loss function are calculated in the roadside domain, respectively. Based on the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, the first 2D bounding box loss function, and the first 3D center point loss function, the first loss function in the roadside image domain is constructed.

[0009] According to a 3D target recognition method provided by the present invention, the step of constructing a first loss function in the roadside image domain based on a first category loss function, a first 3D size loss function, a first orientation angle loss function, a first depth loss function, a first 2D bounding box loss function, and a first 3D center point loss function specifically includes: determining the 3D target detection result label corresponding to the roadside image training sample and the first matching cost of the first initial 3D target detection result based on the first category loss function, the first 3D center point loss function, and the first 2D bounding box loss function; and constructing the first loss function in the roadside image domain based on the first matching cost and the first 3D size loss function, the first orientation angle loss function, and the first depth loss function.

[0010] According to a 3D target recognition method provided by the present invention, after determining the first depth feature corresponding to the first target content feature based on the first target content feature, the method further includes: predicting a first foreground depth prediction map corresponding to the first depth feature through a convolutional layer based on the first depth feature; obtaining a first foreground depth ground truth map corresponding to the roadside image training sample; and determining a first foreground depth map loss function based on the first foreground depth prediction map and the first foreground depth prediction map.

[0011] According to a 3D target recognition method provided by the present invention, the step of constructing a second loss function in the vehicle image domain based on a plurality of vehicle-end image training samples specifically includes: extracting second content features from each of the vehicle-end image training samples; performing spatial dimension alignment processing on the plurality of second content features to obtain a plurality of aligned second content features; performing element-wise averaging processing on the plurality of aligned second content features to obtain a second target content feature; determining a second depth feature corresponding to the second target content feature based on the second target content feature; performing embedding processing on the second target content feature and the second depth feature respectively to obtain a second content embedding feature corresponding to the second target content feature and a second depth embedding feature corresponding to the second depth feature; and performing a second loss function on the vehicle-end image domain based on the second content embedding feature and the second depth embedding feature. The initial target object query features corresponding to the vehicle-side image training samples are updated to obtain the second updated query features. Based on the second updated query features, the detection head network predicts the second initial 3D target detection result in the vehicle-side image domain. According to the 3D target detection result label corresponding to the vehicle-side image training samples and the second initial 3D target detection result, the second category loss function, the second 3D size loss function, the second orientation angle loss function, the second depth loss function, the second 2D bounding box loss function, and the second 3D center point loss function are calculated in the vehicle-side domain, respectively. Based on the second category loss function, the second 3D size loss function, the second orientation angle loss function, the second depth loss function, the second 2D bounding box loss function, and the second 3D center point loss function, the second loss function in the vehicle-side image domain is constructed.

[0012] According to a 3D target recognition method provided by the present invention, the step of constructing a second loss function in the vehicle-end image domain based on a second category loss function, a second 3D size loss function, a second orientation angle loss function, a second depth loss function, a second 2D bounding box loss function, and a second 3D center point loss function specifically includes: determining the 3D target detection result label corresponding to the vehicle-end image training sample and the second matching cost of the second initial 3D target detection result based on the second category loss function, the second 3D center point loss function, and the second 2D bounding box loss function; and constructing the second loss function in the vehicle-end image domain based on the second matching cost and the second 3D size loss function, the second orientation angle loss function, and the second depth loss function.

[0013] According to a 3D target recognition method provided by the present invention, after determining the second depth feature corresponding to the second target content feature based on the second target content feature, the method further includes: predicting a second foreground depth prediction map corresponding to the second depth feature through a convolutional layer based on the second depth feature; obtaining a second foreground depth ground truth map corresponding to the training sample of the vehicle-end image; and determining a second foreground depth map loss function based on the second foreground depth prediction map and the second foreground depth prediction map.

[0014] According to a 3D target recognition method provided by the present invention, the step of constructing a cross-view contrastive learning loss function based on a plurality of vehicle-end image training samples and a plurality of road-end image training samples specifically includes: obtaining a plurality of first updated query features corresponding to the plurality of vehicle-end image training samples based on the plurality of vehicle-end image training samples; obtaining a plurality of second updated query features corresponding to the plurality of road-end image training samples based on the plurality of road-end image training samples; performing matching processing on the plurality of first updated query features and the plurality of second updated query features based on the Hungarian algorithm to construct a matching score matrix; obtaining a first updated query feature with a matching score exceeding a first score threshold based on the matching score matrix to construct a first positive sample in the vehicle-end image domain, and obtaining a first updated query feature with a matching score lower than a second score threshold to construct a first negative sample in the vehicle-end image domain; and obtaining a second updated query feature with a matching score exceeding the first score threshold based on the matching score matrix to construct a first positive sample in the vehicle-end image domain. The system constructs a second positive sample in the roadside image domain and obtains a second updated query feature with a matching score lower than a second score threshold to construct a second negative sample in the roadside image domain. Based on the first positive sample and the second positive sample, it constructs a total positive sample across the domain. Based on the first negative sample and the second negative sample, it constructs a total negative sample across the domain. The updated query feature of each positive sample in the total positive sample across the domain is decoupled from its semantic and geometric features to obtain the semantic query feature of each positive sample. The updated query feature of each negative sample in the total negative sample across the domain is decoupled from its semantic and geometric features to obtain the semantic query feature of each negative sample. The system calculates the target similarity between the semantic query feature of the positive sample and the semantic query feature of other samples, wherein the semantic query feature of other samples includes the semantic query features of other positive samples besides the semantic query feature of the positive sample and the semantic query feature of the negative sample. Based on the target similarity, the system constructs the cross-view contrastive learning loss function.

[0015] The present invention also provides a 3D target recognition device, the device comprising: an acquisition module for acquiring a roadside image to be detected, wherein the roadside image to be detected is an image to be detected acquired by an image acquisition device set at the roadside; and a processing module for inputting the roadside image to be detected into a pre-trained 3D target detection model to obtain a 3D target detection result output by the 3D target detection model corresponding to the roadside image to be detected, wherein the 3D target detection model is obtained by cross-view comparative learning training through a combination of roadside image training samples and vehicle-side image training samples.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the 3D target recognition method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the 3D target recognition method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the 3D target recognition method as described above.

[0019] The 3D target recognition method, apparatus, electronic device, and storage medium provided by this invention acquire a roadside image to be detected, wherein the roadside image to be detected is an image acquired by an image acquisition device installed at the roadside; the roadside image to be detected is input into a pre-trained 3D target detection model, and the 3D target detection result corresponding to the roadside image to be detected can be obtained from the output of the 3D target detection model. The 3D target detection model is trained by combining roadside image training samples and vehicle-side image training samples through cross-view comparative learning. Since the acquisition cost of vehicle-side image training samples is lower, and the 3D target detection model is trained by combining roadside image training samples and vehicle-side image training samples through cross-view comparative learning, the model can ensure the prediction accuracy of the model, thereby reducing the cost of 3D target recognition while ensuring the accuracy of 3D target recognition. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1This is a flowchart illustrating the 3D target recognition method provided by the present invention;

[0022] Figure 2 This is a schematic diagram of the process for training a 3D object detection model provided by the present invention;

[0023] Figure 3 This is a flowchart illustrating the process of constructing a first loss function in the roadside image domain based on multiple roadside image training samples provided by the present invention.

[0024] Figure 4 This is a flowchart illustrating the process of constructing a cross-view contrastive learning loss function based on multiple vehicle-end image training samples and multiple road-end image training samples provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of the 3D target recognition device provided by the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] Figure 1 This is a flowchart illustrating the 3D target recognition method provided by the present invention.

[0029] The following will combine Figure 1 The process of the 3D target recognition method provided by the present invention will be described.

[0030] In an exemplary embodiment of the present invention, combined with Figure 1 As can be seen, the 3D target recognition method may include steps 110 and 120, which will be described in detail below.

[0031] In step 110, the image of the road end to be detected is acquired, wherein the image of the road end to be detected is the image to be detected acquired by the image acquisition device set at the road end.

[0032] In one embodiment, a roadside image to be detected can be acquired, wherein the roadside image can be an image representing an image acquired by an image acquisition device installed at the roadside. A vehicle-side image can be an image representing an image acquired by an image acquisition device installed at the vehicle. In application, acquiring vehicle-side images is easier and less costly than acquiring roadside images.

[0033] In step 120, the roadside image to be detected is input into the pre-trained 3D target detection model to obtain the 3D target detection result output by the 3D target detection model corresponding to the roadside image to be detected.

[0034] In one embodiment, the roadside image to be detected can be input into a pre-trained 3D object detection model, thereby obtaining a 3D object detection result output by the 3D object detection model corresponding to the roadside image to be detected. The 3D object detection result can be considered as determining the position of a preset object in three-dimensional space.

[0035] Among them, the 3D object detection model can be trained by combining roadside image training samples and vehicle-side image training samples through cross-view comparative learning.

[0036] It should be noted that images from roadside cameras and vehicle-mounted cameras often have different perspectives, leading to differences in the distribution of the two data domains. Training a roadside object detection model (corresponding to a 3D object detection model) requires collecting and labeling a large amount of roadside data, a very time-consuming process. This invention utilizes a cross-view contrastive learning method, leveraging a large amount of labeled vehicle-mounted data to enhance the training effect of the roadside model. In this invention, the data domain is decoupled into semantic and geometric parts. The semantic part captures the semantic similarity between the two data domains and calculates the contrastive and classification losses, while the geometric part represents the geometric differences between the two data domains, calculating the regression loss only through ground truth supervision.

[0037] In this embodiment, the 3D object detection model is trained using cross-view comparative learning by combining roadside image training samples and vehicle-side image training samples. Since obtaining vehicle-side image training samples is less costly, training the 3D object detection model using cross-view comparative learning by combining roadside and vehicle-side image training samples ensures the model's prediction accuracy, thereby reducing the cost of 3D object recognition while maintaining accuracy.

[0038] The 3D target recognition method provided by this invention acquires a roadside image to be detected, wherein the roadside image to be detected is an image acquired by an image acquisition device set at the roadside; the roadside image to be detected is input into a pre-trained 3D target detection model, and the 3D target detection result corresponding to the roadside image to be detected can be obtained from the output of the 3D target detection model. The 3D target detection model is trained by combining roadside image training samples and vehicle-side image training samples through cross-view comparative learning. Since the acquisition cost of vehicle-side image training samples is lower, and the 3D target detection model is trained by combining roadside image training samples and vehicle-side image training samples through cross-view comparative learning, the 3D target detection model can ensure the prediction accuracy of the model, thereby reducing the cost of 3D target recognition while ensuring the accuracy of 3D target recognition.

[0039] Figure 2 This is a schematic diagram of the process for training a 3D object detection model provided by the present invention.

[0040] The following will combine Figure 2 The process of training the 3D object detection model provided by this invention will be described.

[0041] In an exemplary embodiment of the present invention, combined with Figure 2 As can be seen, training a 3D object detection model may include steps 210 to 250, which will be described in detail below.

[0042] In step 210, a first loss function and a first foreground depth map loss function are constructed in the roadside image domain based on multiple roadside image training samples.

[0043] In step 220, a second loss function and a second foreground depth map loss function are constructed in the vehicle image domain based on multiple vehicle-end image training samples.

[0044] In one embodiment, multiple roadside image training samples and multiple vehicle-side image training samples can be obtained respectively, and a first loss function L in the roadside image domain can be constructed based on the multiple roadside image training samples. pair,1 And the first foreground depth map loss function L dmap,1 ; and a second loss function L constructed in the vehicle image domain based on multiple vehicle-side image training samples. pair,2 And the second foreground depth map loss function L dmap,2 .

[0045] In step 230, a cross-view contrastive learning loss function is constructed based on multiple vehicle-end image training samples and multiple road-end image training samples.

[0046] In step 240, the total loss function in the process of training the 3D object detection model is determined based on the first loss function, the second loss function, the cross-view contrastive learning loss function, the first foreground depth map loss function, and the second foreground depth map loss function.

[0047] In step 250, the 3D object detection model is trained based on the total loss function to obtain the trained 3D object detection model.

[0048] In another embodiment, a cross-view contrastive learning loss function L can be constructed between the vehicle end and the road end based on multiple vehicle-end image training samples and multiple road end image training samples. cl Then, based on the first loss function L obtained earlier... pair,1 Second loss function L pair,2 Cross-view contrastive learning loss function L cl The first foreground depth map loss function L dmap,1 And the second foreground depth map loss function L dmap,2 The total loss function is determined during the training of the 3D object detection model. Furthermore, this total loss function is used as the loss function for training the 3D object detection model, and the model is then trained to obtain the trained 3D object detection model.

[0049] In another embodiment, the total loss function in the 3D object detection model process can be expressed by the following formula (1):

[0050]

[0051] Where K represents the total number of queries from both the vehicle and road ends; <·> represents the inner product; N V N represents the number of target-tag pairs on the vehicle side. R This indicates the number of target-tag pairs at the road end.

[0052] Figure 3 This is a flowchart illustrating the process of constructing a first loss function in the roadside image domain based on multiple roadside image training samples provided by the present invention.

[0053] The following will combine Figure 3 The process of constructing a first loss function in the roadside image domain based on multiple roadside image training samples provided by the present invention will be described.

[0054] In an exemplary embodiment of the present invention, combined with Figure 3 As can be seen, the first loss function constructed in the roadside image domain based on multiple roadside image training samples may include steps 310 to 390, and each step will be described below.

[0055] In step 310, the first content features of the training samples of each end image are extracted respectively.

[0056] In one embodiment, ResNet-50 can be used as the backbone network to extract images from roadside image training samples. The first content feature. Among them, H represents the real number field, W represents the image height dimension, and 3 represents the image width dimension; 3 represents the RGB three channels.

[0057] In step 320, multiple first content features are spatially aligned to obtain multiple aligned first content features.

[0058] In step 330, the first content features after multiple alignment processes are averaged at the element level to obtain the first target content features.

[0059] In yet another embodiment, multiple multi-scale first content features F can be used. C Multiple aligned first content features are obtained by aligning them to a unified spatial dimension using nearest neighbor interpolation and a convolutional neural network, respectively. Then, element-wise averaging of these aligned first content features yields the first target content feature f. c .

[0060] Among them, F c and f c The relationship has the following formula (2):

[0061]

[0062] In step 340, a first depth feature corresponding to the first target content feature is determined based on the first target content feature.

[0063] In another embodiment, the first target content features at multiple scales can be used to obtain the first depth features through two 3×3 convolutional layers.

[0064] In step 350, the first target content feature and the first depth feature are embedded to obtain the first content embedding feature corresponding to the first target content feature and the first depth embedding feature corresponding to the first depth feature.

[0065] In step 360, based on the first content embedding feature and the first deep embedding feature, the initial target object query feature corresponding to the roadside image training sample is updated to obtain the first updated query feature.

[0066] In another embodiment, the first target content feature and the first depth feature can be embedded using a content encoder and a depth encoder, respectively, to obtain a first content embedding feature corresponding to the first target content feature. and the first deep embedding feature corresponding to the first deep feature Each embedding (the first content embedding feature and the first deep embedding feature) integrates a sin / cosine positional embedding. The embedding features are then updated through a self-attention layer and a feed-forward neural network.

[0067] Furthermore, based on the decoder, a set of target object query features can be initialized first (corresponding to the initialization of target object query features). The initialization target object query feature will be combined with the first content embedding feature. and first deep embedding features The interaction updates its own query features to obtain the updated query features (corresponding to the first updated query features). The decoder consists of deep cross-attention, self-attention, content cross-attention, and feedforward neural network modules. The output of the decoder can be defined as Q... d That is, the query features after the first update.

[0068] In step 370, based on the first updated query features, the detection head network predicts the first initial 3D target detection result in the roadside image domain.

[0069] In step 380, based on the 3D target detection result labels corresponding to the roadside image training samples and the first initial 3D target detection result, the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, and the first 2D bounding box loss function in the roadside domain are calculated respectively.

[0070] In step 390, a first loss function is constructed in the roadside image domain based on the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, and the first 2D bounding box loss function.

[0071] In another embodiment, the first updated query features can learn information in a single-view data domain (in this embodiment, information in the roadside domain) and can predict preliminary 3D target information through a detection head network, that is, the first initial 3D target detection result in the roadside image domain.

[0072] Furthermore, based on the 3D object detection result labels corresponding to the roadside image training samples, and the first initial 3D object detection result, the first category loss function L in the roadside domain can be calculated. cls First 3D size loss function L dim First direction angle loss function L ori First depth loss function L depth First 2D bounding box loss function L edge ,L GIoU and the first 3D center point loss function L 3D A first loss function is constructed in the roadside image domain based on the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, the first 2D bounding box loss function, and the first 3D center point loss function.

[0073] In yet another exemplary embodiment of the present invention, the preceding text continues... Figure 3 The embodiment described above is used as an example for illustration. Based on the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, the first 2D bounding box loss function, and the first 3D center point loss function, a first loss function L is constructed in the roadside image domain. pair,1 (Corresponding to step 390) can be achieved in the following way:

[0074] Based on the first category loss function, the first 3D center point loss function, and the first 2D bounding box loss function, the 3D object detection result label corresponding to the roadside image training sample and the first matching cost of the first initial 3D object detection result are determined.

[0075] A first loss function is constructed in the roadside image domain based on the first matching cost, the first 3D size loss function, the first orientation angle loss function, and the first depth loss function.

[0076] In one embodiment, the matched prediction information and ground truth (corresponding to the 3D object detection result labels corresponding to the roadside image training samples, and the first initial 3D object detection result), for each prediction information and ground truth, can define a first matching cost C. match Then, the Hungarian Algorithm is used to complete the matching.

[0077] In yet another embodiment, the first matching cost C match It can be expressed using the following formula (3):

[0078] C ma =λ1L cls +λ2L 3D +λ3L edge+λ1L GIoU (3)

[0079] Among them, λ1, L2 and λ3 are coefficients that can be adjusted according to the actual situation.

[0080] Furthermore, based on the cost C of the first match... match and the first 3D size loss function L dim First direction angle loss function L ori and the first depth loss function L dept The first loss function L is constructed in the roadside image domain. pair,1 Among them, the first loss function L, constructed in the roadside image domain, is... pair,1 It can be expressed using formula (4):

[0081] L pair,1 =C mat +λ5L dim +λ6L ori +λ7L depth (4)

[0082] In yet another exemplary embodiment of the present invention, the description above continues. Figure 3 Taking the following embodiment as an example, after determining the first depth feature corresponding to the first target content feature based on the first target content feature (corresponding to step 340), the 3D target recognition method may further include the following steps:

[0083] Based on the first depth feature, a first foreground depth prediction map corresponding to the first depth feature is obtained through convolutional layer prediction;

[0084] Obtain the ground truth map of the first foreground depth corresponding to the training samples of the roadside image, and determine the loss function of the first foreground depth map based on the first foreground depth prediction map and the first foreground depth prediction map.

[0085] In one embodiment, a first foreground depth prediction map corresponding to the first depth feature can also be predicted using a 3×3 convolutional layer based on the first depth feature. Where D represents the dimension of depth.

[0086] Furthermore, the ground truth map D of the first foreground depth corresponding to the training samples of the roadside image can also be obtained. gt And using Focal Loss, based on the first foreground depth prediction map and the first foreground depth prediction map, the loss function L of the first foreground depth map is determined. dmap .

[0087] The foregoing embodiments described the process of constructing a loss function related to the roadside domain. The following will describe the process of constructing a loss function related to the vehicle-side domain.

[0088] In yet another exemplary embodiment of the present invention, continuing with Figure 2 The above embodiment will be used as an example for illustration. Based on multiple vehicle-end image training samples, the second loss function (corresponding to step 220) constructed in the vehicle-end image domain can be implemented in the following way:

[0089] Extract the second content features from each vehicle-end image training sample;

[0090] Multiple second content features are spatially aligned to obtain multiple aligned second content features.

[0091] The second target content feature is obtained by averaging multiple aligned second content features at the element level.

[0092] Based on the second target content features, determine the second depth features corresponding to the second target content features;

[0093] The second target content feature and the second depth feature are embedded respectively to obtain the second content embedding feature corresponding to the second target content feature and the second depth embedding feature corresponding to the second depth feature;

[0094] Based on the second content embedding feature and the second deep embedding feature, the initial target object query feature corresponding to the vehicle-end image training sample is updated to obtain the second updated query feature;

[0095] Based on the second updated query features, the second initial 3D target detection result in the vehicle image domain is predicted by the detection head network.

[0096] Based on the 3D target detection result labels corresponding to the training samples of the vehicle-end image, and the second initial 3D target detection result, the second category loss function, the second 3D size loss function, the second orientation angle loss function, the second depth loss function, the second 2D bounding box loss function, and the second 3D center point loss function are calculated in the vehicle-end domain respectively.

[0097] A second loss function is constructed in the vehicle-side image domain based on the second category loss function, the second 3D size loss function, the second orientation angle loss function, the second depth loss function, the second 2D bounding box loss function, and the second 3D center point loss function.

[0098] It should be noted that the second loss function L, constructed within the vehicle-side image domain, pait,2 This is consistent with the first loss function L constructed in the roadside image domain mentioned earlier. pair,1 The process is the same or similar, and will not be specifically limited in this embodiment.

[0099] In another exemplary embodiment of the present invention, the following description continues using the previously described embodiments as examples. The second loss function constructed within the vehicle-end image domain based on the second category loss function, the second 3D size loss function, the second orientation angle loss function, the second depth loss function, the second 2D bounding box loss function, and the second 3D center point loss function can be implemented in the following manner:

[0100] Based on the second category loss function, the second 3D center point loss function, and the second 2D bounding box loss function, the 3D object detection result label corresponding to the training sample of the vehicle end image and the second matching cost of the second initial 3D object detection result are determined.

[0101] A second loss function is constructed in the vehicle-end image domain based on the second matching cost, the second 3D size loss function, the second orientation angle loss function, and the second depth loss function.

[0102] It should be noted that the second loss function L, constructed within the vehicle-side image domain, pair,2 This is consistent with the first loss function L constructed in the roadside image domain mentioned earlier. pair,1 The process is the same or similar, and will not be specifically limited in this embodiment.

[0103] In yet another exemplary embodiment of the present invention, continuing with the previously described embodiments, after determining the second depth feature corresponding to the second target content feature based on the second target content feature, the 3D target recognition method may further include the following steps:

[0104] Based on the second depth feature, a second foreground depth prediction map corresponding to the second depth feature is obtained by a convolutional layer;

[0105] Obtain the ground truth map of the second foreground depth corresponding to the training samples of the vehicle-end image, and determine the loss function of the second foreground depth map based on the second foreground depth prediction map and the second foreground depth prediction map.

[0106] It should be noted that the process of determining the second foreground depth map loss function is the same as or similar to the process of determining the first foreground depth map loss function described above, and will not be specifically limited in this embodiment.

[0107] Figure 4 This is a flowchart illustrating the process of constructing a cross-view contrastive learning loss function based on multiple vehicle-end image training samples and multiple road-end image training samples provided by the present invention.

[0108] The following will combine Figure 4 The 3D target recognition method provided by this invention will be described.

[0109] In an exemplary embodiment of the present invention, combined with Figure 4As can be seen, constructing a cross-view contrastive learning loss function based on multiple vehicle-end image training samples and multiple road-end image training samples can include steps 401 to 411, which will be described in detail below.

[0110] In step 401, based on multiple vehicle-end image training samples, multiple first updated query features corresponding to the multiple vehicle-end image training samples are obtained.

[0111] In step 402, based on multiple roadside image training samples, multiple second updated query features corresponding to the multiple roadside image training samples are obtained.

[0112] In one set of embodiments, as described above, multiple first updated query features corresponding to multiple vehicle-end image training samples can be obtained based on multiple vehicle-end image training samples, and multiple second updated query features corresponding to multiple road-end image training samples can be obtained based on multiple road-end image training samples. The first updated query features in the vehicle-end image domain can be represented as follows: The second updated query feature in the vehicle-side image domain can be represented as:

[0113] In step 403, based on the Hungarian algorithm, multiple first updated query features and multiple second updated query features are matched to construct a matching score matrix.

[0114] In step 404, based on the matching score matrix, the first updated query feature with a matching score exceeding the first score threshold is obtained to construct the first positive sample in the vehicle image domain, and the first updated query feature with a matching score below the second score threshold is obtained to construct the first negative sample in the vehicle image domain.

[0115] In step 405, based on the matching score matrix, a second updated query feature with a matching score exceeding the first score threshold is obtained to construct a second positive sample in the roadside image domain, and a second updated query feature with a matching score below the second score threshold is obtained to construct a second negative sample in the roadside image domain.

[0116] In one embodiment, during the matching process based on the Hungarian algorithm, multiple first updated query features and multiple second updated query features can be matched to construct a matching score matrix.

[0117] Furthermore, based on the matching score matrix, the first updated query features with matching scores exceeding a first score threshold can be obtained to construct the first positive sample in the vehicle-side image domain, and the first updated query features with matching scores below a second score threshold can be obtained to construct the first negative sample in the vehicle-side image domain. In one example, the k queries with the highest matching scores can be defined as the first positive sample. It can also match the k queries with the lowest scores as the first negative sample.

[0118] In another embodiment, based on the same concept, the k queries with the highest matching scores can also be obtained as the second positive sample. It can also match the k lowest-scoring queries as the second negative samples.

[0119] In step 406, a cross-domain total positive sample is constructed based on the first positive sample and the second positive sample.

[0120] In step 407, a cross-domain total negative sample is constructed based on the first negative sample and the second negative sample.

[0121] In another embodiment, a cross-domain total positive sample can be constructed based on the first positive sample and the second positive sample. The cross-domain total positive sample can be represented as... Cross-domain total negative samples can be represented as

[0122] In step 408, the updated query features of each positive sample in the cross-domain total positive samples are decoupled from semantic features and geometric features to obtain the semantic query features of each positive sample.

[0123] In step 409, the updated query features of each negative sample in the cross-domain total negative samples are decoupled from their semantic and geometric features to obtain the semantic query features of each negative sample.

[0124] In one embodiment, the updated query features for each positive and negative sample (wherein, for positive samples, it is the first updated query feature, and for negative samples, it is the second updated query feature) are decoupled into semantic and geometric parts along the channel dimension, with the first N / 2 channels representing semantic features. The latter N / 2 channels represent geometric features. Only semantic features are used for contrastive learning across cross-view data domains to help the decoder learn better feature representations.

[0125] In step 410, the target similarity between the semantic query features of the positive sample and the semantic query features of other samples is calculated.

[0126] In step 411, a cross-view contrastive learning loss function is constructed based on the target similarity.

[0127] The semantic query features of other samples include the semantic query features of other positive samples besides the semantic query features of positive samples and the semantic query features of negative samples.

[0128] In another embodiment, positive and negative samples from the vehicle end and the road end can be merged into one. These correspond to the total positive samples and the total negative samples across the domain, respectively.

[0129] Q can be defined P The i-th query feature in the set (corresponding to the semantic query feature of the positive sample) With {Q P Q N The j-th query feature in the set (corresponding to the semantic query features of other samples) The similarity between targets can be expressed as formula (5):

[0130]

[0131] Furthermore, a cross-view comparison learning loss function can be constructed based on target similarity, whereby the cross-view comparison learning loss function can be expressed as formula (6):

[0132]

[0133] The labels for contrastive learning can be represented by the following formula (7):

[0134]

[0135] K represents the total number of queries from both the vehicle and road ends; <·> represents the inner product; P represents Q. P The index of the set. Thus, for N... V Individual vehicle terminal and N R The total loss function of each target-label pair, that is, the total loss function in the 3D target detection model process, can be expressed by the aforementioned formula (1).

[0136] This invention can use a large amount of vehicle-side data and a small amount of roadside data to jointly train an object detector (corresponding to a 3D object detection model). Through a semantic-set decoupling cross-view contrastive learning method, it leverages a large amount of vehicle-side data to enhance the performance of the roadside object detector.

[0137] Based on the same concept, the present invention also provides a 3D target recognition device.

[0138] The 3D target recognition device provided by the present invention is described below. The 3D target recognition device described below can be referred to in correspondence with the 3D target recognition method described above.

[0139] Figure 5 This is a schematic diagram of the structure of the 3D target recognition device provided by the present invention.

[0140] In an exemplary embodiment of the present invention, combined with Figure 5As can be seen, the 3D target recognition device may include an acquisition module 510 and a processing module 520, and each module will be described in detail below.

[0141] The acquisition module 510 can be configured to acquire a roadside image to be detected, wherein the roadside image to be detected is an image to be detected acquired by an image acquisition device set at the roadside;

[0142] Processing module 520 can be configured to input the roadside image to be detected into a pre-trained 3D object detection model, and obtain a 3D object detection result output by the 3D object detection model corresponding to the roadside image to be detected, wherein...

[0143] The 3D target detection model is trained by combining roadside image training samples and vehicle-side image training samples through cross-view comparative learning.

[0144] In an exemplary embodiment of the present invention, the processing module 520 may train and obtain a 3D object detection model in the following manner:

[0145] Based on multiple roadside image training samples, a first loss function and a first foreground depth map loss function are constructed in the roadside image domain;

[0146] Based on multiple training samples of the vehicle-end images, a second loss function and a second foreground depth map loss function are constructed in the vehicle-end image domain;

[0147] A cross-view contrastive learning loss function is constructed based on multiple vehicle-side image training samples and multiple road-side image training samples.

[0148] Based on the first loss function, the second loss function, the cross-view contrastive learning loss function, the first foreground depth map loss function, and the second foreground depth map loss function, the total loss function in the process of training the 3D object detection model is determined;

[0149] Based on the total loss function, the 3D object detection model is trained to obtain the trained 3D object detection model.

[0150] In an exemplary embodiment of the present invention, the processing module 520 may construct a first loss function in the roadside image domain based on a plurality of roadside image training samples in the following manner:

[0151] Extract the first content features from each of the roadside image training samples;

[0152] Multiple first content features are spatially aligned to obtain multiple aligned first content features.

[0153] The first target content feature is obtained by averaging multiple aligned first content features at the element level.

[0154] Based on the first target content features, a first depth feature corresponding to the first target content features is determined;

[0155] The first target content feature and the first depth feature are respectively embedded to obtain a first content embedding feature corresponding to the first target content feature and a first depth embedding feature corresponding to the first depth feature;

[0156] Based on the first content embedding feature and the first deep embedding feature, the initial target object query feature corresponding to the roadside image training sample is updated to obtain the first updated query feature;

[0157] Based on the first updated query features, the first initial 3D target detection result in the roadside image domain is predicted by the detection head network.

[0158] Based on the 3D target detection result labels corresponding to the roadside image training samples, and the first initial 3D target detection result, the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, the first 2D bounding box loss function, and the first 3D center point loss function are calculated in the roadside domain, respectively.

[0159] A first loss function is constructed in the roadside image domain based on the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, the first 2D bounding box loss function, and the first 3D center point loss function.

[0160] In an exemplary embodiment of the present invention, the processing module 520 may construct a first loss function in the roadside image domain based on the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, the first 2D bounding box loss function, and the first 3D center point loss function in the following manner:

[0161] Based on the first category loss function, the first 3D center point loss function, and the first 2D bounding box loss function, the 3D target detection result label corresponding to the roadside image training sample and the first matching cost of the first initial 3D target detection result are determined.

[0162] A first loss function is constructed in the roadside image domain based on the first matching cost, the first 3D size loss function, the first orientation angle loss function, and the first depth loss function.

[0163] In an exemplary embodiment of the present invention, the processing module 520 may further be configured to:

[0164] Based on the first depth feature, a first foreground depth prediction map corresponding to the first depth feature is obtained by a convolutional layer;

[0165] Obtain the first foreground depth ground truth map corresponding to the training samples of the roadside image, and determine the first foreground depth map loss function based on the first foreground depth prediction map and the first foreground depth prediction map.

[0166] In an exemplary embodiment of the present invention, the processing module 520 may construct a second loss function in the vehicle-end image domain based on a plurality of vehicle-end image training samples in the following manner:

[0167] Extract the second content features from each of the vehicle-end image training samples;

[0168] Multiple second content features are spatially aligned to obtain multiple aligned second content features.

[0169] The second target content feature is obtained by averaging multiple aligned second content features at the element level.

[0170] Based on the second target content features, a second depth feature corresponding to the second target content features is determined;

[0171] The second target content feature and the second depth feature are embedded respectively to obtain the second content embedding feature corresponding to the second target content feature and the second depth embedding feature corresponding to the second depth feature;

[0172] Based on the second content embedding feature and the second deep embedding feature, the initial target object query feature corresponding to the vehicle-end image training sample is updated to obtain the second updated query feature;

[0173] Based on the second updated query features, the second initial 3D target detection result in the vehicle image domain is predicted by the detection head network.

[0174] Based on the 3D target detection result labels corresponding to the vehicle-end image training samples, and the second initial 3D target detection result, the second category loss function, the second 3D size loss function, the second orientation angle loss function, the second depth loss function, the second 2D bounding box loss function, and the second 3D center point loss function are calculated in the vehicle-end domain, respectively.

[0175] A second loss function is constructed in the vehicle-end image domain based on the second category loss function, the second 3D size loss function, the second orientation angle loss function, the second depth loss function, the second 2D bounding box loss function, and the second 3D center point loss function.

[0176] In an exemplary embodiment of the present invention, the processing module 520 may implement a second loss function constructed in the vehicle-end image domain based on the second category loss function, the second 3D size loss function, the second orientation angle loss function, the second depth loss function, the second 2D bounding box loss function, and the second 3D center point loss function in the following manner:

[0177] Based on the second category loss function, the second 3D center point loss function, and the second 2D bounding box loss function, the 3D object detection result label corresponding to the vehicle end image training sample and the second matching cost of the second initial 3D object detection result are determined.

[0178] A second loss function is constructed in the vehicle-end image domain based on the second matching cost, the second 3D size loss function, the second orientation angle loss function, and the second depth loss function.

[0179] In an exemplary embodiment of the present invention, the processing module 520 may further be configured to:

[0180] Based on the second depth feature, a second foreground depth prediction map corresponding to the second depth feature is obtained by a convolutional layer;

[0181] Obtain the second foreground depth ground truth map corresponding to the training samples of the vehicle-end image, and determine the second foreground depth map loss function based on the second foreground depth prediction map and the second foreground depth prediction map.

[0182] In an exemplary embodiment of the present invention, the processing module 520 may construct a cross-view contrastive learning loss function based on a plurality of vehicle-end image training samples and a plurality of road-end image training samples in the following manner:

[0183] Based on multiple vehicle-side image training samples, obtain multiple first updated query features corresponding to the multiple vehicle-side image training samples;

[0184] Based on multiple roadside image training samples, obtain multiple second updated query features corresponding to the multiple roadside image training samples;

[0185] Based on the Hungarian algorithm, multiple first updated query features and multiple second updated query features are matched to construct a matching score matrix;

[0186] Based on the matching score matrix, obtain the first updated query feature whose matching score exceeds the first score threshold, construct the first positive sample in the vehicle image domain, and obtain the first updated query feature whose matching score is lower than the second score threshold, construct the first negative sample in the vehicle image domain.

[0187] Based on the matching score matrix, a second updated query feature with a matching score exceeding a first score threshold is obtained to construct a second positive sample in the roadside image domain, and a second updated query feature with a matching score below a second score threshold is obtained to construct a second negative sample in the roadside image domain.

[0188] Based on the first positive sample and the second positive sample, construct a cross-domain total positive sample;

[0189] Based on the first negative sample and the second negative sample, construct a cross-domain total negative sample;

[0190] The updated query features of each positive sample in the cross-domain total positive samples are decoupled from semantic features and geometric features to obtain the semantic query features of each positive sample.

[0191] The updated query features of each negative sample in the cross-domain total negative samples are decoupled from the semantic features and geometric features to obtain the semantic query features of each negative sample.

[0192] Calculate the target similarity between the semantic query features of the positive sample and the semantic query features of other samples, wherein the semantic query features of other samples include the semantic query features of other positive samples besides the semantic query features of the positive sample and the semantic query features of the negative sample;

[0193] Based on the target similarity, the cross-view contrastive learning loss function is constructed.

[0194] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions from the memory 630 to execute a 3D target recognition method.

[0195] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the 3D target recognition method provided by the above methods.

[0197] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the 3D target recognition methods provided by the methods described above.

[0198] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0199] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0200] It is further understood that although the operations are described in a specific order in the accompanying drawings in the embodiments of the present invention, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all the operations shown to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D target recognition method, characterized in that, The method includes: Acquire an image of the roadside to be detected, wherein the image of the roadside to be detected is an image acquired by an image acquisition device set at the roadside; The roadside image to be detected is input into a pre-trained 3D object detection model to obtain the 3D object detection result output by the 3D object detection model corresponding to the roadside image to be detected. The 3D object detection model is trained by combining roadside image training samples and vehicle-side image training samples through cross-view comparative learning. The 3D object detection model is trained in the following manner: Based on multiple roadside image training samples, a first loss function and a first foreground depth map loss function are constructed in the roadside image domain; Based on multiple training samples of the vehicle-end images, a second loss function and a second foreground depth map loss function are constructed in the vehicle-end image domain; A cross-view contrastive learning loss function is constructed based on multiple vehicle-side image training samples and multiple road-side image training samples. Based on the first loss function, the second loss function, the cross-view contrastive learning loss function, the first foreground depth map loss function, and the second foreground depth map loss function, the total loss function in the process of training the 3D object detection model is determined; Based on the total loss function, the 3D object detection model is trained to obtain the trained 3D object detection model. Specifically, the step of constructing a cross-view contrastive learning loss function based on multiple vehicle-side image training samples and multiple road-side image training samples includes: Based on multiple vehicle-side image training samples, obtain multiple first updated query features corresponding to the multiple vehicle-side image training samples; Based on multiple roadside image training samples, obtain multiple second updated query features corresponding to the multiple roadside image training samples; Based on the Hungarian algorithm, multiple first updated query features and multiple second updated query features are matched to construct a matching score matrix; Based on the matching score matrix, obtain the first updated query feature whose matching score exceeds the first score threshold, construct the first positive sample in the vehicle image domain, and obtain the first updated query feature whose matching score is lower than the second score threshold, construct the first negative sample in the vehicle image domain. Based on the matching score matrix, a second updated query feature with a matching score exceeding a first score threshold is obtained to construct a second positive sample in the roadside image domain, and a second updated query feature with a matching score below a second score threshold is obtained to construct a second negative sample in the roadside image domain. Based on the first positive sample and the second positive sample, construct a cross-domain total positive sample; Based on the first negative sample and the second negative sample, construct a cross-domain total negative sample; The updated query features of each positive sample in the cross-domain total positive samples are decoupled from semantic features and geometric features to obtain the semantic query features of each positive sample. The updated query features of each negative sample in the cross-domain total negative samples are decoupled from the semantic features and geometric features to obtain the semantic query features of each negative sample. Calculate the target similarity between the semantic query features of the positive sample and the semantic query features of other samples, wherein the semantic query features of other samples include the semantic query features of other positive samples besides the semantic query features of the positive sample and the semantic query features of the negative sample; Based on the target similarity, the cross-view contrastive learning loss function is constructed.

2. The 3D target recognition method according to claim 1, characterized in that, The step of constructing a first loss function in the roadside image domain based on multiple roadside image training samples specifically includes: Extract the first content features from each of the roadside image training samples; Multiple first content features are spatially aligned to obtain multiple aligned first content features. The first target content feature is obtained by averaging multiple aligned first content features at the element level. Based on the first target content features, a first depth feature corresponding to the first target content features is determined; The first target content feature and the first depth feature are respectively embedded to obtain a first content embedding feature corresponding to the first target content feature and a first depth embedding feature corresponding to the first depth feature; Based on the first content embedding feature and the first deep embedding feature, the initial target object query feature corresponding to the roadside image training sample is updated to obtain the first updated query feature; Based on the first updated query features, the first initial 3D target detection result in the roadside image domain is predicted by the detection head network. Based on the 3D target detection result labels corresponding to the roadside image training samples, and the first initial 3D target detection result, the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, the first 2D bounding box loss function, and the first 3D center point loss function are calculated in the roadside domain, respectively. A first loss function is constructed in the roadside image domain based on the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, the first 2D bounding box loss function, and the first 3D center point loss function.

3. The 3D target recognition method according to claim 2, characterized in that, The first loss function constructed in the roadside image domain based on the first category loss function, the first 3D size loss function, the first orientation angle loss function, the first depth loss function, the first 2D bounding box loss function, and the first 3D center point loss function specifically includes: Based on the first category loss function, the first 3D center point loss function, and the first 2D bounding box loss function, the 3D target detection result label corresponding to the roadside image training sample and the first matching cost of the first initial 3D target detection result are determined. A first loss function is constructed in the roadside image domain based on the first matching cost, the first 3D size loss function, the first orientation angle loss function, and the first depth loss function.

4. The 3D target recognition method according to claim 2, characterized in that, After determining the first depth feature corresponding to the first target content feature based on the first target content feature, the method further includes: Based on the first depth feature, a first foreground depth prediction map corresponding to the first depth feature is obtained by a convolutional layer; Obtain the first foreground depth ground truth map corresponding to the training samples of the roadside image, and determine the first foreground depth map loss function based on the first foreground depth prediction map and the first foreground depth prediction map.

5. The 3D target recognition method according to claim 1, characterized in that, The second loss function constructed in the vehicle image domain based on multiple vehicle-end image training samples specifically includes: Extract the second content features from each of the vehicle-end image training samples; Multiple second content features are spatially aligned to obtain multiple aligned second content features. The second target content feature is obtained by averaging multiple aligned second content features at the element level. Based on the second target content features, a second depth feature corresponding to the second target content features is determined; The second target content feature and the second depth feature are embedded respectively to obtain the second content embedding feature corresponding to the second target content feature and the second depth embedding feature corresponding to the second depth feature; Based on the second content embedding feature and the second deep embedding feature, the initial target object query feature corresponding to the vehicle-end image training sample is updated to obtain the second updated query feature; Based on the second updated query features, the second initial 3D target detection result in the vehicle image domain is predicted by the detection head network. Based on the 3D target detection result labels corresponding to the vehicle-end image training samples, and the second initial 3D target detection result, the second category loss function, the second 3D size loss function, the second orientation angle loss function, the second depth loss function, the second 2D bounding box loss function, and the second 3D center point loss function are calculated in the vehicle-end domain, respectively. A second loss function is constructed in the vehicle-end image domain based on the second category loss function, the second 3D size loss function, the second orientation angle loss function, the second depth loss function, the second 2D bounding box loss function, and the second 3D center point loss function.

6. The 3D target recognition method according to claim 5, characterized in that, The second loss function constructed in the vehicle-end image domain based on the second category loss function, the second 3D size loss function, the second orientation angle loss function, the second depth loss function, the second 2D bounding box loss function, and the second 3D center point loss function specifically includes: Based on the second category loss function, the second 3D center point loss function, and the second 2D bounding box loss function, the 3D object detection result label corresponding to the vehicle end image training sample and the second matching cost of the second initial 3D object detection result are determined. A second loss function is constructed in the vehicle-end image domain based on the second matching cost, the second 3D size loss function, the second orientation angle loss function, and the second depth loss function.

7. The 3D target recognition method according to claim 5, characterized in that, After determining the second deep feature corresponding to the second target content feature based on the second target content feature, the method further includes: Based on the second depth feature, a second foreground depth prediction map corresponding to the second depth feature is obtained by a convolutional layer; Obtain the second foreground depth ground truth map corresponding to the training samples of the vehicle-end image, and determine the second foreground depth map loss function based on the second foreground depth prediction map and the second foreground depth prediction map.

8. A 3D target recognition device, characterized in that, The apparatus is used to implement the 3D target recognition method according to any one of claims 1 to 7, and the apparatus comprises: The acquisition module is used to acquire the image of the roadside to be detected, wherein the image of the roadside to be detected is the image to be detected acquired by the image acquisition device set at the roadside; The processing module is used to input the roadside image to be detected into a pre-trained 3D object detection model, and obtain the 3D object detection result output by the 3D object detection model corresponding to the roadside image to be detected, wherein... The 3D target detection model is trained by combining roadside image training samples and vehicle-side image training samples through cross-view comparative learning.

Images