Methods, apparatus and equipment for open-set domain generalization of monocular 3D target detection

Abstract

This application relates to a method, apparatus, and device for open-set domain generalization in monocular 3D object detection. The method includes: acquiring an annotated image of at least one source domain and preprocessing it to obtain a processed annotated image; performing feature extraction and bounding box prediction on the processed annotated image based on a preset Deformable DETR object detector to obtain feature parameters of the object to be predicted, the category and size of the bounding box of the object to be predicted; generating a classification result of the object to be predicted using a preset energy function based on the feature parameters, the category and size of the bounding box of the object to be predicted, and obtaining a final 3D bounding box based on the classification result and the feature parameters of the object to be predicted, and performing object detection based on the final 3D bounding box. This solves the problem of domain shift that easily occurs in related detection techniques, leading to performance degradation, and improves the generalization effect on unknown domains.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method, apparatus and device for open set domain generalization of monocular 3D (three-dimensional) target detection. Background Technology

[0002] Autonomous driving technology has developed rapidly in recent years, with various automakers launching their own autonomous driving solutions, and large-scale deployment of autonomous driving technology is imminent. To achieve perception and target detection in autonomous driving, engineers have tried various sensors, the most common being LiDAR, cameras, and millimeter-wave radar. LiDAR can achieve the most accurate perception, but its extremely high price limits its application in non-high-end vehicles. Millimeter-wave radar is inexpensive, but it struggles to accurately perceive the size and shape of objects. Cameras, which mimic the senses of human drivers, are very suitable for autonomous driving scenarios. They are low-cost and can accurately identify objects and their semantic information. Using cameras alone to achieve target detection in autonomous driving scenarios has become a hot research topic.

[0003] Computer vision technologies, including object detection, have benefited from the advancements in deep learning, achieving remarkable results when combined with neural networks. The strong fitting capabilities and large-scale parameters of deep neural networks enable them to overcome various challenges, but this also creates a dependence on the training dataset. For monocular 3D object detection, models trained on the training dataset inevitably encounter domain shifts when deployed in real-world applications.

[0004] In practical deployment, due to the vast and diverse array of objects in the real world, models trained on training datasets may encounter unknown object categories. Existing work can only misidentify unknown categories as known categories or miss detections. This can negatively impact subsequent decision-making and planning, creating security risks that urgently need to be addressed. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for open set domain generalization in monocular 3D target detection. This addresses the problem that related monocular 3D target detection technologies do not consider domain generalization, resulting in significant performance degradation when the test domain shifts, leading to risks and security vulnerabilities. Furthermore, when encountering new categories during deployment, they cannot be correctly rejected, resulting in misidentification. This application is designed with domain generalization in mind, significantly improving generalization performance in unknown domains. It also possesses open set detection capabilities, enabling it to identify new unknown categories as unknown.

[0006] The first aspect of this application provides a method for open-set domain generalization of monocular 3D object detection, comprising the following steps: acquiring at least one labeled image of a source domain, and preprocessing the labeled image of the at least one source domain to obtain a processed labeled image; performing feature extraction and bounding box prediction on the processed labeled image based on a preset Deformable DETR object detector to obtain feature parameters of the object to be predicted, the bounding box category and size of the object to be predicted in the processed labeled image; and generating a classification result of the object to be predicted using the preset energy function based on the feature parameters of the object to be predicted, the bounding box category and size of the object to be predicted, and obtaining a final 3D bounding box (the smallest cuboid enclosing the target object in the real 3D world) based on the classification result and the feature parameters of the object to be predicted, so as to perform object detection based on the final 3D bounding box.

[0007] Optionally, in some embodiments, based on a preset Deformable DETR object detector, feature extraction and bounding box prediction of the processed labeled image further include: extracting features from the preprocessed labeled image using the feature extractor of DETR in the preset Deformable DETR object detector to obtain feature extraction results; processing the feature extraction results through a gradient reversal layer (GRL) to obtain processing results; and using a preset neighborhood discriminator to identify the processing results to obtain the feature parameters of the object to be predicted.

[0008] Optionally, in some embodiments, the step of generating a classification result of the object to be predicted using the preset energy function based on the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted, further includes: calculating the total energy of the object to be predicted using the preset energy function based on the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted; determining whether the total energy of the object to be predicted is less than a preset threshold; if the total energy of the object to be predicted is less than the preset threshold, then marking the object to be predicted as an unknown category.

[0009] Optionally, in some embodiments, the preset energy function is:

[0010]

[0011] Where T is the temperature coefficient, f is the feature of the object to be predicted, and g i (f) represents the probability of the object to be predicted in the i-th category, and C represents the total number of categories.

[0012] Optionally, in some embodiments, the feature parameters of the object to be predicted include at least one of the following: 3D depth, length, width, height, and rotation angle of the object to be predicted.

[0013] A second aspect of this application provides an apparatus for open-set domain generalization of monocular 3D object detection, comprising: an acquisition module for acquiring labeled images of at least one source domain and preprocessing the labeled images of the at least one source domain to obtain a processed labeled image; a prediction module for performing feature extraction and bounding box prediction on the processed labeled image based on a preset Deformable DETR object detector to obtain feature parameters of an object to be predicted, the bounding box category and size of the object to be predicted; and a generation module for generating a classification result of the object to be predicted using a preset energy function based on the feature parameters of the object to be predicted, the bounding box category and size of the object to be predicted, and obtaining a final 3D bounding box based on the classification result and the feature parameters of the object to be predicted, so as to perform object detection based on the final 3D bounding box.

[0014] Optionally, in some embodiments, the prediction module further includes: an extraction unit, used to extract features from the preprocessed labeled image using the feature extractor of the DETR in the preset Deformable DETR object detector to obtain a feature extraction result; and a generation unit, used to process the feature extraction result through a gradient inversion layer (GRL) to obtain a processing result, and to use a preset neighborhood discriminator to identify the processing result to obtain the feature parameters of the object to be predicted.

[0015] Optionally, in some embodiments, the generation module further includes: a calculation unit, configured to calculate the total energy of the object to be predicted based on the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted, and using the preset energy function; and a marking unit, configured to determine whether the total energy of the object to be predicted is less than a preset threshold, and if the total energy of the object to be predicted is less than the preset threshold, then mark the object to be predicted as an unknown category.

[0016] Optionally, in some embodiments, the preset energy function is:

[0017]

[0018] Where T is the temperature coefficient, f is the feature of the object to be predicted, and g i (f) represents the probability of the object to be predicted in the i-th category, and C represents the total number of categories.

[0019] Optionally, in some embodiments, the feature parameters of the object to be predicted include at least one of the following: 3D depth, length, width, height, and rotation angle of the object to be predicted.

[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for open-set domain generalization of monocular 3D target detection as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the method for open-set domain generalization of monocular 3D target detection as described in the above embodiments.

[0022] Therefore, by acquiring at least one labeled image of a source domain and preprocessing it to obtain a processed labeled image, and based on a preset Deformable DETR object detector, feature extraction and bounding box prediction are performed on the processed labeled image to obtain the feature parameters of the object to be predicted, the category and size of the bounding box of the object to be predicted, and based on the feature parameters, the category and size of the bounding box of the object to be predicted, a classification result of the object to be predicted is generated using a preset energy function. The final 3D bounding box is obtained based on the classification result and the feature parameters of the object to be predicted, and object detection is performed based on the final 3D bounding box. This solves the problem that related monocular 3D object detection technologies do not consider domain generalization, resulting in significant performance degradation when the test domain shifts, leading to risks and security vulnerabilities. Furthermore, when encountering new categories during deployment, they cannot be correctly rejected, leading to misidentification. This application is designed to address domain generalization, significantly improving the generalization effect in unknown domains, and also has open set detection capabilities, enabling it to identify new unknown categories as unknown.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0025] Figure 1 This is a flowchart of an open-set domain generalization method for monocular 3D target detection according to an embodiment of this application;

[0026] Figure 2This is a flowchart of an open-set domain generalization method for monocular 3D object detection according to an embodiment of this application;

[0027] Figure 3 This is a block diagram of an open-set domain generalization device for monocular 3D target detection provided in an embodiment of this application;

[0028] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0029] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0030] The following description, with reference to the accompanying drawings, describes a method, apparatus, electronic device, and storage medium for open-set domain generalization of monocular 3D target detection according to embodiments of this application.

[0031] Before introducing the method for open set domain generalization of monocular 3D object detection in the embodiments of this application, we will first introduce the domain shift phenomenon in monocular 3D object detection training in related technologies.

[0032] In monocular 3D object detection training and testing, the domain offset encountered typically falls into the following categories:

[0033] 1. Domain Shift Due to Climate: The same scene and target object can cause a huge domain shift depending on the climate. For example, a model trained on the Cityscapes dataset will encounter domain shift when tested on the FoggyCityscapes dataset.

[0034] 2. Domain offset due to different angles and devices: In real-world scenarios, there may be significant domain offsets between images captured from different angles or by different camera devices.

[0035] 3. Domain Shift Between Real and Synthetic Images: Due to the difficulty of collecting images, many autonomous driving models use artificially synthesized datasets for training. However, artificially synthesized datasets have a significant domain shift compared to real images. For example, a model trained on the SIM10k dataset synthesized using a 3D engine will experience performance degradation when trained on a real dataset.

[0036] To address the aforementioned issues, this application acquires at least one labeled image of a source domain, preprocesses it to obtain a processed labeled image, and then performs feature extraction and bounding box prediction on the processed labeled image based on a pre-defined Deformable DETR object detector. This yields the feature parameters of the object to be predicted, the category and size of the bounding box, and based on these parameters, a pre-defined energy function is used to generate a classification result for the object. Finally, a 3D bounding box is obtained based on the classification result and the object's feature parameters, and then object detection is performed using this final 3D bounding box. This solves the problem that related monocular 3D object detection techniques do not consider domain generalization, resulting in significant performance degradation and security risks when the test domain shifts. Furthermore, when encountering new categories during deployment, they cannot correctly reject them, leading to misidentification. This application is designed to address domain generalization, significantly improving generalization performance in unknown domains and possessing open set detection capabilities, enabling it to identify new unknown categories as unknown.

[0037] Specifically, Figure 1 This is a flowchart illustrating an open-set domain generalization method for monocular 3D target detection provided in an embodiment of this application.

[0038] like Figure 1 As shown, the method for open-set domain generalization of monocular 3D object detection includes the following steps:

[0039] In step S101, at least one source domain labeled image is acquired, and the labeled image of at least one source domain is preprocessed to obtain a processed labeled image.

[0040] Preprocessing may include resizing, cropping, and one or more data augmentation techniques.

[0041] As one possible implementation, embodiments of this application can preprocess labeled images from multiple source domains. During the data processing stage, various data augmentation techniques can be used to preprocess labeled images from at least one source domain to obtain processed labeled images.

[0042] Data augmentation methods may include:

[0043] 1. Flip data augmentation. Flip flips an image horizontally, which is very simple and can be widely used in monocular 3D object detection.

[0044] 2. Gaussian noise, Gaussian blur, image contrast adjustment, and image sharpening. Adding Gaussian noise to an image can prevent the neural network from overfitting to high-frequency noise, effectively improving the generalization ability and robustness of the neural network to images; adding Gaussian blur improves the detection ability of the neural network; adjusting the contrast and sharpening the image can simulate the image's behavior under varying lighting conditions and changes in camera equipment's digital signal processor hardware, improving the neural network's generalization ability in unknown domains.

[0045] 3. Mixup Data Augmentation. Mixup data augmentation uses linear interpolation to obtain new sample data. Let (x n ,y n (x) is the new data generated by interpolation. i ,y i ) and (x j ,y j If ) are two data points randomly selected from the training set, then the data generation method is as follows, where λ takes the value of a real number between 0 and 1.

[0046] (x n ,y n )=λ(x i ,y i )+(1-λ)(x j ,y j );

[0047] When using Mixup data augmentation, this application embodiment fuses two images from different source domains to obtain sample points with continuous differences from multiple source domains. This enables the neural network to learn that foreground objects can appear in any background, reducing foreground-background dependence and enhancing the neural network's ability to detect foreground objects in different environments and backgrounds, as well as the robustness of 3D attribute prediction such as length, width, height, depth, and rotation angle.

[0048] Furthermore, embodiments of this application can also use Dir-mixup to perform continuous interpolation on samples from three or more source domains, significantly improving the distribution range of samples in the sample space.

[0049] It's important to note that when using Mixup data augmentation, especially Dir-mixup, the synthesized image is a weighted average of the original images. This can cause foreground objects to be faded in the synthesized image. Using these faded objects in neural network training can enhance the learning ability of the neural network, but it can also introduce potential noise. When calculating the loss function for the foreground objects in the Mixup-augmented image during neural network training, the weights can be appropriately reduced to mitigate the impact of noise.

[0050] Of the above methods, Flip is a geometric transformation data augmentation that alters the geometric position of objects in an image. Gaussian noise, Gaussian blur, image contrast adjustment, and image sharpening are also included. Mixup is a geometrically invariant data augmentation. The data augmentation used in the embodiments of this application can be implemented using the open-source library imgaug.

[0051] In step S102, based on the preset Deformable DETR object detector, feature extraction and bounding box prediction are performed on the processed labeled image to obtain the feature parameters of the object to be predicted, the bounding box category and size of the object to be predicted in the processed labeled image.

[0052] Optionally, in some embodiments, the feature parameters of the object to be predicted include at least one of the following: 3D depth, length, width, height, and rotation angle of the object to be predicted.

[0053] Specifically, in this embodiment of the application, after preprocessing the labeled image of at least one source domain, the Deformable DETR object detector is used to extract features and predict bounding boxes for the preprocessed image.

[0054] Understandably, to achieve monocular 3D object detection, in addition to predicting the size and category of the detection box, each object query also predicts the object's 3D depth, length, width, height, and rotation angle, thereby ultimately achieving monocular 3D object detection.

[0055] This application's embodiments can improve the accuracy of monocular 3D object detection by employing geometric information priors. During the research process, it was found that since monocular 3D object detection uses only one image as input, a single image naturally lacks 3D depth information. Therefore, the most crucial aspect of monocular 3D object detection is the prediction of the object's depth value. This application's embodiments use geometric information to assist in predicting the depth value. Specifically, for an object, the predicted 2D bounding box height h of the object is... 2D and 3D height h 3D Then, the object depth d = fh is calculated using the camera focal length f. 3D / h 2D .

[0056] Geometric information-assisted depth calculation has the following advantages:

[0057] 1. Compared to neural networks that predict depth directly from images, geometric information utilizes geometric priors, resulting in more accurate depth calculations.

[0058] 2. In domain generalization tasks, the target domain and the source domain are very different. Such differences will have a significant negative impact on the direct prediction of depth. Geometric priors are independent of data distribution and are domain-independent, and have stronger generalization and robustness in cross-domain tasks.

[0059] For cross-domain generalization of depth prediction using geometric priors, only the detection accuracy of the object's 2D and 3D height values ​​needs to be considered. This greatly reduces the difficulty of cross-domain generalization and makes it easier to use more data augmentation techniques.

[0060] Optionally, in some embodiments, based on a preset Deformable DETR object detector, feature extraction and bounding box prediction are performed on the processed labeled image, which further includes: extracting features from the preprocessed labeled image using the feature extractor of DETR in the preset Deformable DETR object detector to obtain feature extraction results; processing the feature extraction results through a gradient inversion layer (GRL) to obtain processing results; and using a preset neighborhood discriminator to identify the processing results to obtain feature parameters of the object to be predicted.

[0061] Those skilled in the art should understand that, in order to enable the neural network to possess domain invariance and to make the features extracted by the neural network from multiple source domains spatially similar, the embodiments of this application may employ adversarial learning to enhance the feature extraction generalization of the neural network.

[0062] Specifically, in combination Figure 2 As shown, embodiments of this application can employ a discriminator to distinguish features from different domains. The discriminator aims to differentiate DETR features from different domains as much as possible, while the DETR feature extractor aims to make the feature distributions from different domains as similar as possible, so that the discriminator cannot distinguish them. In a specific implementation, a gradient inversion layer (GRL) can be added after the DETR feature extractor and before the discriminator.

[0063] In step S103, based on the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted, a classification result of the object to be predicted is generated using a preset energy function, and the final 3D bounding box is obtained based on the classification result and the feature parameters of the object to be predicted, so as to perform target detection based on the final 3D bounding box.

[0064] Optionally, in some embodiments, generating a classification result of the object to be predicted using a preset energy function based on the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted, further includes: calculating the total energy of the object to be predicted using a preset energy function based on the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted; determining whether the total energy of the object to be predicted is less than a preset threshold; if the total energy of the object to be predicted is less than the preset threshold, then marking the object to be predicted as an unknown category.

[0065] In practice, it is also necessary to identify location categories not encountered in the training set, labeling unknown categories as "unknown" and rejecting them. To achieve unknown category discovery, an energy function can be used, which assigns higher energy values ​​to known categories and lower energy values ​​to unknown categories.

[0066] Specifically, in this application embodiment, the Helmholtz free energy formula can be used. Let the feature of a test object be f, and the logits of the test object in the i-th category be gi(f). Then the energy function of the test object is defined as E(f; g):

[0067]

[0068] Where T is the temperature coefficient.

[0069] For the object to be tested, if the total energy is less than the threshold C, the target is identified as unknown; otherwise, it is identified as a known category.

[0070] Therefore, combined Figure 2 As shown, based on the feature parameters of the object to be predicted, such as the 3D depth, length, width, height, rotation angle, detection box category and size of the object to be predicted, a classification result of the object to be predicted is generated using a preset energy function, and the final 3D bounding box is obtained based on the classification result and the feature parameters of the object to be predicted, so as to perform target detection based on the final 3D bounding box.

[0071] According to the method for open-set domain generalization of monocular 3D object detection proposed in the embodiments of this application, at least one source domain labeled image is acquired, and the labeled image of at least one source domain is preprocessed to obtain a processed labeled image. Based on a preset Deformable DETR object detector, feature extraction and detection box prediction are performed on the processed labeled image to obtain the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted in the processed labeled image. Based on the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted, a preset energy function is used to generate the classification result of the object to be predicted. The final 3D bounding box is obtained according to the classification result and the feature parameters of the object to be predicted, so as to perform object detection based on the final 3D bounding box. Therefore, the existing monocular 3D target detection technologies have not considered domain generalization, which leads to a significant performance drop when the test domain shifts, resulting in risks and security vulnerabilities. Furthermore, when encountering new categories during deployment, they cannot be correctly rejected, leading to misidentification and other problems. This application is designed to address domain generalization, which can significantly improve the generalization effect on unknown domains. It also has open set detection capabilities, enabling it to identify new unknown categories as unknown.

[0072] Next, referring to the accompanying drawings, an open-set domain generalization device for monocular 3D target detection according to an embodiment of this application is described.

[0073] Figure 3 This is a block diagram of an open-set domain generalization device for monocular 3D target detection according to an embodiment of this application.

[0074] like Figure 3 As shown, the monocular 3D target detection open set domain generalization device 10 includes: an acquisition module 100, a prediction module 200, and a generation module 300.

[0075] The system includes an acquisition module 100, which acquires at least one labeled image of a source domain and preprocesses the labeled image of the at least one source domain to obtain a processed labeled image; a prediction module 200, which performs feature extraction and bounding box prediction on the processed labeled image based on a preset Deformable DETR object detector to obtain feature parameters of the object to be predicted, the bounding box category and size of the object to be predicted; and a generation module 300, which generates a classification result of the object to be predicted using a preset energy function based on the feature parameters of the object to be predicted, the bounding box category and size of the object to be predicted, and obtains a final 3Dbbox based on the classification result and the feature parameters of the object to be predicted, so as to perform object detection based on the final 3Dbbox.

[0076] Optionally, in some embodiments, the prediction module 200 further includes an extraction unit.

[0077] The extraction unit is used to extract features from the preprocessed labeled image using the feature extractor of the DETR in the preset Deformable DETR object detector to obtain the feature extraction result; the generation unit is used to process the feature extraction result through the gradient inversion layer GRL to obtain the processing result, and to use a preset neighborhood discriminator to identify the processing result to obtain the feature parameters of the object to be predicted.

[0078] Optionally, in some embodiments, the generation module 300 further includes a calculation unit and a tagging unit.

[0079] The calculation unit is used to calculate the total energy of the object to be predicted based on the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted, and using the preset energy function.

[0080] The labeling unit is used to determine whether the total energy of the object to be predicted is less than a preset threshold. If the total energy of the object to be predicted is less than the preset threshold, the object to be predicted is labeled as an unknown category.

[0081] Optionally, in some embodiments, the preset energy function is:

[0082]

[0083] Where T is the temperature coefficient, f is the feature of the object to be predicted, and g i (f) represents the probability of the object to be predicted in the i-th category, and C represents the total number of categories.

[0084] Optionally, in some embodiments, the feature parameters of the object to be predicted include at least one of the following: 3D depth, length, width, height, and rotation angle of the object to be predicted.

[0085] It should be noted that the explanation of the above-described embodiment of the open set domain generalization method for monocular 3D target detection also applies to the open set domain generalization device for monocular 3D target detection in this embodiment, and will not be repeated here.

[0086] According to the open-set domain generalization device for monocular 3D target detection proposed in the embodiments of this application, the device acquires at least one labeled image of a source domain and preprocesses the labeled image of the at least one source domain to obtain a processed labeled image. Based on a preset Deformable DETR target detector, the device extracts features and predicts bounding boxes on the processed labeled image to obtain the feature parameters of the object to be predicted, the category and size of the bounding box of the object to be predicted, and the classification result of the object to be predicted is generated using a preset energy function based on the feature parameters of the object to be predicted, the category and size of the bounding box of the object to be predicted, and the final 3D bounding box is obtained according to the classification result and the feature parameters of the object to be predicted, so as to perform target detection based on the final 3D bounding box. Therefore, the existing monocular 3D target detection technologies have not considered domain generalization, which leads to a significant performance drop when the test domain shifts, resulting in risks and security vulnerabilities. Furthermore, when encountering new categories during deployment, they cannot be correctly rejected, leading to misidentification and other problems. This application is designed to address domain generalization, which can significantly improve the generalization effect on unknown domains. It also has open set detection capabilities, enabling it to identify new unknown categories as unknown.

[0087] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0088] The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0089] When the processor 402 executes the program, it implements the method for open set domain generalization of monocular 3D target detection provided in the above embodiments.

[0090] Furthermore, electronic devices also include:

[0091] Communication interface 403 is used for communication between memory 401 and processor 402.

[0092] The memory 401 is used to store computer programs that can run on the processor 402.

[0093] The memory 401 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0094] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0095] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0096] Processor 402 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of this application.

[0097] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for open-set domain generalization of monocular 3D target detection.

[0098] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0099] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0100] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0101] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0102] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0103] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for open-set domain generalization in monocular 3D target detection, characterized in that, Includes the following steps: Obtain labeled images of at least one source domain, and preprocess the labeled images of the at least one source domain to obtain processed labeled images; Based on a preset Deformable DETR object detector, feature extraction and bounding box prediction are performed on the processed labeled image to obtain the feature parameters of the object to be predicted in the processed labeled image, the bounding box category and size of the object to be predicted; as well as Based on the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted, a classification result of the object to be predicted is generated using the preset energy function. Then, a final 3D bounding box is obtained based on the classification result and the feature parameters of the object to be predicted, and target detection is performed based on the final 3D bounding box. Based on a preset Deformable DETR object detector, feature extraction and bounding box prediction are performed on the processed labeled image, including: The preprocessed labeled image is used to extract features using the feature extractor of DETR in the preset Deformable DETR object detector to obtain the feature extraction result; The feature extraction results are processed through a gradient inversion layer (GRL) to obtain the processing result, and a preset neighborhood discriminator is used to identify the processing result to obtain the feature parameters of the object to be predicted. The step of generating a classification result for the object to be predicted based on its feature parameters, the detection box category and size, and using the preset energy function further includes: Based on the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted, the total energy of the object to be predicted is calculated using the preset energy function; Determine whether the total energy of the object to be predicted is less than a preset threshold. If the total energy of the object to be predicted is less than the preset threshold, then mark the object to be predicted as an unknown category. The preset energy function is: ； Where T is the temperature coefficient. Features of the object to be predicted For the object to be predicted in the th The probability of each category, where C is the total number of categories.

2. The method according to claim 1, characterized in that, The feature parameters of the object to be predicted include at least one of the following: 3D depth, length, width, height, and rotation angle of the object to be predicted.

3. A device for open-set domain generalization of monocular 3D target detection, characterized in that, include: An acquisition module is used to acquire at least one source domain labeled image and preprocess the at least one source domain labeled image to obtain a processed labeled image; The prediction module is used to perform feature extraction and bounding box prediction on the processed labeled image based on a preset Deformable DETR object detector, so as to obtain the feature parameters of the object to be predicted in the processed labeled image, the bounding box category and size of the object to be predicted; as well as A generation module is used to generate a classification result for the object to be predicted based on its feature parameters, the detection box category and size, and a preset energy function. It then obtains a final 3D bounding box based on the classification result and the object's feature parameters, and performs object detection based on the final 3D bounding box. The prediction module further includes: The extraction unit is used to extract features from the preprocessed labeled image using the feature extractor of DETR in the preset Deformable DETR target detector, and obtain the feature extraction result. The generation unit is used to process the feature extraction result through the gradient inversion layer (GRL) to obtain the processing result, and to use a preset neighborhood discriminator to identify the processing result to obtain the feature parameters of the object to be predicted. The generation module further includes: The calculation unit is used to calculate the total energy of the object to be predicted based on the feature parameters of the object to be predicted, the detection box category and size of the object to be predicted, and using the preset energy function. A labeling unit is used to determine whether the total energy of the object to be predicted is less than a preset threshold. If the total energy of the object to be predicted is less than the preset threshold, the object to be predicted is labeled as an unknown category. The preset energy function is: ； Where T is the temperature coefficient. Features of the object to be predicted For the object to be predicted in the th The probability of each category, where C is the total number of categories.

4. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method for open-set domain generalization of monocular 3D object detection as described in any one of claims 1-2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method for open-set domain generalization of monocular 3D object detection as described in any one of claims 1-2.

Images