An open world three-dimensional object detection method and device

Abstract

This application discloses an open-world 3D target detection method and apparatus, comprising: acquiring multimodal data of a target scene; performing 3D target detection on 3D point cloud data to obtain 3D candidate targets; inputting the geometric features of each 3D candidate target into an out-of-distribution target classifier to determine whether the 3D candidate target belongs to a known category set and to identify unknown targets; projecting the 3D candidate targets corresponding to the unknown targets onto 2D image data, obtaining the 2D image region, and inputting it into a visual language model, using natural language prompts to guide the visual language model to output the semantic category name of the unknown targets; fusing the semantic category name and the spatial location information of the 3D candidate targets to generate open-world 3D detection results. This application can effectively identify unknown 3D targets in an open world and generate open-world 3D detection results that combine spatial positioning and semantic description.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, specifically to an open-world 3D target detection method and apparatus. Background Technology

[0002] 3D object detection is a core technology for machine perception of the 3D environment. It primarily involves integrating multimodal data, such as 3D point clouds acquired by LiDAR and 2D images acquired by visual cameras, to perform detection, enabling spatial localization and category recognition of objects within a scene. This provides fundamental support for intelligent decision-making processes such as autonomous driving path planning and autonomous robot navigation. Current mainstream 3D object detection technologies are all based on closed-type sets for model training and inference, meaning the models can only recognize a limited number of known object categories predefined during the training phase.

[0003] However, in real-world open-world scenarios such as autonomous driving and intelligent robot operations, the diversity and uncertainty of object categories within the environment result in a large number of unknown objects outside the model training distribution. Existing 3D object detection technologies cannot effectively identify these unknown objects, making it difficult to adapt to the dynamic perception requirements of open scenarios. Furthermore, existing technologies can only utilize multimodal data to detect objects of known categories, failing to effectively combine spatial location and semantic information for unknown targets. They cannot output detection results that combine accurate 3D spatial positioning and effective category description, thus failing to meet the practical application requirements of 3D object detection in open-world scenarios. Therefore, existing technologies suffer from the problem of being unable to identify unknown targets and generate effective detection results.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] This application provides an open-world 3D target detection method and apparatus that can effectively identify unknown 3D targets in an open world and generate open-world 3D detection results that combine spatial positioning and semantic description.

[0006] In a first aspect, embodiments of this application provide an open-world 3D target detection method, including: Acquire multimodal data of the target scene, wherein the multimodal data includes at least three-dimensional point cloud data and two-dimensional image data; Three-dimensional target detection is performed on the three-dimensional point cloud data to obtain at least one three-dimensional candidate target. Each three-dimensional candidate target includes corresponding spatial location information, geometric features and initial classification confidence. The geometric features of each of the three-dimensional candidate targets are input into a pre-trained out-of-distribution target classifier to determine whether the three-dimensional candidate target belongs to a known category set in order to identify unknown targets; Project the three-dimensional candidate target corresponding to the unknown target onto the two-dimensional image data to obtain the corresponding two-dimensional image region; The two-dimensional image region is input into a pre-trained visual language model, and natural language prompts are used to guide the visual language model to output the semantic category name of the unknown target. The semantic category name and the spatial location information of the corresponding 3D candidate target are fused to generate the final open-world 3D detection result.

[0007] Secondly, embodiments of this application provide an open-world 3D target detection device, comprising: The data acquisition module is used to acquire multimodal data of the target scene, wherein the multimodal data includes at least three-dimensional point cloud data and two-dimensional image data; The target detection module is used to perform three-dimensional target detection on the three-dimensional point cloud data to obtain at least one three-dimensional candidate target. Each three-dimensional candidate target includes corresponding spatial location information, geometric features and initial classification confidence. The target classification module is used to input the geometric features of each of the three-dimensional candidate targets into a pre-trained out-of-distribution target classifier to determine whether the three-dimensional candidate targets belong to a known category set, so as to identify unknown targets; The target projection module is used to project the three-dimensional candidate target corresponding to the unknown target onto the two-dimensional image data to obtain the corresponding two-dimensional image region; The semantic category module is used to input the two-dimensional image region into a pre-trained visual language large model, and use natural language prompts to guide the visual language large model to output the semantic category name of the unknown target; The result generation module is used to fuse the semantic category name and the spatial location information of the corresponding 3D candidate target to generate the final open-world 3D detection result.

[0008] This application provides a method and apparatus for open-world 3D target detection. First, multimodal data containing 3D point cloud data and 2D image data is acquired. Then, 3D target detection is performed on the 3D point cloud data to obtain 3D candidate targets. A pre-trained out-of-distribution target classifier is used to effectively identify unknown targets, overcoming the technical limitation of traditional 3D target detection technology, which is restricted by closed category sets and cannot identify out-of-distribution unknown targets. This enables effective differentiation between known and unknown 3D targets in open-world scenes. Simultaneously, the identified unknown targets are projected onto the 2D image data to obtain corresponding 2D image regions. A pre-trained visual language model, combined with natural language prompts, is used to match accurate semantic category names for the unknown targets, addressing the lack of effective semantic category descriptions for unknown targets in open worlds. Finally, the semantic category name of the unknown target is fused with its 3D spatial location information to generate the final open-world 3D detection result. This result possesses both accurate 3D spatial positioning attributes and clear semantic category attributes, enabling the 3D target detection result to simultaneously meet the dual requirements of target spatial positioning and semantic recognition in open-world scenes. Therefore, it can effectively adapt to practical application scenarios in open worlds such as autonomous driving and intelligent robots. In summary, this application effectively overcomes the limitation of closed category sets in traditional 3D target detection by integrating 3D geometric perception and a large visual language model, enabling the identification and semantic category matching of unknown 3D targets in the open world. Furthermore, it fuses the semantic category name with the 3D spatial location information of the unknown target to generate open-world 3D detection results that combine spatial positioning and semantic description. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is an application environment diagram of the open-world 3D target detection method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the open-world 3D target detection method provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the open-world detection system provided in this application embodiment; Figure 4 This is a flowchart illustrating the target positioning module provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the open-world 3D target detection device provided in the embodiments of this application; Figure 6This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0011] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of systems and methods consistent with those detailed in the appended claims or with some aspects of this application.

[0012] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover descriptions such as non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0013] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0014] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0015] To address the aforementioned technical problems and overcome the shortcomings of existing technologies, this application provides an open-world 3D target detection method and apparatus, which can effectively identify unknown 3D targets in an open world and generate open-world 3D detection results that combine spatial positioning and semantic description.

[0016] Figure 1 This is a diagram illustrating the application environment of an open-world 3D object detection method in one embodiment. (Refer to...) Figure 1This open-world 3D object detection method is applied to an open-world 3D object detection system. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; a mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of multiple servers. Server 120 is configured to execute the above-described open-world 3D object detection method, including: acquiring multimodal data of the target scene, the multimodal data including at least 3D point cloud data and 2D image data; performing 3D object detection on the 3D point cloud data to obtain at least one 3D candidate object, each 3D candidate object including corresponding spatial location information, geometric features and initial classification confidence; inputting the geometric features of each 3D candidate object into a pre-trained out-of-distribution object classifier to determine whether the 3D candidate object belongs to a known category set in order to identify unknown objects; projecting the 3D candidate object corresponding to the unknown object into the 2D image data to obtain the corresponding 2D image region; inputting the 2D image region into a pre-trained visual language large model, using natural language prompts to guide the visual language large model to output the semantic category name of the unknown object; fusing the semantic category name and the spatial location information of the corresponding 3D candidate object to generate the final open-world 3D detection result.

[0017] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of an open-world 3D object detection method provided in this application. This embodiment primarily uses the application of this open-world 3D object detection method to a computer device as an example for illustration. Specifically, the open-world 3D object detection method provided in an embodiment of this application may include the following steps: S1. Acquire multimodal data of the target scene, which includes at least 3D point cloud data and 2D image data; Specifically, for step S1, the target scene can be an autonomous driving road environment, an intelligent robot operating environment, or other scenes requiring 3D target perception. Multimodal data includes 3D point cloud data and 2D image data. The 3D point cloud data is typically acquired by LiDAR; a single frame of point cloud data contains information about several points, each with 3D spatial coordinates and reflection intensity characteristics, accurately representing the 3D geometric shape and spatial position of objects within the scene. The 2D image data is simultaneously acquired by a visual camera and is used to present the texture, color, appearance, and other visual semantic information of objects within the scene. Both provide multi-dimensional data support for subsequent 3D target detection and semantic recognition.

[0018] For example, in autonomous driving scenarios, vehicle-mounted LiDAR collects 3D point cloud data of targets such as vehicles, pedestrians, and roadside obstacles on the road, while vehicle-mounted vision cameras simultaneously collect high-definition 2D visual images of the same scene. Together, they constitute the multimodal data of the driving scenario.

[0019] S2. Perform 3D target detection on the 3D point cloud data to obtain at least one 3D candidate target. Each 3D candidate target includes corresponding spatial location information, geometric features and initial classification confidence. Specifically, in step S2, professional 3D target detection processing is performed on the collected 3D point cloud data. A 3D target detection algorithm is used to extract features, identify targets, and locate them in the disordered point cloud data. All suspected targets within the scene are identified from the point cloud data and designated as 3D candidate targets. Each 3D candidate target carries three types of core information: spatial location information, which characterizes the target's specific location in 3D space and may include parameters such as the center point coordinates of the 3D bounding box and yaw angle; geometric features, which characterize the target's 3D shape and may include features such as length, width, height, overall outline, and attitude; and initial classification confidence, which is numerical and represents the probability that the 3D candidate target is classified into a known category.

[0020] For example, after performing 3D target detection on point cloud data of an autonomous driving scenario, three 3D candidate targets are identified: a car, a pedestrian, and a cone-shaped obstacle on the road. Each target has corresponding 3D bounding box coordinates, geometric features such as length / width / height / attitude, and initial classification confidence values ​​for the car and pedestrian.

[0021] S3. Input the geometric features of each 3D candidate target into a pre-trained out-of-distribution target classifier to determine whether the 3D candidate target belongs to a known category set in order to identify unknown targets; Specifically, in step S3, the geometric features of each obtained 3D candidate target are individually input into the classifier. The classifier analyzes the input geometric features based on preset judgment logic, determining whether the corresponding 3D candidate target belongs to a predefined set of known categories (such as vehicles, pedestrians, and traffic signs in autonomous driving scenarios). Ultimately, it distinguishes between known and unknown targets, filtering and identifying all unknown targets. The out-of-distribution target classifier used in this step is a professionally trained model capable of distinguishing between known and unknown targets, accurately identifying targets outside the preset set of known categories.

[0022] For example, the geometric features of three 3D candidate targets—a car, a pedestrian, and a cone-shaped obstacle—are input into an out-of-distribution target classifier. The classifier determines that the car and the pedestrian belong to the known category set, while the cone-shaped obstacle does not belong to the known category set. Therefore, the cone-shaped obstacle is identified as an unknown target.

[0023] S4. Project the three-dimensional candidate target corresponding to the unknown target onto the two-dimensional image data to obtain the corresponding two-dimensional image region; Specifically, for step S4, for the identified unknown target, a spatial mapping relationship between its three-dimensional candidate target and two-dimensional image data is established. Through professional coordinate transformation, projection calculation and image cropping methods, the unknown target candidate in three-dimensional space is accurately projected onto the synchronously acquired two-dimensional image plane to determine the position range of the unknown target in the two-dimensional image. The image region within this position range is then cropped from the original two-dimensional image data. This region fully contains the visual features of the unknown target, providing visual material for subsequent semantic category recognition.

[0024] For example, the three-dimensional candidate target of the identified cone-shaped obstacle is transformed to the camera coordinate system of the vehicle camera through coordinate system transformation, and then the three-dimensional to two-dimensional projection calculation is completed using the camera intrinsic parameters. The rectangular region corresponding to the cone-shaped obstacle is determined in the two-dimensional image, and then the rectangular region is cropped from the original high-definition image.

[0025] S5. Input the two-dimensional image region into the pre-trained visual language model, and use natural language prompts to guide the visual language model to output the semantic category name of the unknown target; Specifically, in step S5, the cropped 2D image region of the unknown target is input into the model, along with customized natural language prompts. These prompts guide and constrain the model's recognition task, focusing it on the target scene and recognition requirements. Based on the visual features of the input image region and the guidance of the prompts, the model completes the visual recognition and semantic matching of the unknown target, ultimately outputting the semantic category name corresponding to the unknown target. The large-scale visual language model used in this step is a multimodal model pre-trained on massive amounts of data, possessing visual perception and language understanding capabilities, used to achieve accurate matching of visual features and semantic categories.

[0026] For example, the cropped 2D image region of the cone-shaped obstacle is input into a pre-trained visual language model, along with the natural language prompt "In a vehicle driving scenario, what is the object in the image? Output only the object name". After analysis, the model outputs "construction cone" as the semantic category name of the unknown target.

[0027] S6. The semantic category name and the corresponding spatial location information of the 3D candidate target are fused to generate the final open-world 3D detection result; Specifically, for step S6, the obtained semantic category name of the unknown target and its corresponding three-dimensional spatial location information are integrated and associated. The semantic category name is bound to the original three-dimensional spatial location parameters of the unknown target to form a complete detection result that combines three-dimensional spatial positioning information and clear semantic category description. This result can directly reflect the two core information of "where" and "what" of the unknown target in the open world scene, which is the final open world three-dimensional detection result.

[0028] For example, the semantic category name of "construction cone" is fused with the spatial location information such as the coordinates of the 3D bounding box and the yaw angle of the target to generate the final open-world 3D detection result of "3D bounding box (center point x: xx, y: xx, z: xx; length / width / height: xx / xx / xx; yaw angle: xx) + construction cone".

[0029] This embodiment, based on multimodal data of 3D point cloud and 2D image, sequentially completes 3D candidate target extraction, unknown target recognition, image region mapping, semantic category matching, and information fusion. It breaks through the limitation of traditional 3D target detection relying on closed category sets, and can realize the recognition and semantic annotation of unknown 3D targets in open world scenes, and finally outputs a complete detection result with both spatial positioning and semantic description.

[0030] Furthermore, in some embodiments, step S2, "performing 3D target detection on the 3D point cloud data to obtain at least one 3D candidate target," may specifically include: S21. Preprocess the 3D point cloud data to generate voxel feature tensors; Specifically, for step S21, the original 3D point cloud data is an unordered set of discrete points collected by LiDAR. Each point contains four features: 3D spatial coordinates (x, y, z) and reflection intensity. This type of unordered data cannot be directly processed by the neural network, so it needs to be standardized and preprocessed first. The core of the preprocessing is voxelization. First, a fixed 3D voxel size is set for the target scene, and the 3D space is divided into a regular voxel grid. Then, all points in the original point cloud are assigned to the corresponding voxels. Feature aggregation is performed on all points falling within the same voxel. Aggregation methods can include coordinate mean, maximum reflection intensity, feature stitching, etc. Finally, the unordered point cloud data is converted into a 3D voxel feature tensor with regular structure and dense features. This tensor can perfectly adapt to the input requirements of the subsequent neural network, preparing the data for feature extraction.

[0031] S22. Input the voxel feature tensor into the pre-trained 3D target detection backbone network to extract multi-scale feature maps; Specifically, for step S22, the 3D object detection backbone network used is a dedicated network trained with a large-scale known-class point cloud dataset. It adopts an Anchor-Free architecture based on center points, and its internal structure includes a collaborative structure of an encoder and a decoder, which can efficiently extract the geometric and semantic features of the point cloud. After inputting the voxel feature tensor obtained by preprocessing into this backbone network, first, the sparse 3D convolutional network in the encoder downsamples the voxel features multiple times, gradually compressing the feature space while extracting deep features with rich high-level semantic information. Subsequently, the decoder gradually restores the spatial resolution of the features through upsampling operations, and combines with the skip connection mechanism to fuse the shallow geometric features and deep semantic features extracted at different downsampling stages of the encoder. Finally, a high-resolution feature map that combines multi-scale features is output. Each spatial position of this feature map corresponds to a specific region in the original point cloud, and simultaneously contains the low-level geometric details and high-level semantic information of this region, providing comprehensive feature support for subsequent object detection.

[0032] S23. Based on the multi-scale feature map, generate 3D candidate objects through the detection head, as well as the corresponding spatial position information, geometric features, and initial classification confidence. Specifically, for step S23, the detection head is an object recognition and parameter regression module配套 with the backbone network, which can parallelly complete three major tasks of object localization, parameter regression, and class confidence prediction based on the multi-scale feature map. First, the detection head processes the multi-scale feature map through convolutional layers to predict and generate the object center point heat map. The peak position in the heat map is the corresponding position of the object center in the scene on the feature map, thereby achieving the preliminary localization of the object. Secondly, at the center point position corresponding to each peak, the detection head directly predicts the parameters related to the 3D bounding box of the object through a dedicated regression head, and simultaneously extracts the geometric shape features of the object. Finally, the detection head calculates the probability values of the object belonging to each preset known class through the classification head, and selects the maximum probability value as the initial classification confidence of the object. Finally, integrate the localization results, 3D bounding box parameters, geometric features, and initial classification confidence of each object to form independent 3D candidate objects. Each 3D candidate object carries unique and corresponding spatial position information, geometric features, and initial classification confidence. Among them, the spatial position information mainly includes parameters such as the center point coordinates and yaw angle of the 3D bounding box, which characterize the position and orientation of the object in 3D space; the geometric features mainly include parameters such as the length, width, height, and overall pose of the 3D bounding box, which characterize the shape of the object; the initial classification confidence is a value between 0 and 1, which characterizes the probability that the object is determined to be a certain known class.

[0033] This embodiment preprocesses the 3D point cloud by voxelization, extracts multi-scale features using a 3D detection backbone network, and then uses the detection head to complete target localization and parameter prediction. This generates 3D candidate targets carrying precise spatial location, geometric features, and initial classification confidence, providing standardized and reliable basic data for subsequent unknown target discrimination.

[0034] Furthermore, in some embodiments, step S3, "inputting the geometric features of each 3D candidate target into a pre-trained out-of-distribution target classifier to determine whether the 3D candidate target belongs to a known category set, so as to identify unknown targets," may specifically include: S31. Construct the comprehensive feature vector corresponding to each 3D candidate target; Specifically, in step S31, a comprehensive feature vector is constructed for each 3D candidate target to fully and multidimensionally represent its attributes, providing complete and accurate feature basis for subsequent determination of unknown targets. During the construction process, features reflecting the core attributes of candidate targets are extracted from different stages of 3D target detection, including features representing the visual and geometric context of the target region, geometric features representing the target's shape and posture, and classification-related features representing the target's classification result. These extracted multidimensional features are then fused to form a unified comprehensive feature vector. This vector integrates perceptual, geometric, and classification information of candidate targets, fully reflecting the target's feature patterns and avoiding the limitations of single-dimensional features on the determination result, thus making subsequent determination of unknown targets more accurate.

[0035] S32. Input the comprehensive feature vector into the out-of-distribution target classifier to obtain the probability value of the three-dimensional candidate target belonging to the unknown target; Specifically, in step S32, after inputting the individually constructed comprehensive feature vector for each 3D candidate target into the classifier, the classifier performs deep feature analysis and pattern matching on the feature vector. Based on the feature distribution patterns of known target categories learned during training, it calculates and outputs the probability value of the 3D candidate target belonging to an unknown target. This probability value is a value between 0 and 1. The higher the value, the greater the probability that the 3D candidate target belongs to an unknown target outside the training distribution; the lower the value, the greater the probability that it belongs to a known target category. The out-of-distribution target classifier used in this step is a professionally trained model with the ability to distinguish between known and unknown target feature patterns. It can accurately identify the feature patterns contained in the comprehensive feature vector and distinguish the feature differences between targets in the known category set and out-of-distribution targets.

[0036] S33. Compare the probability value with the preset probability threshold. If the probability value is greater than the preset probability threshold, then determine that the three-dimensional candidate target is an unknown target. Specifically, for step S33, the preset probability threshold is a judgment criterion pre-set based on factors such as the actual 3D target detection application scenario and detection accuracy requirements. The core principle of setting this threshold is to ensure the accuracy of the judgment of known category targets, and to ensure that known targets are not misclassified as unknown targets as much as possible. The unknown target probability value of each 3D candidate target output by the out-of-distribution target classifier is compared one-to-one with the preset probability threshold. The final judgment of the unknown target is completed based on the comparison result. If the unknown target probability value of a certain 3D candidate target is greater than the preset probability threshold, it means that the comprehensive feature pattern of the target is significantly different from the feature pattern of the known category set targets, and exceeds the feature distribution range of the known category set. Therefore, the 3D candidate target is judged as an unknown target. If the probability value is less than or equal to the preset probability threshold, the 3D candidate target is judged as a known category target.

[0037] This embodiment calculates the comprehensive feature vector of three-dimensional candidate targets, calculates the unknown probability through an out-of-distribution target classifier and compares it with a threshold, which can objectively and accurately distinguish unknown targets from candidate targets, improving the accuracy and stability of known and unknown target recognition in open scenes.

[0038] Furthermore, in some embodiments, step S31, "constructing a comprehensive feature vector corresponding to each three-dimensional candidate target," may specifically include: S311. Extract the backbone features corresponding to the center point positions of the three-dimensional candidate targets from the three-dimensional target detection backbone network; Specifically, in step S311, the multi-scale feature map output by the 3D object detection backbone network integrates low-level geometric details and high-level semantic information of the target region, and is the core feature representing the visual and geometric context of the 3D candidate target. In this step, the center point of the 3D candidate target is first projected back to the spatial position corresponding to the feature map output by the backbone network. Then, through bilinear interpolation, the feature values ​​of all feature channels at that position are extracted, and these feature values ​​are integrated to form the backbone feature. This backbone feature can truly reflect the original visual features and geometric environment context of the region where the 3D candidate target is located, and is an important basic feature representing the target attributes.

[0039] S312. Input the geometric features of the 3D candidate target into the fully connected layer, and map the geometric features to the high-dimensional semantic space through linear transformation to obtain geometric encoding features; Specifically, in step S312, the original geometric features of the 3D candidate target are the basic parameters representing the target's shape and posture. Direct use of these features is insufficient to accurately reflect the anomalies in its shape and posture, and cannot meet the feature requirements for subsequent unknown target determination. In this step, the geometric features, including the length, width, height, posture, and bounding box parameters of the 3D candidate target, are input into a pre-defined fully connected layer (linear layer). This fully connected layer encodes and upscales the original geometric features, mapping them from the original low-dimensional feature space to a higher-dimensional semantic space (e.g., 64-dimensional). The geometrically encoded features obtained after linear transformation can more accurately and meticulously represent the characteristic attributes of the target's shape and posture, and can also better reflect the anomalies in the target's geometric shape, becoming an important geometric basis for distinguishing known and unknown targets.

[0040] S313. The original classification logical value corresponding to the initial classification confidence of the three-dimensional candidate target and the one-hot encoding of the corresponding predicted category are concatenated to obtain the concatenated feature. The concatenated feature is then input into the fully connected layer for fusion transformation to obtain the classification logical feature. Specifically, in step S313, the original classification logical value is the original calculated value for classifying candidate targets during 3D target detection, while the one-hot encoding of the predicted category is a vector that digitally represents the predicted category of the candidate target. The combination of these two values ​​reflects the detection model's "cognitive certainty" regarding the candidate target. In this step, the original classification logical value corresponding to the initial classification confidence of the 3D candidate target is first extracted, and then the one-hot encoded category vector of its predicted category is obtained. These two types of data are concatenated dimensionally to form a concatenated feature. This concatenated feature is then input into a fully connected layer, and through fusion and transformation processing by the network layers, the classification logical feature is obtained. This feature clearly reflects the detection model's judgment state regarding target classification. Unknown targets often exhibit a flat distribution of logical values ​​or characteristics contradicting the predicted category on this feature, making it a key basis for determining whether a target belongs to an unknown category.

[0041] S314. Integrate the backbone features, geometric coding features, and classification logic features to form a comprehensive feature vector; Specifically, in step S314, the backbone features, geometric coding features, and classification logic features represent the core attributes of the 3D candidate target from three dimensions: visual geometric context, shape and pose attributes, and classification cognitive certainty, respectively. A single feature cannot fully reflect the overall feature pattern of the target. In this step, the backbone features, geometric coding features, and classification logic features of the same 3D candidate target obtained in the previous steps are concatenated and fused, integrating features from three different dimensions and representation directions into a unified feature vector, i.e., a comprehensive feature vector. This vector achieves multi-dimensional and comprehensive feature representation of the 3D candidate target, fully integrating key information such as perceptual uncertainty, geometric anomalies, and cognitive contradictions, providing complete and accurate feature basis for subsequent determination of unknown targets.

[0042] This embodiment extracts and fuses features from three dimensions: backbone features, geometric coding features, and classification logic features, to construct a comprehensive feature vector that fully represents the target attributes. This solves the problem of one-sided representation by a single feature and provides a complete feature basis for the accurate determination of unknown targets in the future.

[0043] Furthermore, in some embodiments, the training method for the out-of-distribution target classifier may specifically include: Select real object samples of known categories from the training data, and apply geometric deformation to some of the real object samples to generate synthetic unknown samples; Extract comprehensive feature vectors from real object samples and synthetic unknown samples respectively, and assign them to known category labels and unknown category labels respectively; The out-of-distribution target classifier is trained using the labeled comprehensive feature vector, and the network parameters of the out-of-distribution target classifier are optimized to obtain the trained out-of-distribution target classifier.

[0044] Specifically, this embodiment also provides a training method for an out-of-distribution target classifier, the specific process of which is as follows: First, the training data is a 3D object detection training dataset constructed for a pre-defined set of known categories. The selected real object samples are all valid samples within the known categories, containing complete feature information for each type of known target, and serve as the foundational samples for training the out-of-distribution target classifier. To address the difficulty of collecting and labeling unknown target samples in real-world scenes, geometric deformation is applied to selected real object samples of known categories. By altering the 3D geometry of the samples, their feature patterns deviate from the feature distribution of the known category set, thus simulating the features of unknown targets outside the training distribution and generating synthetic unknown samples. The geometric deformation employs anisotropic scaling, independently selecting scaling factors in different dimensions (length, width, and height). This allows for either shrinking or enlarging the samples, with a higher probability of choosing shrinking transformations. This simulates the geometric anomalies of unknown targets such as "incomplete objects" and "small objects" commonly found in real-world scenes, making the generated synthetic unknown samples more closely resemble real-world open-world scenarios.

[0045] Then, comprehensive feature vectors that can represent the characteristic attributes of the two types of samples in a multi-dimensional and comprehensive manner are extracted. These feature vectors integrate core information such as the visual geometric context, shape and pose attributes, and classification cognition-related features of the samples. They can clearly reflect the feature differences between known category samples and synthetic unknown samples, and serve as the core input features for training the out-of-distribution target classifier. After feature extraction, uniform label assignment is performed on the two types of samples. Following the label setting rules of supervised training, all real object samples of known categories are assigned known category labels, and all synthetic unknown samples generated by geometric deformation are assigned unknown category labels. The labels are numerical identifiers that can be recognized by the classifier, forming a one-to-one correspondence between "comprehensive feature vector - category label" training sample pairs. This constructs a complete supervised training dataset that meets the training data requirements of the out-of-distribution target classifier.

[0046] Finally, supervised learning is used to train the out-of-distribution target classifier. The out-of-distribution target classifier is a network model adapted for binary classification tasks involving known and unknown targets. It can perform feature analysis and pattern matching on the input comprehensive feature vector and output the probability value of a sample being an unknown target. During training, the constructed labeled comprehensive feature vector training set is input into the classifier. The classifier predicts each input comprehensive feature vector and outputs the probability value that the sample belongs to an unknown target. Then, using a preset loss function, the error between the classifier's prediction result and the sample's true class label is calculated. Based on this error, a gradient descent-like optimization algorithm is used for backpropagation, adjusting the classifier's network parameters layer by layer to continuously reduce the prediction error. This training process is iterated until the classifier's prediction accuracy on the validation set reaches the preset performance requirements, at which point training stops. The resulting out-of-distribution target classifier has completed parameter optimization and possesses stable decision-making capabilities.

[0047] This embodiment generates synthetic unknown samples by applying geometric deformation to samples of known categories, constructs labeled training data, and conducts supervised training on an out-of-distribution target classifier. This solves the problem of the scarcity of real unknown samples and enables the trained classifier to have a stable and accurate ability to distinguish between known and unknown targets.

[0048] Furthermore, in some embodiments, step S4, "projecting the three-dimensional candidate target corresponding to the unknown target onto the two-dimensional image data to obtain the corresponding two-dimensional image region," may specifically include: S41. Transform the three-dimensional candidate target corresponding to the unknown target from the world coordinate system or vehicle coordinate system to the camera coordinate system to obtain the three-dimensional corner coordinates in the camera coordinate system; Specifically, in step S41, the 3D candidate target of the unknown target is defined by the center point coordinates, size parameters, and orientation parameters. Its initial spatial coordinates are based on the world coordinate system or vehicle coordinate system and cannot be directly used for projection onto the 2D image. A coordinate system transformation must be performed first. In this step, all 3D corner points (usually 8) of the 3D candidate target are extracted first. Then, using the extrinsic parameters of the coordinate system transformation, including the rotation matrix and translation vector of the corresponding coordinate system, the coordinates of each corner point are linearly transformed sequentially. First, the corner point coordinates in the world coordinate system are transformed to the vehicle coordinate system, and then further transformed to the camera sensor coordinate system. During the transformation, the orientation difference between coordinate systems is eliminated by the rotation matrix, and the position offset between coordinate systems is eliminated by the translation vector. Finally, the 3D coordinates of all corner points in the camera coordinate system are obtained, laying the spatial coordinate foundation for the subsequent 3D to 2D projection.

[0049] S42. Using the camera intrinsic parameter matrix, project the three-dimensional corner points in the camera coordinate system onto the image plane to obtain a set of two-dimensional projection points; Specifically, in step S42, the camera intrinsic parameter matrix is ​​an inherent parameter of the camera, containing core information such as focal length and principal point coordinates. It is the key basis for realizing the projection of three-dimensional spatial points onto a two-dimensional image plane. In this step, the coordinates of each three-dimensional corner point in the camera coordinate system are substituted into the projection formula based on the camera intrinsic parameter matrix. The depth dimension in the three-dimensional coordinates is eliminated by the formula, and the pixel coordinates corresponding to each three-dimensional corner point on the two-dimensional image plane are obtained, i.e., the two-dimensional projection points. During the calculation, it is necessary to ensure that the depth value of the three-dimensional corner point in the camera coordinate system is greater than 0, ensuring that the point is within the camera's field of view. Finally, a set of two-dimensional projection points corresponding one-to-one with the three-dimensional corner points are obtained, realizing the mapping from three-dimensional spatial points to two-dimensional image points.

[0050] S43. Calculate the axis-aligned 2D bounding box based on 2D projection points; Specifically, in step S43, the axis-aligned two-dimensional bounding box can accurately define the overall range of the two-dimensional projection point set on the image plane, determining the basic region for subsequent image cropping. In this step, extreme value calculations are performed on the pixel coordinates of all obtained two-dimensional projection points to find the minimum and maximum values ​​of all projection points in the horizontal direction (u-axis) and the vertical direction (v-axis). The minimum and maximum values ​​in the horizontal direction are used as the left and right boundaries of the bounding box, and the minimum and maximum values ​​in the vertical direction are used as the top and bottom boundaries of the bounding box. Based on this, the coordinates of the four vertices of the axis-aligned two-dimensional bounding box are determined. This bounding box is a rectangle parallel to the image coordinate axes, which can completely enclose all two-dimensional projection points, and preliminarily determine the position range of the unknown target in the two-dimensional image.

[0051] S44. Perform boundary expansion or size normalization on the two-dimensional bounding box to obtain the optimized two-dimensional bounding box; Specifically, in step S44, the initial axis-aligned 2D bounding box only fits the projection point set, which can easily lead to problems such as overly tight clipping regions, truncated object edges, or some boxes being too small or having abnormal aspect ratios. This fails to provide a high-quality visual region for subsequent semantic recognition, thus requiring optimization. In this step, boundary expansion or size normalization operations are performed on the initial bounding box according to actual detection needs: boundary expansion involves uniformly expanding the top, bottom, left, and right boundaries of the box outwards by pixels to ensure that the clipping region contains the complete visual features of the unknown target and surrounding contextual information; size normalization involves proportionally enlarging boxes that are too small and adjusting the proportions of boxes with extreme aspect ratios to make the box size and proportions suitable for the input requirements of the large visual language model. After one or both of these operations, an optimized 2D bounding box is obtained.

[0052] S45. Based on the optimized two-dimensional bounding box, crop out the corresponding two-dimensional image region from the two-dimensional image data; Specifically, for step S45, based on the coordinates of the four vertices determined by the optimized two-dimensional bounding box, the image region within the rectangular coordinate range is cropped from the original two-dimensional image data according to the row and column order of the image pixels. The cropped two-dimensional image region fully contains the main visual features of the unknown target and has an appropriate amount of contextual information. Its size and proportion are suitable for the recognition requirements of the subsequent large visual language model, and it can be directly used as the visual input material of the model.

[0053] This embodiment achieves accurate mapping of unknown targets from three-dimensional space to two-dimensional images through coordinate system transformation, three-dimensional to two-dimensional projection calculation, bounding box generation and optimization processing, and can crop out two-dimensional image regions that contain complete target visual features and are suitable for semantic recognition.

[0054] Furthermore, in some embodiments, the method may further include: S46. If the 3D candidate target is completely behind the camera, or its projection has no effective intersection with the image, or the calculated 2D bounding box size is smaller than the preset size threshold, then the projection is determined to be a failure, and the 3D candidate target is discarded or marked as invisible.

[0055] Specifically, this embodiment also provides a method for determining and processing projection anomalies of unknown 3D candidate targets, the specific process of which is as follows: In the coordinate system transformation stage of projecting an unknown 3D candidate target onto a 2D image, the depth dimension (Z-axis) value of the camera coordinate system directly represents the relative position of the target to the camera. A Z-value greater than 0 indicates that the target is in front of the camera's field of view and can be captured by the camera; a Z-value less than or equal to 0 indicates that the target is behind the camera and cannot be displayed in the 2D image. After obtaining the 3D coordinates of all corner points of the 3D candidate target in the camera coordinate system, the depth values ​​of all corner points are verified. If the depth value of any corner point of the 3D candidate target is ≤0, it can be determined that the target is completely behind the camera. In this case, the projection of the 3D target onto the 2D image is meaningless, and the projection is directly judged as a failure. The 3D candidate target is then discarded, and no further projection, cropping, or other processing is performed.

[0056] After generating the axis-aligned 2D bounding boxes of 3D candidate targets, the spatial overlap between these bounding boxes and the original 2D image is verified to determine if there is a valid croppable image region. The intersection of the initial axis-aligned 2D bounding box and the image boundaries (the pixel coordinate range corresponding to the image width and height) is calculated. If the resulting bounding box shows a left boundary ≥ right boundary, or a top boundary ≥ bottom boundary, or if the convex hull of the projection point set calculated by the convex hull algorithm has no spatial intersection with the image rectangle, it can be determined that the projection of the 3D candidate target has no valid intersection with the image, and the projection fails. Such targets are not discarded; instead, they are marked as "invisible," and relevant log information is recorded. In subsequent detection data streams, these targets are skipped directly without further bounding box optimization, image cropping, or other processing. This marking operation does not affect the normal projection process of other 3D candidate targets.

[0057] The size of the 2D bounding box after projecting the 3D candidate target directly determines the visual information richness of the cropped region. If the box size is too small, the cropped image region will lose the core visual features of the target due to insufficient pixels, failing to provide effective information for subsequent semantic recognition. In this step, the actual width and height (in pixels) of the axis-aligned 2D bounding box generated by the projection are first calculated, and then compared with a preset size threshold. If the width or height of the bounding box is less than the preset size threshold, the projection is deemed to have failed. To address this situation, this embodiment provides a flexible processing strategy. Depending on the needs of the actual detection scenario, the 3D candidate target can be either directly discarded or marked as invisible and skipped from subsequent processing. Both strategies avoid meaningless image cropping and semantic recognition operations.

[0058] This embodiment identifies abnormal projection scenarios such as targets behind the camera, projections without effective intersection, and bounding box sizes that are too small, and then discards or marks these targets to eliminate invalid projection targets, thereby improving the robustness and efficiency of the projection process and ensuring the input quality for subsequent semantic recognition.

[0059] Furthermore, in some embodiments, step S5, "inputting the two-dimensional image region into a pre-trained visual language model, and using natural language prompts to guide the visual language model to output the semantic category name of the unknown target," may specifically include: S51. Construct natural language prompts, which include scene-specific information and output format constraints. Specifically, for step S51, the natural language prompts are the core guidelines for the visual language large-scale model to complete the semantic recognition of unknown targets. Their construction requires consideration of both scene adaptability and output standardization, and core components include scene constraint information and output format constraint information. Scene constraint information is a descriptive content set according to the actual application scenario of 3D object detection, used to focus the model's recognition attention on the object features and category distribution of the target scene, avoiding the model outputting recognition results irrelevant to the scene. Output format constraint information is a clear instruction limiting the model's output format and content length, used to standardize the model's output results, avoiding the generation of lengthy and invalid descriptions, and ensuring that the output is a concise object category name. The overall expression of the prompts is concise and the instructions are clear, enabling the pre-trained visual language large-scale model to quickly understand the recognition task requirements and adapt to the needs of zero-shot recognition.

[0060] S52. Input the two-dimensional image region and natural language prompts together into the pre-trained visual language large model; Specifically, in step S52, the pre-trained visual language model used is a multimodal model trained on massive image-text cross-modal data. It possesses powerful capabilities in visual feature extraction, language feature understanding, and cross-modal feature fusion, enabling zero-shot object semantic recognition. In this step, the cropped unknown target 2D image region and the constructed natural language prompts are used as joint multimodal inputs and simultaneously fed into the visual language model. Upon receiving the input, the model activates the visual encoding module and the language encoding module respectively. It performs pixel-level feature extraction and encoding on the 2D image region to generate visual feature vectors that represent the visual features of the object, and performs word segmentation and semantic encoding on the natural language prompts to generate language feature vectors that represent the requirements of the recognition task, laying the foundation for subsequent cross-modal fusion and semantic reasoning.

[0061] S53. Obtain the text response output by the large visual language model and use it as the semantic category name of the unknown target; Specifically, in step S53, after encoding visual and linguistic features, the pre-trained visual-language large model deeply fuses these two types of features through a cross-modal attention layer to uncover the semantic relationships between them. Based on the fused features, it then performs text generation inference in an autoregressive manner. During inference, the model strictly adheres to the scene constraints and output format limitations in the natural language prompts, focusing on object category determination within the target scene and generating only concise object category names as the text response, without outputting any additional descriptive content. After the model completes text generation, it directly obtains this text response, which is the precise semantic category name corresponding to the unknown target.

[0062] This embodiment guides a pre-trained visual language model to identify unknown target image regions by constructing prompt words with scene limitations and format constraints. It can obtain standardized and accurate semantic category names with zero samples without additional training, thus completing the semantic annotation of unknown targets.

[0063] Furthermore, in some embodiments, step S6, "fusing the semantic category name and the corresponding spatial location information of the 3D candidate target to generate the final open-world 3D detection result," may specifically include: S61. Associate the semantic category name with the spatial location information of the corresponding 3D candidate target; Specifically, in step S61, a one-to-one, uniquely bound association is established between the identified semantic category name of the unknown target and the spatial location information of the corresponding 3D candidate target. This ensures accurate matching between the semantic description and the spatial location, avoiding information misalignment between different targets. The spatial location information consists of the spatial positioning parameters inherent in the 3D candidate target, and the semantic category name is the textual category description of the unknown target. This mapping relationship establishes the correspondence between "what the object is" and "where the object is."

[0064] S62. Add confidence information for unknown targets; Specifically, in step S62, after associating the semantic category name with the spatial location information, a confidence score value representing the reliability of the unknown target determination is appended to the aforementioned association information. This confidence score information is used to quantify the credibility of the unknown target identification result, serving as an intuitive reference indicator for the reliability of the detection result. Adding this information makes the detection information more complete and more valuable for reference.

[0065] S63. Generate 3D detection results containing 3D bounding boxes, semantic labels, and confidence information; Specifically, in step S63, the associated spatial location information is converted into standardized 3D bounding box parameters, the semantic category names are standardized into semantic labels in a unified format, and then combined with the additional confidence information, they are integrated and encapsulated according to a preset structured format to finally form an open-world 3D detection result that simultaneously contains three core contents: 3D bounding boxes, semantic labels, and confidence information.

[0066] This embodiment accurately associates the semantic category name of the unknown target with its three-dimensional spatial location information. After adding confidence information, it generates a structured detection result containing a three-dimensional bounding box, semantic label, and confidence score, making the output information more complete and reliable, and facilitating subsequent decision-making.

[0067] To facilitate understanding of the open-world 3D target detection method provided in this embodiment, this embodiment will be described in detail with reference to the open-world detection system that implements the above-described open-world 3D target detection method. This system includes a target localization module and a target classification module. For example... Figure 3As shown, the system takes 3D point cloud and 2D image as input. First, it uses a 3D out-of-distribution target detector to locate targets in the 3D point cloud and outputs unlabeled 3D target bounding boxes. Then, it projects these 3D bounding boxes onto the 2D image to obtain the corresponding 2D coordinates and completes image region cropping. Then, combined with the prompt "In a vehicle driving scene, what is the target in the image? Just output its name", it guides the visual-language multimodal large model to identify the cropped image region and output the semantic category of the target. Finally, it fuses the unlabeled 3D target bounding boxes with the identified semantic categories to generate a 3D detection result that combines spatial localization and semantic description, realizing 3D target detection in open-world scenes.

[0068] like Figure 4 As shown, the target localization module is mainly implemented by a 3D out-of-distribution target detector. This detector is a two-stage modular system. Its core idea is: first, a pre-trained 3D target detector focused on known categories (called the "base detector") is used to complete the initial perception and localization of the input point cloud; then, a lightweight, trainable "out-of-distribution classifier" performs deep analysis on each proposal output by the base detector to determine whether it belongs to a known distribution (In-Distribution, ID) or an unknown distribution (Out-of-Distribution, OOD). The detailed implementation mechanism and steps are as follows: Step 1: Point Cloud Data Input and Preprocessing Input representation: The input is the raw LiDAR point cloud, typically represented as... ,in This refers to the number of points in a single frame of point cloud, where each point contains three-dimensional spatial coordinates. It has four characteristics, including reflection intensity.

[0069] Voxelization: To facilitate neural network processing, the disordered point cloud is first converted into a regular three-dimensional voxel mesh. A fixed voxel size is set (e.g., 0.1m × 0.1m × 0.2m), and points falling within the same voxel are subjected to feature aggregation (e.g., taking the mean of coordinates, maximum reflection intensity, etc.) to form a dense voxel feature tensor.

[0070] Step 2: 3D Object Detection Backbone Network and Feature Extraction In this invention, an advanced center-point-based 3D object detection architecture (such as a variant of CenterPoint) is employed as the backbone network. This network does not rely on preset anchor boxes, but instead characterizes the probability of object presence by predicting dense center-point heatmaps.

[0071] The backbone network comprises an encoder-decoder structure. The encoder (a sparse 3D convolutional network) progressively downsamples to extract deep features with rich semantics. The decoder (containing upsampling and skip connections) progressively restores the spatial resolution, ultimately outputting a high-resolution multi-channel feature map. This feature map integrates low-level geometric details and high-level semantic information, where each spatial location corresponds to a region in the original point cloud, and C is the number of feature channels.

[0072] Step 3: Generation of 3D Detection Box The base detector is based on the feature map The following two tasks are executed in parallel to generate preliminary detection results: Center point localization: A heatmap is predicted using a convolutional layer, and the peak value is the predicted location of the object's center on the feature map. .

[0073] Attribute Regression: At the center point, another regression head directly predicts the 7-DOF 3D bounding box parameters of the object: center point offset. ,size and yaw angle .

[0074] Finally, a preliminary set of predicted 3D bounding boxes is output. Each of them It includes its spatial attributes and a classification confidence score based on the known categories.

[0075] Step 4: Design of Out-of-Distribution Target Classifier and Feature Monitoring Its goal is to learn patterns that distinguish between ID and OOD samples by utilizing the features and outputs of the base detector.

[0076] Input feature construction: For each detection proposal Features are extracted and fused from the following three levels to form a comprehensive feature vector. : skeletal characteristics ( ): The prediction results Projecting the center point back to the feature map The corresponding position The feature values ​​of all C channels at this location are extracted using bilinear interpolation. These features contain the original visual and geometric context of the region.

[0077] Geometric features ( ): The 7D detection box parameters of the prediction results Encoding and upsizing are performed through a fully connected layer (linear layer), mapping it to a higher-dimensional semantic space (such as 64-dimensional) to better characterize the anomalousness of its shape and pose.

[0078] Classification logical features ( ): The base detector in the detection results The original classification logits output at this point. And compare it with the predicted one-hot encoded class vector. The data is then stitched together. Subsequently, it is fused and transformed through a fully connected layer. This feature reflects the "cognitive certainty" of the base detector regarding the proposal; OOD objects often exhibit a flat distribution of logistic values ​​or contradict the predicted category.

[0079] Feature fusion: The three sets of feature vectors mentioned above are simply concatenated. =[ , , It makes full use of perceptual uncertainty (backbone features), geometric anomalies (frame features), and cognitive contradictions (classification logic) to provide multi-dimensional basis for OOD judgment.

[0080] Classifier Network: A lightweight multilayer perceptron is used as the out-of-distribution (OOD) classifier. Its structure consists of three fully connected layers, each followed by a ReLU activation function and a Dropout layer (with a dropout rate of approximately 30%) to prevent overfitting. The network ultimately outputs a scalar value g(x)∈[0,1], activated by a Sigmoid function, representing the probability that the predicted target is OOD.

[0081] Step 5: Training and Reasoning Process Training phase: Base detector fixing: First, a high-performance 3D object detector (base detector) is trained independently on a large known class dataset (nuScenes), and its weights are frozen after completion.

[0082] Synthetic Anomaly Generation: To train an out-of-distribution classifier, OOD samples need to be created. This invention employs a data augmentation strategy based on geometric deformation to synthesize virtual anomalies: From the real-world object annotations in the training set, a subset of object instances are randomly selected, and their point clouds and corresponding 3D bounding boxes are subjected to random, large-amplitude anisotropic scaling. The scaling factor is independently and randomly selected in each dimension; for example, the width, length, and height take values ​​within the range of [0.1, 0.5] (shrinking) or [1.5, 3.0] (enlarging), respectively, with a higher probability (e.g., 80%) of shrinking transformation to simulate more common "incomplete" or "small object" anomalies. These drastically deformed objects are considered as synthetic OOD samples.

[0083] Feature extraction and label assignment: For ID objects, the label is 0; for synthetic OOD objects, the label is 1.

[0084] Loss function and optimization: The MLP classifier is trained using binary cross-entropy loss. The optimization objective is to make the classifier output a low probability value for ID samples and a high probability value for synthetic OOD samples.

[0085] Reasoning stage: The input point cloud is processed by a fixed base detector to generate predicted bounding box proposals. .

[0086] For each prediction proposal Extract fusion features using the method described above. .

[0087] Will Input a pre-trained out-of-distribution classifier (MLP) to obtain an OOD probability score. .

[0088] Set a threshold (This is typically determined by ensuring a 95% accuracy rate for ID samples on the validation set). If If the condition is met, the proposal is determined to be an OOD target; otherwise, it is an ID target.

[0089] The system ultimately outputs the attributes of each detected target: 3D target detection box parameters, known category label (or "unknown" identifier), and its confidence level as an OOD target.

[0090] For the object classification module, the core task is to accurately project the output 3D object detection boxes (3DBoundingBoxes) from the object localization module onto the 2D image plane and crop the corresponding regions for processing by the Visual Language Model (VLM). The specific steps of the algorithm are as follows: Step 1: Coordinate system transformation of the 3D bounding box (from world coordinate system to camera coordinate system) First, transform the 3D bounding box of the target in the world coordinate system to the camera coordinate system. Let a 3D bounding box be defined by its center point. ,size directional quaternions (i.e., yaw angle) (represented by quaternions in the code) is defined.

[0091] Transformation to the ego vehicle coordinate system: Utilizing the extrinsic parameters of the ego pose (i.e., the vehicle's position and orientation in the world). Let the rotation matrix of the ego pose be... The translation vector is Then point The coordinates in the ego coordinate system are: For the entire 3D frame, transform all its corner points (usually 8) according to this formula.

[0092] Transformation to the camera sensor coordinate system: Using camera calibration extrinsic parameters. Let the rotation matrix of the camera relative to ego be... The translation vector is Then point The coordinates in the camera coordinate system are: After these two steps, we obtained the set of corner coordinates of the 3D bounding box in the camera coordinate system. , where each point .

[0093] Step 2: Projection of 3D points onto a 2D image Using the camera's intrinsic parameter matrix Project 3D points in the camera coordinate system onto the image plane. The intrinsic parameter matrix is ​​in the form of: in Focal length The coordinates of the main point.

[0094] For each 3D point The normalized image coordinates after projection $ The calculation is as follows: Right now: here The depth of the point (must be greater than 0, and the point must be in front of the camera).

[0095] Step 3: Generation and Post-processing of 2D Bounding Boxes Obtain the projected coordinates of all 8 corner points { After that, calculate the axis-aligned bounding box (AABB): Initial 2D bounding box coordinates obtained: .

[0096] Then, find the intersection with the image boundary, where the image size is... To ensure the bounding box is within the image, calculate the effective intersection: like or If the bounding box and the image have no valid intersection, the projection is considered to have failed.

[0097] Next, convex hull clipping optimization is performed. To handle cases where some bounding boxes are outside the image, a geometric method is used: the projected point set is treated as a two-dimensional point set, its convex hull is calculated, and then the convex hull and the image rectangle are compared. The intersection of the two bounding rectangles is the final optimized 2D bounding box. If the intersection is empty, the bounding box is not visible.

[0098] Step 4: Image region cropping Obtain the final 2D bounding box coordinates After (among them) ), from the original image Medium cropping rectangle area: This area will serve as the visual input for the VLM.

[0099] During the projection and cropping process, various abnormal situations may occur. This system also incorporates the following handling mechanisms to ensure robustness.

[0100] 1. Handling Process for Projection Failure: Projection failure mainly occurs when the 3D bounding box fails to form a valid 2D region on the image plane. The judgment conditions and handling process are as follows: Condition 1: The frame is completely behind the camera. After the coordinate system transformation, the depth of any corner point of the cube If the bounding box is behind the camera and not visible, the action is to discard the bounding box without further cropping or recognition.

[0101] Condition 2: The bounding box and the image do not intersect. As mentioned earlier, if the optimized 2D bounding box coordinates are invalid (e.g., width or height is negative), or the convex hull does not intersect with the image boundary, the procedure is as follows: mark the bounding box as "invisible" and log it. Skip this bounding box in the data stream; it will not affect the processing of other bounding boxes.

[0102] Condition 3: Frame size is too small The width or height of the 2D frame after projection is less than the threshold. (For example, 4 pixels), possibly due to excessive distance or occlusion. Handling: The system provides two optional strategies: 2. Frame Optimization Strategy To improve the quality of the cropping area, the following optimization strategies are adopted: Boundary expansion: To prevent overly tight cropping from truncating parts of the object, the final 2D bounding box is uniformly expanded. Let the expansion number be... (Default 10 pixels), then the new coordinates are calculated as follows: The expanded version can contain more contextual information, which helps with VLM recognition.

[0103] Minimum Size Guarantee: If the cropped image area is too small (e.g., width or height ≤ 28 pixels), it will be proportionally enlarged to avoid difficulties in VLM recognition due to low resolution. Let the original size be (w, h), and the target minimum size be... Magnification factor The new size is High-quality interpolation algorithms are used for resampling to preserve image details.

[0104] Aspect Ratio Constraints: Extreme aspect ratios (such as very narrow or very flat frames) can distort the appearance of an object. The system allows setting a range for aspect ratios. (For example, 0.5 to 2.0). If the aspect ratio of the frame exceeds this range, it will be adjusted to the target ratio while keeping the center point unchanged.

[0105] In addition, this system also provides specific strategies for VLM prompt word engineering. Prompt words are crucial for guiding VLM to perform zero-shot recognition. This system employs a hierarchical, configurable prompt word design: Basic template: Optimized for driving scenarios, for example: "What is the object in the picture in a vehicle driving scenario? Just output its name." This template includes: Scenario limitation: "inavehicledrivingscenario" focuses VLM's attention on the road environment.

[0106] The task instruction, "What is the object?", explicitly requires object identification.

[0107] Output format constraint: "Justoutputitsname" forces the model to output concise category names, avoiding the generation of lengthy descriptions.

[0108] Dynamic enhancement: The prompt words are dynamically adjusted based on contextual information, and the 3D detector provides category confidence. And OOD rating The prompt can be expanded to: "An object detected with confidence" andnoveltyscore .Whatisthisobjectinadrivingscene?Outputonlythename." The zero-shot inference implemented in this embodiment refers to the model directly processing unseen categories or scenes without fine-tuning with specific task data. Its feasibility is based on pre-trained knowledge generalization: the VLM learns massive amounts of image-text pairs during training, constructing cross-modal alignment between visual concepts and linguistic descriptions. For example, although it wasn't specifically trained to recognize "construction cones," the model has seen images and descriptions of similar objects during pre-training and can infer them through semantic similarity. Furthermore, based on prompts as task guidance, natural language prompts "express" the recognition task in a common form used during model pre-training (such as image question answering), activating the model's relevant knowledge modules.

[0109] Model selection: A large-scale pre-trained VLM (such as Qwen-VL) is adopted, which has powerful vision-language alignment capabilities and supports open vocabulary recognition.

[0110] Input construction: cropped image region With natural language prompts As a multimodal input, the model internally encodes images into visual features and text into linguistic features, which are then fused in a cross-modal attention layer.

[0111] Inference Mechanism: The model generates text responses through autoregression based on fused features. Essentially, it calculates conditional probabilities. The most likely word sequence is selected as the output. Because the pre-training data is extensive, the model can generate semantically relevant category names outside the training set.

[0112] Knowledge Utilization: VLM's "knowledge" comes from association patterns in the pre-training corpus. For example, if a prompt mentions "driving scenario," the model will implicitly call upon the distribution of concepts related to roads and vehicles, thereby improving the relevance of the recognition.

[0113] In addition, this system also provides a result fusion module, which spatially aligns and fuses the semantic labels returned by VLM with the 3D detection boxes to generate 3D detection results with natural language descriptions, in the format of: 3D box coordinates + category label + confidence score.

[0114] In summary, the open-world 3D object detection method provided in this embodiment can overcome the limitations of closed category sets, identify objects not seen in the training set, and improve the robustness of the system in dynamic environments. Through the zero-shot capability of VLM, it provides natural language descriptions for unknown objects, supports advanced decision-making, such as autonomous driving path planning. In fields such as autonomous driving and robotics, the identification of unknown objects enhances safety redundancy and reduces the risk of accidents.

[0115] To facilitate better implementation of the open-world 3D object detection method of this application, this application also provides an open-world 3D object detection device based on the above-described open-world 3D object detection method. The meanings of the terms used are the same as in the above-described open-world 3D object detection method, and specific implementation details can be found in the descriptions within the method embodiments.

[0116] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of the open-world 3D target detection device provided in an embodiment of this application. The open-world 3D target detection device may specifically include a data acquisition module 201, a target detection module 202, a target classification module 203, a target projection module 204, a semantic category module 205, and a result generation module 206, as follows: The data acquisition module 201 is used to acquire multimodal data of the target scene, which includes at least three-dimensional point cloud data and two-dimensional image data. The target detection module 202 is used to perform three-dimensional target detection on three-dimensional point cloud data to obtain at least one three-dimensional candidate target. Each three-dimensional candidate target includes corresponding spatial location information, geometric features and initial classification confidence. The target classification module 203 is used to input the geometric features of each three-dimensional candidate target into a pre-trained out-of-distribution target classifier to determine whether the three-dimensional candidate target belongs to a known category set in order to identify unknown targets; The target projection module 204 is used to project the three-dimensional candidate target corresponding to the unknown target onto the two-dimensional image data to obtain the corresponding two-dimensional image region; The semantic category module 205 is used to input two-dimensional image regions into a pre-trained visual language large model and use natural language prompts to guide the visual language large model to output the semantic category name of the unknown target. The result generation module 206 is used to fuse the semantic category name and the spatial location information of the corresponding 3D candidate target to generate the final open-world 3D detection result.

[0117] Furthermore, in some embodiments, the target detection module 202 is specifically used for: Preprocess the 3D point cloud data to generate voxel feature tensors; The voxel feature tensor is input into a pre-trained 3D target detection backbone network to extract multi-scale feature maps; Based on multi-scale feature maps, a 3D candidate target is generated by a detection head, along with its corresponding spatial location information, geometric features, and initial classification confidence.

[0118] Furthermore, in some embodiments, the target classification module 203 is specifically used for: Construct a comprehensive feature vector for each 3D candidate target; The comprehensive feature vector is input into the out-of-distribution target classifier to obtain the probability value of the three-dimensional candidate target belonging to the unknown target; The probability value is compared with a preset probability threshold. If the probability value is greater than the preset probability threshold, the three-dimensional candidate target is determined to be an unknown target.

[0119] Furthermore, in some embodiments, the target classification module 203 is specifically used for: Extract backbone features corresponding to the center point positions of 3D candidate targets from the 3D target detection backbone network; The geometric features of the 3D candidate target are input into the fully connected layer, and the geometric features are mapped to the high-dimensional semantic space through linear transformation to obtain geometrically encoded features; The original classification logical value corresponding to the initial classification confidence of the 3D candidate target and the one-hot encoding of the corresponding predicted category are concatenated to obtain the concatenated feature. The concatenated feature is then input into the fully connected layer for fusion transformation to obtain the classification logical feature. The core features, geometric coding features, and classification logic features are integrated to form a comprehensive feature vector.

[0120] Furthermore, in some embodiments, a classifier training module is also included, specifically for: Select real object samples of known categories from the training data, and apply geometric deformation to some of the real object samples to generate synthetic unknown samples; Extract comprehensive feature vectors from real object samples and synthetic unknown samples respectively, and assign them to known category labels and unknown category labels respectively; The out-of-distribution target classifier is trained using the labeled comprehensive feature vector, and the network parameters of the out-of-distribution target classifier are optimized to obtain the trained out-of-distribution target classifier.

[0121] Furthermore, in some embodiments, the target projection module 204 is specifically used for: Transform the 3D candidate target corresponding to the unknown target from the world coordinate system or vehicle coordinate system to the camera coordinate system to obtain the 3D corner coordinates in the camera coordinate system; Using the camera intrinsic parameter matrix, the three-dimensional corner points in the camera coordinate system are projected onto the image plane to obtain a set of two-dimensional projection points; Calculate axis-aligned 2D bounding boxes based on 2D projection points; The boundary of the 2D bounding box is expanded or its size is normalized to obtain an optimized 2D bounding box. The corresponding two-dimensional image region is cropped from the two-dimensional image data based on the optimized two-dimensional bounding box.

[0122] Furthermore, in some embodiments, the target projection module 204 is specifically used for: If the 3D candidate target is completely behind the camera, or its projection has no effective intersection with the image, or the calculated 2D bounding box size is smaller than a preset size threshold, then the projection is deemed a failure, and the 3D candidate target is discarded or marked as invisible.

[0123] Furthermore, in some embodiments, the semantic category module 205 is specifically used for: Construct natural language prompts, which include scene-specific information and output format constraints. Two-dimensional image regions and natural language prompts are input together into a pre-trained visual language model; Obtain the text response output by the large visual language model and use it as the semantic category name of the unknown target.

[0124] Furthermore, in some embodiments, the result generation module 206 is specifically used for: Associate semantic category names with the spatial location information of the corresponding 3D candidate targets; Add confidence information for unknown targets; Generate 3D detection results that include 3D bounding boxes, semantic labels, and confidence information.

[0125] For specific limitations regarding the open-world 3D object detection device, please refer to the limitations of the open-world 3D object detection method above, which will not be repeated here. Each module in the aforementioned open-world 3D object detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0126] The open-world 3D target detection device provided in this embodiment effectively breaks through the limitation of closed category sets in traditional 3D target detection by integrating 3D geometric perception and a large visual language model. It realizes the identification and semantic category matching of unknown 3D targets in the open world, and integrates the semantic category name and the 3D spatial location information of the unknown target to generate an open-world 3D detection result that combines spatial positioning and semantic description.

[0127] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 6 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will understand that... Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 301 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 302, and by calling data stored in the memory 302, thereby providing overall monitoring of the electronic device. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 301.

[0128] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and open-world 3D object detection methods by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0129] The electronic device also includes a power supply 303 that supplies power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 303 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0130] The electronic device may also include an input unit 304, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0131] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302 to realize various functions, as follows: Acquire multimodal data of the target scene, including at least 3D point cloud data and 2D image data; perform 3D target detection on the 3D point cloud data to obtain at least one 3D candidate target, each 3D candidate target including corresponding spatial location information, geometric features, and initial classification confidence; input the geometric features of each 3D candidate target into a pre-trained out-of-distribution target classifier to determine whether the 3D candidate target belongs to a known category set, in order to identify unknown targets; project the 3D candidate target corresponding to the unknown target onto the 2D image data to obtain the corresponding 2D image region; input the 2D image region into a pre-trained visual language model, and use natural language prompts to guide the visual language model to output the semantic category name of the unknown target; fuse the semantic category name and the spatial location information of the corresponding 3D candidate target to generate the final open-world 3D detection result.

[0132] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0133] This application embodiment effectively breaks through the limitation of closed category sets in traditional 3D target detection by integrating 3D geometric perception and visual language large model, realizes the identification and semantic category matching of unknown 3D targets in open world, and integrates the semantic category name and the 3D spatial location information of unknown targets to generate open world 3D detection results that combine spatial positioning and semantic description.

[0134] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0135] Therefore, embodiments of this application provide a storage medium storing multiple instructions that can be loaded by a processor to execute steps in any of the open-world 3D object detection methods provided in embodiments of this application. For example, the instructions can execute the following steps: Acquire multimodal data of the target scene, including at least 3D point cloud data and 2D image data; perform 3D target detection on the 3D point cloud data to obtain at least one 3D candidate target, each 3D candidate target including corresponding spatial location information, geometric features, and initial classification confidence; input the geometric features of each 3D candidate target into a pre-trained out-of-distribution target classifier to determine whether the 3D candidate target belongs to a known category set, in order to identify unknown targets; project the 3D candidate target corresponding to the unknown target onto the 2D image data to obtain the corresponding 2D image region; input the 2D image region into a pre-trained visual language model, and use natural language prompts to guide the visual language model to output the semantic category name of the unknown target; fuse the semantic category name and the spatial location information of the corresponding 3D candidate target to generate the final open-world 3D detection result.

[0136] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0137] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0138] Since the instructions stored in the storage medium can execute the steps of any of the open-world 3D target detection methods provided in the embodiments of this application, the beneficial effects that any of the open-world 3D target detection methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0139] The above provides a detailed description of an open-world 3D target detection method and apparatus provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An open-world 3D target detection method, characterized in that, include: Acquire multimodal data of the target scene, wherein the multimodal data includes at least three-dimensional point cloud data and two-dimensional image data; Three-dimensional target detection is performed on the three-dimensional point cloud data to obtain at least one three-dimensional candidate target. Each three-dimensional candidate target includes corresponding spatial location information, geometric features and initial classification confidence. The geometric features of each of the three-dimensional candidate targets are input into a pre-trained out-of-distribution target classifier to determine whether the three-dimensional candidate target belongs to a known category set in order to identify unknown targets; Project the three-dimensional candidate target corresponding to the unknown target onto the two-dimensional image data to obtain the corresponding two-dimensional image region; The two-dimensional image region is input into a pre-trained visual language model, and natural language prompts are used to guide the visual language model to output the semantic category name of the unknown target. The semantic category name and the spatial location information of the corresponding 3D candidate target are fused to generate the final open-world 3D detection result.

2. The open-world 3D target detection method according to claim 1, characterized in that, The step of performing 3D target detection on the 3D point cloud data to obtain at least one 3D candidate target includes: The three-dimensional point cloud data is preprocessed to generate voxel feature tensors; The voxel feature tensor is input into a pre-trained 3D target detection backbone network to extract multi-scale feature maps; Based on the multi-scale feature map, a three-dimensional candidate target, along with its corresponding spatial location information, geometric features, and initial classification confidence, is generated by the detection head.

3. The open-world 3D target detection method according to claim 1, characterized in that, The step of inputting the geometric features of each of the three-dimensional candidate targets into a pre-trained out-of-distribution target classifier to determine whether the three-dimensional candidate targets belong to a known category set in order to identify unknown targets includes: Construct a comprehensive feature vector for each 3D candidate target; The comprehensive feature vector is input into the out-of-distribution target classifier to obtain the probability value that the three-dimensional candidate target belongs to the unknown target; The probability value is compared with a preset probability threshold. If the probability value is greater than the preset probability threshold, the three-dimensional candidate target is determined to be an unknown target.

4. The open-world 3D target detection method according to claim 3, characterized in that, The construction of the comprehensive feature vector corresponding to each 3D candidate target includes: Extract the backbone features corresponding to the center point positions of the three-dimensional candidate targets from the three-dimensional target detection backbone network; The geometric features of the three-dimensional candidate target are input into a fully connected layer, and the geometric features are mapped to a high-dimensional semantic space through linear transformation to obtain geometrically encoded features; The original classification logical value corresponding to the initial classification confidence of the three-dimensional candidate target and the one-hot encoding of the corresponding predicted category are concatenated to obtain the concatenated feature. The concatenated feature is then input into a fully connected layer for fusion transformation to obtain the classification logical feature. The backbone features, the geometric coding features, and the classification logic features are fused to form the comprehensive feature vector.

5. The open-world 3D target detection method according to claim 1, characterized in that, The training method for the out-of-distribution target classifier includes: Select real object samples of known categories from the training data, and apply geometric deformation to some of the real object samples to generate synthetic unknown samples; Extract comprehensive feature vectors from the real object samples and the synthetic unknown samples respectively, and assign them to known category labels and unknown category labels respectively; The out-of-distribution target classifier is trained using the labeled comprehensive feature vector, and the network parameters of the out-of-distribution target classifier are optimized to obtain the trained out-of-distribution target classifier.

6. The open-world 3D target detection method according to claim 1, characterized in that, The step of projecting the three-dimensional candidate target corresponding to the unknown target onto the two-dimensional image data to obtain the corresponding two-dimensional image region includes: The three-dimensional candidate target corresponding to the unknown target is transformed from the world coordinate system or vehicle coordinate system to the camera coordinate system to obtain the three-dimensional corner coordinates in the camera coordinate system; Using the camera intrinsic parameter matrix, the three-dimensional corner points in the camera coordinate system are projected onto the image plane to obtain a set of two-dimensional projection points; Calculate the axis-aligned two-dimensional bounding box based on the two-dimensional projection points; The two-dimensional bounding box is subjected to boundary expansion or size normalization to obtain an optimized two-dimensional bounding box. The corresponding two-dimensional image region is cropped from the two-dimensional image data based on the optimized two-dimensional bounding box.

7. The open-world 3D target detection method according to claim 6, characterized in that, The method further includes: If the 3D candidate target is completely behind the camera, or its projection has no effective intersection with the image, or the calculated 2D bounding box size is smaller than a preset size threshold, then the projection is determined to be a failure, and the 3D candidate target is discarded or marked as invisible.

8. The open-world 3D target detection method according to claim 1, characterized in that, The step of inputting the two-dimensional image region into a pre-trained visual language model, and using natural language prompts to guide the visual language model to output the semantic category name of the unknown target, includes: Construct natural language prompts, which include scene-specific information and output format constraints. The two-dimensional image region and the natural language prompt are input together into a pre-trained visual language large model; Obtain the text response output by the large visual language model and use it as the semantic category name of the unknown target.

9. The open-world 3D target detection method according to claim 1, characterized in that, The process of fusing the semantic category name and the corresponding spatial location information of the 3D candidate target to generate the final open-world 3D detection result includes: Associate the semantic category name with the spatial location information of the corresponding three-dimensional candidate target; Add confidence information for the unknown target; Generate 3D detection results that include 3D bounding boxes, semantic labels, and confidence information.

10. An open-world 3D target detection device, characterized in that, include: The data acquisition module is used to acquire multimodal data of the target scene, wherein the multimodal data includes at least three-dimensional point cloud data and two-dimensional image data; The target detection module is used to perform three-dimensional target detection on the three-dimensional point cloud data to obtain at least one three-dimensional candidate target. Each three-dimensional candidate target includes corresponding spatial location information, geometric features and initial classification confidence. The target classification module is used to input the geometric features of each of the three-dimensional candidate targets into a pre-trained out-of-distribution target classifier to determine whether the three-dimensional candidate targets belong to a known category set, so as to identify unknown targets; The target projection module is used to project the three-dimensional candidate target corresponding to the unknown target onto the two-dimensional image data to obtain the corresponding two-dimensional image region; The semantic category module is used to input the two-dimensional image region into a pre-trained visual language large model, and use natural language prompts to guide the visual language large model to output the semantic category name of the unknown target; The result generation module is used to fuse the semantic category name and the spatial location information of the corresponding 3D candidate target to generate the final open-world 3D detection result.

Images