Three-dimensional small object target detection method and system based on text prompt, medium and equipment

Abstract

The invention relates to the field of computer vision and automatic driving, and discloses a three-dimensional small object target detection method and system based on text prompt, a medium and equipment, and the method comprises the steps: obtaining a multi-view image, processing the input multi-view image through a backbone network and a feature pyramid network, and extracting image features; the image feature generates a two-dimensional category perception query key based on an open vocabulary target detector and a text prompt, and generates a two-dimensional bounding box for a small object target; combining the two-dimensional bounding box with a geometric clue to generate a three-dimensional query key, wherein the geometric clue comprises three-dimensional positioning, three-dimensional size and direction based on a lightweight depth estimation network; optimizing the generated three-dimensional query key by adopting a comparative learning mechanism to inhibit a false positive query key and improve the confidence of a correct query key; and generating a final three-dimensional bounding box based on the optimized query key for target detection. According to the invention, the detection precision of the three-dimensional small object target can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and autonomous driving, and in particular to a method, system, medium and device for detecting three-dimensional small objects based on text prompts. Background Art

[0002] Existing multi-view 3D object detection methods perform well in detecting large objects, but face significant challenges in detecting small objects. Since small objects occupy fewer pixels in an image, traditional methods based on dense feature maps have difficulty effectively capturing the features of small objects. Sparse query methods provide a more efficient way to detect objects, but are prone to excessive false positives when dealing with small objects. Existing technologies lack effective mechanisms to optimize queries and enhance the detection performance of small objects. Summary of the invention

[0003] In view of the above problems, the purpose of the present invention is to provide a method, system, medium and device for three-dimensional small object target detection based on text prompts, which improves the detection accuracy of three-dimensional small object targets by introducing sparse query and contrast learning mechanisms. Specifically, the present invention aims to generate high-quality three-dimensional query keys by combining two-dimensional category-aware query keys with geometric clues, and suppress false positive query keys through contrast learning, thereby significantly improving the perception ability of small object targets in multi-view three-dimensional detection tasks.

[0004] To achieve the above-mentioned purpose, in a first aspect, the technical solution adopted by the present invention is: a three-dimensional small object target detection method based on text prompts, which includes: obtaining multi-view images, processing the input multi-view images through a backbone network and a feature pyramid network, and extracting image features; the image features generate a two-dimensional category-aware query key based on an open vocabulary target detector and text prompts, and generate a two-dimensional bounding box for the small object target; the two-dimensional bounding box is combined with geometric clues to generate a three-dimensional query key, and the geometric clues include three-dimensional positioning, three-dimensional size and direction based on a lightweight depth estimation network; a contrastive learning mechanism is used to optimize the generated three-dimensional query key to suppress false positive query keys and improve the confidence of the correct query key; based on the optimized query key, a final three-dimensional bounding box is generated for target detection.

[0005] Furthermore, the image features generate 2D category-aware query keys based on an open vocabulary object detector and textual cues, and 2D bounding boxes for small object targets, including:

[0006] Image features are fed into the heatmap head to generate a 2D global query key; meanwhile, an open vocabulary object detector is combined with the textual prompt input to generate a 2D category-aware query key.

[0007] The 2D category-aware query key generates a 2D bounding box based on the category information of the small object target, and the center point of the bounding box is used to initialize the category-aware query key.

[0008] Furthermore, the 2D bounding box is combined with geometric clues to generate a 3D query key, including:

[0009] The two-dimensional global query key and the two-dimensional category-aware query key are merged to form a complete two-dimensional query set;

[0010] Using geometric clues for processing, the geometric information of each query key is combined to generate a three-dimensional query key;

[0011] During training, the 3D and 2D ground truth labels are paired with the corresponding 3D and 2D query keys, respectively, for optimization in the contrastive learning module.

[0012] Furthermore, after generating the three-dimensional query key, a step of aggregating semantic information around the two-dimensional query key by using a deformable attention mechanism is included:

[0013] The 2D query key is projected back into the image features to generate an updated 2D query key;

[0014] Semantic information is aggregated from the surrounding pixels of the 2D query key position via a deformable attention mechanism to fully capture the rich semantic information in the image.

[0015] Furthermore, 3D positioning based on lightweight depth estimation network includes:

[0016] Each 2D query key reference point in the image feature is combined with the predicted depth distribution and projected into 3D space to generate a 3D proposal center point;

[0017] The object direction candidate values ​​are uniformly sampled in the interval [0,2π) to determine the direction information of the object.

[0018] Furthermore, a contrastive learning mechanism is used to optimize the generated three-dimensional query key, including:

[0019] Pair the two-dimensional and three-dimensional true labels with the generated query keys, and calculate the matching degree of each query key through the Hungarian matching algorithm;

[0020] By contrasting the learning loss function, correct query keys are strengthened and incorrect query keys are suppressed;

[0021] The loss function is:

[0022]

[0023] in, is the contrast loss of the two-dimensional query key, is the contrast loss of the 3D query key, and α and β are weight coefficients.

[0024] Furthermore, the size of the 3D bounding box is generated by sampling the length, width, and height of objects of different categories in the dataset and using a linearly increasing distribution.

[0025] In the second aspect, the technical solution adopted by the present invention is: a three-dimensional small object target detection system based on text prompts, which includes: an image feature extraction module, which obtains multi-view images, processes the input multi-view images through a backbone network and a feature pyramid network, and extracts image features; a two-dimensional bounding box generation module, in which the image features generate two-dimensional category-aware query keys based on an open vocabulary target detector and text prompts, and generate a two-dimensional bounding box for small object targets; a three-dimensional query key generation module, which combines the two-dimensional bounding box with geometric clues to generate a three-dimensional query key, and the geometric clues include three-dimensional positioning, three-dimensional size and direction based on a lightweight depth estimation network; an optimization module, which uses a contrastive learning mechanism to optimize the generated three-dimensional query keys to suppress false positive query keys and improve the confidence of correct query keys; a target detection module, which generates a final three-dimensional bounding box based on the optimized query key for target detection.

[0026] In a third aspect, the technical solution adopted by the present invention is: a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device executes any one of the above methods.

[0027] In a fourth aspect, the technical solution adopted by the present invention is: a computing device, comprising: one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0028] The present invention adopts the above technical solution, which has the following advantages:

[0029] 1. The present invention solves the shortcomings of existing query methods in small object target detection. It combines the two-dimensional category-aware query keys generated by the open vocabulary target detector with geometric clues and elevates them to three-dimensional space for detection, significantly improving the detection accuracy of small object targets such as pedestrians and bicycles.

[0030] 2. By introducing a contrastive learning mechanism, the present invention effectively suppresses false positive query keys and enhances the robustness and detection performance of the system. The method is verified on the nuScenes dataset and shows good performance improvement, especially in the small object target detection task. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is an overall flow chart of a three-dimensional small object target detection method based on text prompts in an embodiment of the present invention;

[0032] Figure 2 is a detailed flow chart of a three-dimensional small object target detection method based on text prompts in an embodiment of the present invention;

[0033] Figure 3 It is a three-dimensional target detection effect diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to solve the problem that the prior art lacks an effective mechanism for optimizing queries and cannot enhance the detection performance of small object targets, the present invention provides a three-dimensional small object target detection method, system, medium and device based on text prompts, which is particularly optimized for the detection performance of small targets. The present invention combines sparse query technology with contrastive learning to enhance the detection capability of small object targets by promoting two-dimensional query keys to three-dimensional space. During the detection process, an open vocabulary target detector and text prompts are used to generate two-dimensional category-aware query keys, and three-dimensional query keys are generated in combination with geometric clues. Through the contrastive learning mechanism, overlapping false positive query keys are suppressed, and the detection precision and accuracy of small object targets are ultimately improved.

[0035] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0037] In one embodiment of the present invention, a 3D small object target detection method based on text prompts is provided, and a 3D target detection technology based on multi-view images is mainly applied to the environmental perception task of autonomous driving vehicles. Specifically, the present invention is a 3D target detection method based on sparse query and contrastive learning, which enhances the system's perception ability of small object targets.

[0038] The present invention first includes image feature extraction and two-dimensional query key generation processes. The input image is extracted with features through a backbone network to generate two-dimensional global query keys and category-aware query keys. These queries are promoted to a three-dimensional query key space through geometric clues and optimized in combination with a contrastive learning module. Semantic information is aggregated through a deformable attention mechanism to enhance the contextual representation of the two-dimensional query key. Next, the center point of the three-dimensional proposal is generated through a lightweight depth estimation network, and the size and orientation information of the three-dimensional bounding box is generated based on the sampling results of the data set. Finally, the query is optimized through a contrastive learning mechanism to ensure that the detection accuracy and recall rate of small object targets are optimal.

[0039] Specifically, Figures 1 to 3 As shown, the three-dimensional small object target detection method based on text prompts includes the following steps:

[0040] 1) Obtain multi-view images, process the input multi-view images through the backbone network and feature pyramid network, and extract image features;

[0041] 2) Image features generate 2D category-aware query keys based on an open vocabulary object detector and textual cues, and generate 2D bounding boxes for small object targets;

[0042] 3) Generate a 3D query key by combining the 2D bounding box with geometric clues, including 3D positioning based on a lightweight depth estimation network, and convert the 2D query key into a point in 3D space, 3D size and direction through the camera's intrinsic and extrinsic matrices;

[0043] 4) using a contrastive learning mechanism to optimize the generated three-dimensional query keys to suppress false positive query keys and improve the confidence of the correct query keys;

[0044] 5) Generate the final 3D bounding box based on the optimized query key for object detection.

[0045] In the above step 1), the input multi-view image is obtained through the multi-view camera, and the image data is transmitted to the backbone network and the feature pyramid network (FPN) to generate image features. Where B is the batch size, N is the number of views, C is the number of channels, and H and W are the height and width of the image, respectively.

[0046] In the above step 2), the image features generate a two-dimensional category-aware query key based on an open vocabulary object detector and textual cues, and generate a two-dimensional bounding box for a small object target, including the following steps:

[0047] 2.1) Image features are fed into the heatmap header to generate a 2D global query key; meanwhile, an open vocabulary object detector is combined with the textual prompt input to generate a 2D category-aware query key;

[0048] Among them, text prompts are based on small object target categories, including pedestrians, bicycles and motorcycles.

[0049] 2.2) 2D Category-Aware Query Key A 2D bounding box is generated based on the category information of small object targets (e.g., pedestrians, bicycles), and the center point of the bounding box is used to initialize the category-aware query key.

[0050] In the above step 3), the two-dimensional bounding box is combined with geometric clues to generate a three-dimensional query key, including the following steps:

[0051] 3.1) Set the two-dimensional global query key Q g and a two-dimensional category-aware query key Q c Fusion to form a complete two-dimensional query set Q 2D ;

[0052] Q 2D =Q g ∪Q c

[0053] 3.2) In order to promote the 2D query key to 3D space, geometric clues (such as 3D positioning, 3D size and direction) are used for processing. In this 3D query key generation process, the geometric information of each query key is combined to generate a 3D query key; a contrastive learning mechanism is used to suppress false positive samples in the query key and optimize the confidence of the correct query key.

[0054] 3.3) During training, the 3D and 2D true labels are paired with the corresponding 3D and 2D query keys, respectively, for optimization of the contrastive learning module.

[0055] Since the initialization of the two-dimensional query key is relatively sparse, it is difficult to fully capture the rich semantic information in the image. Therefore, after the three-dimensional query key is generated in the above step 3), this embodiment also includes a step of using a deformable attention mechanism to aggregate the semantic information around the two-dimensional query key: the two-dimensional query key Q 2D is projected back into the image feature F to generate the updated two-dimensional query key Q′ 2D , calculate the correlation between the query and the surrounding pixels, and extract the contextual information of the query key; aggregate semantic information from the surrounding pixels of the two-dimensional query key position through the deformable attention mechanism to fully capture the rich semantic information in the image.

[0056] Specifically,

[0057]

[0058] Where DeformAttn is the deformable attention calculation function; c 2D is the reference point for the query key; A ijis the attention weight of the query key and surrounding pixels; Δc ij is the predicted offset; W i is the weight matrix of the two-dimensional query key; W′ i is the weight matrix of the updated two-dimensional query key; N head N is the number of attention heads in the multi-head attention mechanism; key The number of query keys.

[0059] In the above step 3), based on the three-dimensional positioning of the lightweight depth estimation network, the lightweight depth estimation network is used to predict the depth map from the image features; specifically, the following steps are included:

[0060] 3.2.1) Each 2D query key reference point c in the image feature f 2D =(c w ,c h ) combined with the predicted depth distribution d c , and is projected into 3D space to generate the center point of the 3D proposal. This process relies on the camera's intrinsic and extrinsic matrices, and transforms the 2D query key into a point in 3D space through projection.

[0061] The three-dimensional proposed center point is:

[0062]

[0063] Among them, K j and I j are the external and internal parameter matrices of the camera respectively; c w is the horizontal coordinate of the center point; c h is the ordinate of the center point; T is the matrix transpose.

[0064] 3.2.2) The object direction candidate values ​​are uniformly sampled in the interval [0,2π) to determine the direction information of the object.

[0065] Specifically, the processing of object direction does not depend on the object category, but uses uniform sampling to generate direction candidate values ​​in the interval [0,2π). The formula is as follows:

[0066]

[0067] Among them, n θ is the index of the candidate direction value; N θ is the total number of direction candidates; Θ is the set of orientation angle candidates; θ is the orientation angle.

[0068] In the above step 4), the generated three-dimensional query key is optimized by using a contrastive learning mechanism, including the following steps:

[0069] 4.1) Pair the two-dimensional and three-dimensional true labels with the generated query keys, and calculate the matching degree of each query key through the Hungarian matching algorithm;

[0070] 4.2) Strengthen the correct query keys and suppress the wrong query keys by contrasting the learning loss function;

[0071] Loss Function for:

[0072]

[0073] in, is the contrast loss of the two-dimensional query key, is the contrast loss of the 3D query key, and α and β are weight coefficients.

[0074] The two-dimensional and three-dimensional contrast loss formulas are:

[0075]

[0076] in, and are the true labels for two-dimensional and three-dimensional respectively, and are the two-dimensional and three-dimensional query keys respectively, φ(·) is the multilayer perceptron used to project the three-dimensional query key to a higher dimensional space; τ is the temperature coefficient; K is the number of negative sample query keys; and N is the number of samples.

[0077] In the above step 5), for objects of different categories, the size of the three-dimensional bounding box can be determined by analyzing the range of length, width and height in the data set. In this embodiment, the size of the three-dimensional bounding box is generated by sampling the length, width and height of objects of different categories in the data set and adopting a linear increasing distribution.

[0078] The specific sampling method is as follows:

[0079]

[0080]

[0081] Among them, L i , H i and W i are the length, height and width of the i-th bounding box respectively, N is the number of samples; L min is the minimum length of the i-th bounding box, L max is the maximum length of the i-th bounding box; H min is the minimum height of the i-th bounding box, H max is the maximum height of the i-th bounding box; Wmin is the minimum width of the i-th bounding box, W max is the maximum width of the i-th bounding box.

[0082] In one embodiment of the present invention, a three-dimensional small object detection system based on text prompts is provided, which includes:

[0083] The image feature extraction module obtains multi-view images, processes the input multi-view images through the backbone network and feature pyramid network, and extracts image features;

[0084] 2D bounding box generation module, image features generate 2D category-aware query keys based on open vocabulary object detectors and textual cues, and generate 2D bounding boxes for small object targets;

[0085] A 3D query key generation module that generates a 3D query key by combining a 2D bounding box with geometric cues, including 3D location, 3D size, and orientation based on a lightweight depth estimation network.

[0086] The optimization module uses a contrastive learning mechanism to optimize the generated three-dimensional query keys to suppress false positive query keys and improve the confidence of correct query keys;

[0087] The object detection module generates the final 3D bounding box based on the optimized query key for object detection.

[0088] In the above embodiment, the image features generate a two-dimensional category-aware query key based on an open vocabulary object detector and textual cues, and generate a two-dimensional bounding box for a small object target, including:

[0089] Image features are fed into the heatmap head to generate a 2D global query key; meanwhile, an open vocabulary object detector is combined with the textual prompt input to generate a 2D category-aware query key.

[0090] The 2D category-aware query key generates a 2D bounding box based on the category information of the small object target, and the center point of the bounding box is used to initialize the category-aware query key.

[0091] In the above embodiment, the two-dimensional bounding box is combined with geometric clues to generate a three-dimensional query key, including:

[0092] The two-dimensional global query key and the two-dimensional category-aware query key are merged to form a complete two-dimensional query set;

[0093] Using geometric clues for processing, the geometric information of each query key is combined to generate a three-dimensional query key;

[0094] During training, the 3D and 2D ground truth labels are paired with the corresponding 3D and 2D query keys, respectively, for optimization in the contrastive learning module.

[0095] In this embodiment, before generating the three-dimensional query key, the process further includes:

[0096] The 2D query key is projected back into the image features to generate an updated 2D query key;

[0097] Semantic information is aggregated from the surrounding pixels of the 2D query key position via a deformable attention mechanism to fully capture the rich semantic information in the image.

[0098] In the above embodiment, the three-dimensional positioning based on the lightweight depth estimation network includes:

[0099] Each 2D query key reference point in the image feature is combined with the predicted depth distribution and projected into 3D space to generate a 3D proposal center point;

[0100] The object direction candidate values ​​are uniformly sampled in the interval [0,2π) to determine the direction information of the object.

[0101] In the above embodiment, the generated three-dimensional query key is optimized by using a contrastive learning mechanism, including:

[0102] Pair the two-dimensional and three-dimensional true labels with the generated query keys, and calculate the matching degree of each query key through the Hungarian matching algorithm;

[0103] By contrasting the learning loss function, correct query keys are strengthened and incorrect query keys are suppressed;

[0104] The loss function is:

[0105]

[0106] in, is the contrast loss of the two-dimensional query key, is the contrast loss of the 3D query key, and α and β are weight coefficients.

[0107] In the above embodiment, the size of the three-dimensional bounding box is generated by sampling the length, width and height of objects of different categories in the data set and adopting a linear increasing distribution.

[0108] The system provided in this embodiment is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for specific processes and detailed contents, which will not be repeated here.

[0109] A computing device provided in one embodiment of the present invention may be a terminal, which may include: a processor, a communications interface, a memory, a display screen, and an input device. The processor, the communications interface, and the memory communicate with each other through a communication bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory, wherein the non-volatile storage medium stores an operating system and a computer program, and the computer program is executed by the processor to implement the methods in the above embodiments; the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a management network, NFC (near field communication) or other technologies. The display screen may be a liquid crystal display screen or an electronic ink display screen, and the input device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computing device, or an external keyboard, touchpad or mouse, etc. The processor may call the logic instructions in the memory.

[0110] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0111] In one embodiment of the present invention, a computer program product is provided, wherein the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments.

[0112] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores server instructions, and the computer instructions enable a computer to execute the methods provided in the above embodiments.

[0113] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0114] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional small object detection method based on text prompts, characterized in that: include: Obtain multi-view images, process the input multi-view images through the backbone network and feature pyramid network, and extract image features; Image features generate 2D category-aware query keys based on an open vocabulary object detector and textual cues, and 2D bounding boxes for small object targets; Generate a 3D query key by combining a 2D bounding box with geometric cues including 3D location, 3D size, and orientation based on a lightweight depth estimation network. The generated three-dimensional query keys are optimized using a contrastive learning mechanism to suppress false positive query keys and improve the confidence of correct query keys; Generate the final 3D bounding box based on the refined query key for object detection.

2. The three-dimensional small object detection method based on text prompts as claimed in claim 1, characterized in that: Image features generate 2D category-aware query keys based on an open vocabulary object detector and textual cues, and 2D bounding boxes for small object targets, including: Image features are fed into the heatmap head to generate a 2D global query key; meanwhile, an open vocabulary object detector is combined with the textual prompt input to generate a 2D category-aware query key. The 2D category-aware query key generates a 2D bounding box based on the category information of the small object target, and the center point of the bounding box is used to initialize the category-aware query key.

3. The three-dimensional small object detection method based on text prompts as claimed in claim 1, characterized in that: Combine the 2D bounding box with geometric clues to generate a 3D query key, including: The two-dimensional global query key and the two-dimensional category-aware query key are merged to form a complete two-dimensional query set; Using geometric clues for processing, the geometric information of each query key is combined to generate a three-dimensional query key; During training, the 3D and 2D ground truth labels are paired with the corresponding 3D and 2D query keys, respectively, for optimization in the contrastive learning module.

4. The three-dimensional small object detection method based on text prompts as claimed in claim 3, characterized in that: After generating the three-dimensional query key, the step of adopting a deformable attention mechanism to aggregate the semantic information around the two-dimensional query key is also included: The 2D query key is projected back into the image features to generate an updated 2D query key; Semantic information is aggregated from the surrounding pixels of the 2D query key position via a deformable attention mechanism to fully capture the rich semantic information in the image.

5. The three-dimensional small object detection method based on text prompts as claimed in claim 1, characterized in that: 3D positioning based on lightweight depth estimation network, including: Each 2D query key reference point in the image features is combined with the predicted depth distribution and projected into 3D space. Generate 3D proposal center points; The object direction candidate values ​​are uniformly sampled in the interval [0,2π) to determine the direction information of the object.

6. The three-dimensional small object detection method based on text prompts as claimed in claim 1, characterized in that: The generated three-dimensional query key is optimized using a contrastive learning mechanism, including: Pair the two-dimensional and three-dimensional true labels with the generated query keys, and calculate the matching degree of each query key through the Hungarian matching algorithm; By contrasting the learning loss function, correct query keys are strengthened and incorrect query keys are suppressed; The loss function is: in, is the contrast loss of the two-dimensional query key, is the contrast loss of the 3D query key, and α and β are weight coefficients.

7. The three-dimensional small object detection method based on text prompts as claimed in claim 1, characterized in that: The size of the 3D bounding box is generated by sampling the length, width, and height of objects of different categories in the dataset and using a linearly increasing distribution.

8. A three-dimensional small object detection system based on text prompts, characterized in that: include: The image feature extraction module obtains multi-view images, processes the input multi-view images through the backbone network and feature pyramid network, and extracts image features; 2D bounding box generation module, image features generate 2D category-aware query keys based on open vocabulary object detectors and textual cues, and generate 2D bounding boxes for small object targets; A 3D query key generation module that generates a 3D query key by combining a 2D bounding box with geometric cues, including 3D location, 3D size, and orientation based on a lightweight depth estimation network. The optimization module uses a contrastive learning mechanism to optimize the generated three-dimensional query keys to suppress false positive query keys and improve the confidence of correct query keys; The object detection module generates the final 3D bounding box based on the optimized query key for object detection.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 7.

10. A computing device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any one of the methods described in claims 1 to 7.

Images