Methods, systems, media, and devices for 3D small object detection based on text prompts.
By introducing sparse query and contrastive learning techniques into multi-view images, and combining two-dimensional category-aware query keys with geometric cues to generate three-dimensional query keys, false positive query keys are suppressed, thus solving the problem of low accuracy in small object target detection and improving detection accuracy and robustness.
Patent Information
- Application Number
- CN202411982694.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2026-01-06
- Estimated Expiration
- 2044-12-31
AI Technical Summary
Existing multi-view 3D target detection methods suffer from low detection accuracy and excessive false positives when detecting small objects, and lack an effective mechanism to optimize the query.
By introducing sparse query and contrastive learning mechanisms, combining two-dimensional category-aware query keys with geometric cues, high-quality three-dimensional query keys are generated. Furthermore, by suppressing false positive query keys through contrastive learning, the detection accuracy for small objects is improved.
It significantly improves the detection accuracy and robustness of small objects, especially performing exceptionally well on the nuScenes dataset.
Smart Images
Figure CN119919928B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of computer vision and autonomous driving, and particularly to a three-dimensional small object target detection method, system, medium and equipment based on text prompts. BACKGROUND
[0002] Existing multi-view three-dimensional target detection methods perform well in detecting large objects, but there are significant challenges in small object target detection. Since small targets occupy fewer pixels in the image, traditional dense feature map-based methods are difficult to effectively capture the features of small object targets. Sparse query methods provide a more efficient target detection method, but when dealing with small targets, there are too many false positives. Existing technologies lack effective mechanisms to optimize queries and enhance the detection performance of small object targets. SUMMARY
[0003] To solve the above problems, the present application aims to provide a three-dimensional small object target detection method, system, medium and equipment based on text prompts, which improves the detection accuracy of three-dimensional small object targets by introducing sparse query and contrast learning mechanisms. Specifically, the present application aims to generate high-quality three-dimensional query keys by combining two-dimensional class-aware query keys and geometric cues, and to suppress false positive query keys through contrast learning, thereby significantly improving the perception of small object targets in multi-view three-dimensional detection tasks.
[0004] To achieve the above-mentioned purpose, in the first aspect, the technical solution adopted by the present application is as follows: a three-dimensional small object target detection method based on text prompts, comprising: acquiring multi-view images, processing the input multi-view images through a backbone network and a feature pyramid network to extract image features; generating two-dimensional class-aware query keys based on an open vocabulary target detector and text prompts, and generating two-dimensional bounding boxes for small object targets; generating three-dimensional query keys by combining two-dimensional bounding boxes with geometric cues, including three-dimensional positioning, three-dimensional size and direction based on a lightweight depth estimation network; using a contrast learning mechanism to optimize the generated three-dimensional query keys to suppress false positive query keys and improve the confidence of correct query keys; generating final three-dimensional bounding boxes based on the optimized query keys for target detection.
[0005] Further, the image features generate two-dimensional class-aware query keys based on an open vocabulary target detector and text prompts, and generate two-dimensional bounding boxes for small object targets, comprising:
[0006] The image features are input into a heat map head to generate two-dimensional global query keys; at the same time, the open vocabulary target detector generates two-dimensional class-aware query keys in combination with text prompt input;
[0007] The two-dimensional category-aware query key generates a two-dimensional bounding box based on the category information of the small object target. The center point of the bounding box is used to initialize the category-aware query key.
[0008] Furthermore, the two-dimensional bounding box is combined with geometric cues to generate a three-dimensional query key, including:
[0009] The two-dimensional global query key and the two-dimensional category-aware query key are merged to form a complete two-dimensional query set;
[0010] Geometric clues are used for processing, and three-dimensional query keys are generated by combining the geometric information of each query key;
[0011] During training, the 3D and 2D real labels are paired with the corresponding 3D and 2D query keys, respectively, for optimization of the contrastive learning module.
[0012] Furthermore, after generating the 3D query key, the process also includes a step of aggregating semantic information surrounding the 2D query key using a deformable attention mechanism:
[0013] The two-dimensional query key is projected back into the image features to generate an updated two-dimensional query key;
[0014] Semantic information is aggregated from the surrounding pixels of the two-dimensional query key location through a deformable attention mechanism to fully capture the rich semantic information in the image.
[0015] Furthermore, 3D localization based on lightweight depth estimation networks includes:
[0016] Each two-dimensional query key reference point in the image features is combined with the predicted depth distribution and projected into three-dimensional space to generate a three-dimensional proposed center point.
[0017] Candidate values for object orientation are uniformly sampled and generated within the interval [0, 2π) to determine the orientation information of the object.
[0018] Furthermore, a contrastive learning mechanism is employed to optimize the generated 3D query key, including:
[0019] The two-dimensional and three-dimensional real labels are paired with the generated query keys, and the matching degree of each query key is calculated using the Hungarian matching algorithm;
[0020] By using a contrastive learning loss function, correct query keys are reinforced while incorrect query keys are suppressed.
[0021] The loss function is:
[0022]
[0023] in, It is the comparison loss of the two-dimensional query key. It is the contrast loss of the three-dimensional query key, where α and β are weighting coefficients.
[0024] Furthermore, the size of the 3D bounding box is generated by sampling the length, width, and height of different categories of objects in the dataset and using a linearly increasing distribution.
[0025] Secondly, the technical solution adopted by this invention is as follows: a three-dimensional small object target detection system based on text prompts, comprising: an image feature extraction module, which acquires multi-view images, processes the input multi-view images through a backbone network and a feature pyramid network, and extracts image features; a two-dimensional bounding box generation module, which generates two-dimensional category-aware query keys based on the image features using an open-vocabulary target detector and text prompts, and generates two-dimensional bounding boxes for small object targets; a three-dimensional query key generation module, which combines the two-dimensional bounding boxes with geometric cues to generate three-dimensional query keys, wherein the geometric cues include three-dimensional localization, three-dimensional size, and orientation based on a lightweight depth estimation network; an optimization module, which optimizes the generated three-dimensional query keys using a contrastive learning mechanism to suppress false positive query keys and improve the confidence of correct query keys; and a target detection module, which generates the final three-dimensional bounding box based on the optimized query key for target detection.
[0026] Thirdly, the technical solution adopted by the present invention is: a computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the methods described above.
[0027] Fourthly, the technical solution adopted by the present invention is: a computing device comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.
[0028] The present invention has the following advantages due to the adoption of the above technical solutions:
[0029] 1. This invention addresses the shortcomings of existing query methods in small object detection. By combining a two-dimensional category-aware query key generated by an open-vocabulary object detector with geometric cues, it elevates the detection to three-dimensional space, significantly improving the accuracy of detecting small objects such as pedestrians and bicycles.
[0030] 2. This invention effectively suppresses false positive query keys by introducing a contrastive learning mechanism, thereby enhancing the robustness and detection performance of the system. The method is validated on the nuScenes dataset, demonstrating significant performance improvements, particularly excelling in small object detection tasks. Attached Figure Description
[0031] Figure 1 This is an overall flowchart of the three-dimensional small object target detection method based on text prompts in this embodiment of the invention;
[0032] Figure 2 This is a detailed flowchart of the three-dimensional small object target detection method based on text prompts in an embodiment of the present invention;
[0033] Figure 3 This is a diagram showing the effect of three-dimensional target detection in an embodiment of the present invention. Detailed Implementation
[0034] To address the lack of effective mechanisms for optimizing queries in existing technologies, which hinders the detection performance of small objects, this invention provides a text-based 3D small object detection method, system, medium, and device, specifically optimized for small object detection. This invention combines sparse query technology with contrastive learning, enhancing the detection capability of small objects by elevating the two-dimensional query key to three-dimensional space. During detection, an open-vocabulary object detector and text prompts are used to generate two-dimensional category-aware query keys, while geometric cues are combined to generate three-dimensional query keys. Through a contrastive learning mechanism, overlapping false positive query keys are suppressed, ultimately improving the detection accuracy and precision of small objects.
[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.
[0036] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0037] In one embodiment of the present invention, a method for detecting small 3D objects based on text prompts is provided. This method, based on multi-view image 3D object detection technology, is mainly applied to environmental perception tasks in autonomous vehicles. Specifically, the present invention is a 3D object detection method based on sparse query and contrastive learning, which enhances the system's ability to perceive small objects.
[0038] This invention first includes image feature extraction and 2D query key generation processes. The input image is processed by a backbone network to extract features, generating 2D global query keys and category-aware query keys. These queries are then uplifted to a 3D query key space using geometric cues and optimized using a contrastive learning module. Semantic information is aggregated through a deformable attention mechanism, thereby enhancing the contextual representation of the 2D query keys. Next, a lightweight depth estimation network generates the center points of 3D proposals, and the size and orientation information of 3D bounding boxes are generated based on sampling results from the dataset. Finally, the queries are optimized through a contrastive learning mechanism to ensure optimal detection accuracy and recall for small objects.
[0039] Specifically, such as Figures 1 to 3 As shown, the text-based 3D small object detection method includes the following steps:
[0040] 1) Acquire multi-view images, process the input multi-view images through backbone network and feature pyramid network, and extract image features;
[0041] 2) Image features are used to generate two-dimensional category-aware query keys based on an open-vocabulary object detector and text prompts, and two-dimensional bounding boxes are generated for small object targets;
[0042] 3) Combine the 2D bounding box with geometric cues to generate a 3D query key. The geometric cues include 3D localization based on a lightweight depth estimation network, and convert the 2D query key into a point in 3D space, with 3D size and orientation, through the camera's intrinsic and extrinsic parameter matrices.
[0043] 4) A contrastive learning mechanism is used to optimize the generated 3D query keys in order to suppress false positive query keys and improve the confidence of correct query keys;
[0044] 5) Generate the final 3D bounding box based on the optimized query key for object detection.
[0045] In step 1) above, multi-view images are acquired using a multi-view camera, and the image data is transmitted to the backbone network and the Feature Pyramid Network (FPN) to generate image features. Where B is the batch size, N is the number of viewpoints, C is the number of channels, and H and W are the height and width of the image, respectively.
[0046] In step 2) above, the image features generate a two-dimensional category-aware query key based on an open-vocabulary object detector and text prompts, and generate a two-dimensional bounding box for small object targets, including the following steps:
[0047] 2.1) Image features are input into the heatmap header to generate a two-dimensional global query key; at the same time, the open vocabulary target detector, combined with text prompt input, generates a two-dimensional category-aware query key;
[0048] The text prompts are based on small object target categories, specifically pedestrians, bicycles, and motorcycles.
[0049] 2.2) The two-dimensional category-aware query key generates a two-dimensional bounding box based on the category information of small object targets (such as pedestrians and bicycles). The center point of the bounding box is used to initialize the category-aware query key.
[0050] In step 3) above, the three-dimensional query key is generated by combining the two-dimensional bounding box with geometric cues, including the following steps:
[0051] 3.1) Set the two-dimensional global query key Q g and the 2D category-aware query key Q c The queries are merged to form a complete two-dimensional query set Q. 2D ;
[0052] Q 2D =Q g ∪Q c
[0053] 3.2) To elevate the two-dimensional query keys to three-dimensional space, geometric cues (such as 3D positioning, 3D size, and orientation) are used for processing. During this 3D query key generation process, the geometric information of each query key is combined to generate the 3D query key; a contrastive learning mechanism is employed to suppress false positive samples in the query keys and optimize the confidence of correct query keys.
[0054] 3.3) During training, the 3D and 2D real labels are paired with the corresponding 3D and 2D query keys, respectively, for optimization of the contrastive learning module.
[0055] Because the initialization of two-dimensional query keys is relatively sparse, it is difficult to fully capture the rich semantic information in the image. Therefore, after generating the three-dimensional query key in step 3) above, this embodiment also includes a step of aggregating the semantic information around the two-dimensional query key using a deformable attention mechanism: two-dimensional query key Q 2D The image features are projected back into the image feature F to generate the updated two-dimensional query key Q′. 2D The algorithm calculates the correlation between the query and surrounding pixels and extracts the contextual information of the query key. It also aggregates semantic information from the surrounding pixels of the two-dimensional query key location through a deformable attention mechanism to fully capture the rich semantic information in the image.
[0056] Specifically,
[0057]
[0058] In the formula, DeformAttn is the deformable attention calculation function; c 2D It is the reference point for the query key; A ijIt is the attention weight between the query key and surrounding pixels; Δc ij It is the predicted offset; W i W′ is the weight matrix of the two-dimensional query key. i This is the weight matrix for the updated two-dimensional query keys; N head N represents the number of attention heads in the multi-head attention mechanism used. key This represents the number of query keys.
[0059] In step 3) above, 3D localization based on a lightweight depth estimation network is performed by predicting a depth map from image features using a lightweight depth estimation network; specifically, this includes the following steps:
[0060] 3.2.1) Reference point c for each two-dimensional query key in image feature f 2D =(c w ,c h Combined with the predicted depth distribution d c The data is then projected into three-dimensional space to generate the center point of the proposed three-dimensional solution. This process relies on the camera's intrinsic and extrinsic parameter matrices, transforming the two-dimensional query key into a point in three-dimensional space through projection.
[0061] The center point of the 3D proposal is:
[0062]
[0063] Among them, K j and I j These are the extrinsic and intrinsic parameter matrices of the camera, respectively; c w c is the x-coordinate of the center point; h y is the ordinate of the center point; T is the matrix transpose.
[0064] 3.2.2) Candidate values for object orientation are uniformly sampled and generated in the interval [0, 2π) to determine the orientation information of the object.
[0065] Specifically, the handling of object orientation does not depend on the object category, but instead uses uniform sampling to generate candidate orientation values within the interval [0, 2π). The formula is as follows:
[0066]
[0067] Where, n θ N is the index of the candidate direction value; θ Θ represents the total number of direction candidates; Θ represents the candidate set of orientation angles; θ represents the orientation angle.
[0068] In step 4) above, the generated 3D query key is optimized using a contrastive learning mechanism, including the following steps:
[0069] 4.1) Pair the two-dimensional and three-dimensional real labels with the generated query keys, and calculate the matching degree of each query key using the Hungarian matching algorithm;
[0070] 4.2) Reinforce correct query keys and suppress incorrect query keys by using a contrastive learning loss function;
[0071] loss function for:
[0072]
[0073] in, It is the comparison loss of the two-dimensional query key. It is the contrast loss of the three-dimensional query key, where α and β are weighting coefficients.
[0074] The formulas for 2D and 3D contrast loss are as follows:
[0075]
[0076] in, and These are real labels in two dimensions and three dimensions, respectively. and φ(·) represents the two-dimensional and three-dimensional query keys, respectively; φ(·) represents the multilayer perceptron used to project the three-dimensional query key to a higher-dimensional space; τ represents the temperature coefficient; K represents the number of negative sample query keys; and N represents the number of samples.
[0077] In step 5) above, the size of the 3D bounding box for different categories of objects can be determined by analyzing the range of length, width, and height in the dataset. In this embodiment, the size of the 3D bounding box is generated by sampling the length, width, and height of different categories of objects in the dataset and using a linearly increasing distribution.
[0078] The specific sampling method is as follows:
[0079]
[0080]
[0081] Among them, L i H i and W i These represent the length, height, and width of the i-th bounding box, respectively, where N is the number of samples; L min L is the minimum length of the i-th bounding box. max H represents the maximum length of the i-th bounding box. min H is the minimum height of the i-th bounding box. max W represents the maximum height of the i-th bounding box.min W is the minimum width of the i-th bounding box. max This represents the maximum width of the i-th bounding box.
[0082] In one embodiment of the present invention, a three-dimensional small object detection system based on text prompts is provided, comprising:
[0083] The image feature extraction module acquires multi-view images and processes the input multi-view images through a backbone network and a feature pyramid network to extract image features.
[0084] The 2D bounding box generation module generates 2D category-aware query keys based on image features using an open-vocabulary object detector and text prompts, and generates 2D bounding boxes for small object targets.
[0085] The 3D query key generation module combines 2D bounding boxes with geometric cues to generate 3D query keys. The geometric cues include 3D localization, 3D size, and orientation based on a lightweight depth estimation network.
[0086] The optimization module employs a contrastive learning mechanism to optimize the generated 3D query keys, thereby suppressing false positive query keys and increasing the confidence of correct query keys.
[0087] The object detection module generates the final 3D bounding box based on the optimized query key for object detection.
[0088] In the above embodiments, image features are used to generate a two-dimensional category-aware query key based on an open-vocabulary object detector and text prompts, and a two-dimensional bounding box is generated for small object targets, including:
[0089] Image features are input into the heatmap header to generate a two-dimensional global query key; at the same time, the open vocabulary object detector, combined with text prompts, generates a two-dimensional category-aware query key.
[0090] The two-dimensional category-aware query key generates a two-dimensional bounding box based on the category information of the small object target. The center point of the bounding box is used to initialize the category-aware query key.
[0091] In the above embodiments, generating a 3D query key by combining a 2D bounding box with geometric cues includes:
[0092] The two-dimensional global query key and the two-dimensional category-aware query key are merged to form a complete two-dimensional query set;
[0093] Geometric clues are used for processing, and three-dimensional query keys are generated by combining the geometric information of each query key;
[0094] During training, the 3D and 2D real labels are paired with the corresponding 3D and 2D query keys, respectively, for optimization of the contrastive learning module.
[0095] In this embodiment, before generating the three-dimensional query key, the following is also included:
[0096] The two-dimensional query key is projected back into the image features to generate an updated two-dimensional query key;
[0097] Semantic information is aggregated from the surrounding pixels of the two-dimensional query key location through a deformable attention mechanism to fully capture the rich semantic information in the image.
[0098] In the above embodiments, the 3D localization based on a lightweight depth estimation network includes:
[0099] Each two-dimensional query key reference point in the image features is combined with the predicted depth distribution and projected into three-dimensional space to generate a three-dimensional proposed center point.
[0100] Candidate values for object orientation are uniformly sampled and generated within the interval [0, 2π) to determine the orientation information of the object.
[0101] In the above embodiments, a contrastive learning mechanism is used to optimize the generated three-dimensional query key, including:
[0102] The two-dimensional and three-dimensional real labels are paired with the generated query keys, and the matching degree of each query key is calculated using the Hungarian matching algorithm;
[0103] By using a contrastive learning loss function, correct query keys are reinforced while incorrect query keys are suppressed.
[0104] The loss function is:
[0105]
[0106] in, It is the comparison loss of the two-dimensional query key. It is the contrast loss of the three-dimensional query key, where α and β are weighting coefficients.
[0107] In the above embodiments, the size of the three-dimensional bounding box is generated by sampling the length, width, and height of different categories of objects in the dataset and using a linearly increasing distribution.
[0108] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.
[0109] In one embodiment of the present invention, a computing device is provided, which can be a terminal and may include: a processor, a communication interface, memory, a display screen, and an input device. The processor, communication interface, and memory communicate with each other via a communication bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs, which, when executed by the processor, implement the methods described in the above embodiments. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, a management network, NFC (Near Field Communication), or other technologies. The display screen can be a liquid crystal display or an e-ink display. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computing device, or an external keyboard, touchpad, or mouse. The processor can call logical instructions stored in the memory.
[0110] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0111] In one embodiment of the present invention, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments.
[0112] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores server instructions that cause a computer to perform the methods provided in the above embodiments.
[0113] The computer-readable storage medium provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.
[0114] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A three-dimensional small object target detection method based on a text prompt, characterized in that, Comprise: Obtaining multi-view images, processing the input multi-view images through a backbone network and a feature pyramid network to extract image features; The image features generate two-dimensional class-aware query keys based on an open-vocabulary object detector and a text prompt, and generate two-dimensional bounding boxes for small object targets, specifically: the image features are input into a heat map head to generate two-dimensional global query keys; at the same time, the open-vocabulary object detector generates two-dimensional class-aware query keys in combination with the text prompt input; the two-dimensional class-aware query keys generate two-dimensional bounding boxes according to the class information of the small object targets, and the center points of the bounding boxes are used to initialize the class-aware query keys; Generate three-dimensional query keys by combining the two-dimensional bounding boxes with geometric cues, including three-dimensional positioning, three-dimensional size and direction based on a lightweight depth estimation network, specifically: fuse the two-dimensional global query keys and the two-dimensional class-aware query keys to form a complete two-dimensional query set; process the geometric cues to generate three-dimensional query keys in combination with the geometric information of each query key; during training, the real labels of three-dimensional and two-dimensional are respectively paired with the corresponding three-dimensional and two-dimensional query keys for optimization of the contrast learning module; After generating the three-dimensional query keys, it further comprises the step of aggregating semantic information around the two-dimensional query keys using a deformable attention mechanism: the two-dimensional query keys are projected back into the image features to generate updated two-dimensional query keys; semantic information is aggregated from the surrounding pixels of the two-dimensional query key position through the deformable attention mechanism to fully capture the rich semantic information in the image; Optimize the generated three-dimensional query keys using a contrast learning mechanism to suppress false positive query keys and improve the confidence of correct query keys, specifically: pair the real labels of two-dimensional and three-dimensional with the generated query keys, and calculate the matching degree of each query key through the Hungarian matching algorithm; correct query keys are strengthened and incorrect query keys are suppressed through a contrast learning loss function; The loss function is: wherein, is the contrastive loss of the two-dimensional query key, is the contrastive loss of the three-dimensional query key, and are weight coefficients; Generate the final three-dimensional bounding box based on the optimized query keys for object detection.
2. The method of claim 1, wherein the text cue-based three-dimensional small object target detection method is characterized by, The three-dimensional positioning based on the lightweight depth estimation network comprises: Each two-dimensional query key reference point in the image features is combined with the predicted depth distribution and projected into three-dimensional space to generate three-dimensional proposal center points; Object direction candidate values are uniformly sampled in the interval [0, 2π) to determine the direction information of the object.
3. The method of claim 1, wherein the text cue-based three-dimensional small object target detection method is characterized by, The size of the three-dimensional bounding box is generated by sampling the length, width and height of objects of different categories in the data set and using a linearly increasing distribution.
4. A text cue based 3D small object target detection system for implementing the text cue based 3D small object target detection method according to any one of claims 1 to 3, characterized in that, Comprise: An image feature extraction module obtains multi-view images, processes the input multi-view images through a backbone network and a feature pyramid network to extract image features; A two-dimensional bounding box generation module generates two-dimensional class-aware query keys based on an open-vocabulary object detector and a text prompt, and generates two-dimensional bounding boxes for small object targets; A three-dimensional query key generation module generates three-dimensional query keys by combining the two-dimensional bounding boxes with geometric cues, including three-dimensional positioning, three-dimensional size and direction based on a lightweight depth estimation network; An optimization module employs a contrastive learning mechanism to optimize the generated three-dimensional query key to suppress false positive query keys and improve the confidence of correct query keys. A target detection module generates a final three-dimensional bounding box based on the optimized query key for target detection.
5. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that when executed by a computer cause the computer to perform a method comprising: The one or more programs include instructions that when executed by a computing device, cause the computing device to perform any of the methods of claims 1-3.
6. A computing device, comprising: Comprise: One or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods of claims 1-3.