Joint inference method and device for three-dimensional point cloud dense caption generation and visual positioning

Abstract

The application discloses a kind of three-dimensional point cloud dense caption generation and visual positioning combined inference method and device, it is related to three-dimensional visual task technical field.The method includes: introducing double clue description generator DCC as dense caption generation module into the 3DVG model based on DETR architecture to constitute combined model, DCC processes double visual clues Vc =( Q , V ), in describing a proposal, standard "sequence start mark" prefix is replaced in query in query Q To identify the object in focus, while introducing visual features V As the carrier of the interaction and relationship information of object and surrounding environment;When model inference, descriptive text includes 3DVG description text and / or 3DDC description text, and the target frame and / or all label frame and corresponding dense caption of description are obtained.The three-dimensional point cloud dense caption generation and visual positioning combined inference method and device provided by the application realize efficient end-to-end training by constructing single-stage combined inference framework.

Description

Technical Field

[0001] This invention relates to the field of 3D vision task technology, and in particular to a method and apparatus for joint reasoning of 3D point cloud dense subtitle generation and visual positioning. Background Technology

[0002] In recent years, with the development of artificial intelligence, 3D vision tasks based on deep neural networks have attracted increasing attention. Among them, 3D dense captioning (3DDC) and 3D visual grounding (3DVG) are becoming increasingly important. The 3DDC task mainly involves identifying and locating objects in a 3D point cloud scene and generating dense captions for these objects. These captions primarily describe the object's attribute characteristics and its positional relationship within the scene. The 3DVG task, on the other hand, involves finding and identifying relevant objects or regions in a 3D scene based on provided descriptive text, mainly using 3D bounding boxes for identification. These two tasks have significant application value in areas such as robot navigation, virtual reality, and language-based 3D scene understanding and interaction.

[0003] Since both 3DDC and 3DVG tasks require interaction with human language, they are widely considered complementary in 3D vision practice. Therefore, many methods combine the two approaches into a unified framework for inference. For example, the D3Net method proposes an attribute-perception module to enhance the performance of task-independent modules in the model; the Unit3D method proposes a fusion model based on the Transformer architecture, using the PointGroup 3D object detection framework for multimodal learning. However, these methods are all two-stage frameworks that rely on the output of 3D object detection methods to further generate object localization or object descriptions. Such two-stage frameworks heavily depend on the performance of the object detector, leading to suboptimal task performance.

[0004] In recent years, DETR-like architectures have received much attention, and the 3DDC method Vote2Cap-DETR and the 3DVG method EDA, both based on the DETR architecture, have achieved state-of-the-art performance in their respective fields. A natural idea is to combine these two tasks on a DETR-like architecture. However, directly combining these two types of tasks encounters the following two problems: (1) different input formats. 3DDC only requires a 3D scene as model input, while 3DVG requires not only a 3D scene but also related descriptive text; (2) different optimization objectives for query embedding. Query embedding, as a representation of the target object in the DETR framework, has different optimization objectives. 3DDC's optimization objective is to align all objects in the scene, while 3DVG's optimization objective is to align objects mentioned in the descriptive text. Therefore, there is currently no solution for combining these two types of tasks on a DETR-like or DETR-based architecture. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a joint reasoning method and apparatus for 3D point cloud dense subtitle generation and visual localization. By adding a lightweight dense subtitle generation module to the 3DVG model and using text prompts carefully designed for the 3DDC task, the 3DVG and 3DDC tasks are integrated within the DETR-like architecture. Efficient end-to-end training is achieved by constructing a single-stage joint reasoning framework.

[0006] In a first aspect, the present invention provides a joint reasoning method for generating dense subtitles from three-dimensional point clouds and for visual localization, comprising:

[0007] Joint model construction process: The dual-cue description generator DCC is introduced as a dense captioning generation module into the 3DVG model based on the DETR architecture to form a joint model. DCC processes dual visual cues. Vc =( Q , V When describing a proposal, the standard "sequence start marker" prefix is ​​used to describe the query in the query. Q Replacement to identify objects in focus while introducing visual features. V As a carrier of information about the interaction and relationship between objects and their surrounding environment; among which, query Q Generated by an object decoder based on a 3DVG model using the DETR architecture, visual features V A cross-attention encoder based on visual tokens from a 3DVG model with a DETR architecture. V 0 and text tokens T 0 generated;

[0008] Joint model inference process: Obtain input point cloud and corresponding descriptive text, wherein the descriptive text includes 3DVG descriptive text and / or 3DDC descriptive text, wherein the 3DDC descriptive text consists of a series of labels of the object to be identified;

[0009] When the descriptive text includes 3DVG descriptive text, the Box Head of the joint model outputs the object bounding box, and then selects the target bounding box described in the descriptive text based on the reference score;

[0010] When the descriptive text includes 3DDC descriptive text, the Box Head of the joint model outputs object boxes, then selects all the label boxes described in the 3DDC descriptive text, and the Caption Head generates corresponding dense captions for each label box.

[0011] Furthermore, the 3DVG model based on the DETR architecture is an improved EDA model, which replaces PointNet++, which extracts visual features from the input point cloud, with PointMetaBase, and uses the farthest point sampling algorithm to sample a fixed number of n points from the original point cloud P as candidate points. Then, given the token output by PointMetaBase Using a sphere with a radius of r The ball query is C Find each candidate point in In coordinate space Find the nearest neighbor tokens, and then use max pooling to... Features are assigned to them, ultimately resulting in visual tokens. :

[0012]

[0013] Among them, the input point cloud .

[0014] Furthermore, prior to the joint model inference process, the following is also included:

[0015] Joint model training process: For the 3DVG task, five loss functions are used for each layer of the object decoder, including: smoothing... L 1. Loss, size loss, GIoU loss, semantic alignment loss, and position alignment loss; the average loss across all decoder layers forms the total 3DVG loss. ;

[0016] For the 3DDC task, the standard cross-entropy loss is applied, and then fine-tuned by training with self-critical sequences, with the reward function being the CIDEr score.

[0017] The total loss function of the joint model, expressed end-to-end, is as follows:

[0018]

[0019] in, Due to subtitle loss, This represents the KPS loss used for query selection.

[0020] Furthermore, Roberta-base is used as a text feature extraction network to process descriptive text and generate text tokens. ,in, Indicates the number of visual tokens. Indicates the number of text tokens. Indicates the size of the point cloud. It is the feature dimension.

[0021] Furthermore, the 3DDC description text is generated by concatenating labels from the trained dataset or by accepting user input.

[0022] Secondly, the present invention provides a joint reasoning device for three-dimensional point cloud dense subtitle generation and visual positioning, comprising:

[0023] The joint model building module is used to incorporate the Dual Cue Description Generator (DCC) as a dense captioning generation module into a 3DVG model based on the DETR architecture to form a joint model. DCC processes dual visual cues. Vc =( Q , V When describing a proposal, the standard "sequence start marker" prefix is ​​used to describe the query in the query. Q Replacement to identify objects in focus while introducing visual features. V As a carrier of information about the interaction and relationship between objects and their surrounding environment; among which, query Q Generated by an object decoder based on a 3DVG model using the DETR architecture, visual features V A cross-attention encoder based on visual tokens from a 3DVG model with a DETR architecture. V 0 and text tokens T 0 generated;

[0024] The joint model inference module is used to acquire the input point cloud and the corresponding descriptive text, wherein the descriptive text includes 3DVG descriptive text and / or 3DDC descriptive text, wherein the 3DDC descriptive text consists of a series of labels of the object to be identified;

[0025] When the descriptive text includes 3DVG descriptive text, the Box Head of the joint model outputs the object bounding box, and then selects the target bounding box described in the descriptive text based on the reference score;

[0026] When the descriptive text includes 3DDC descriptive text, the Box Head of the joint model outputs object boxes, then selects all the label boxes described in the 3DDC descriptive text, and the Caption Head generates corresponding dense captions for each label box.

[0027] Furthermore, in the joint model construction module, the 3DVG model based on the DETR architecture is an improved EDA model. PointNet++, which extracts visual features from the input point cloud, is replaced with PointMetaBase. The farthest point sampling algorithm is used to sample a fixed number of n points from the original point cloud P as candidate points. Then, given the token output by PointMetaBase Using a sphere with a radius of r The ball query is C Find each candidate point in In coordinate space Find the nearest neighbor tokens, and then use max pooling to... Features are assigned to them, ultimately resulting in visual tokens. :

[0028]

[0029] Among them, the input point cloud .

[0030] Furthermore, prior to the joint model inference module, it also includes:

[0031] Joint model training module: For 3DVG tasks, it employs five loss functions for each layer of the object decoder, including: smoothing... L 1. Loss, size loss, GIoU loss, semantic alignment loss, and position alignment loss; the average loss across all decoder layers forms the total 3DVG loss. ;

[0032] For the 3DDC task, the standard cross-entropy loss is applied, and then fine-tuned by training with self-critical sequences, with the reward function being the CIDEr score.

[0033] The total loss function of the joint model, expressed end-to-end, is as follows:

[0034]

[0035] in, Due to subtitle loss, This represents the KPS loss used for query selection.

[0036] Furthermore, in the joint model construction module, Roberta-base is used as the text feature extraction network to process descriptive text and generate text tokens. ,in, Indicates the number of visual tokens. Indicates the number of text tokens. Indicates the size of the point cloud. It is the feature dimension.

[0037] Furthermore, in the joint model inference module, the 3DDC description text is generated by concatenating the labels of the training dataset or by accepting user input.

[0038] The technical solutions provided in the embodiments of the present invention have at least the following technical effects:

[0039] This invention innovatively leverages the characteristic of 3DVG models to select target objects based on descriptive text. By using specially designed descriptive text to stimulate the 3DVG model to perform the 3DDC task, it naturally unites the two tasks into a unified DETR framework. Furthermore, by integrating a lightweight dense caption generation module onto the 3DVG model, it can generate dense captions for all located objects. Through this design, the technical solution of this invention no longer depends on the performance of the object detector, enabling end-to-end model optimization and serving as a single-stage joint inference framework. In end-to-end training, the DC and VG tasks mutually reinforce each other, setting a new benchmark in the field. Attached Figure Description

[0040] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0041] Figure 1 This is a schematic diagram of the overall process of the method in Embodiment 1 of the present invention;

[0042] Figure 2 This is a flowchart illustrating the joint framework in Embodiment 1 of the present invention;

[0043] Figure 3 The simulation results of the 3DVG task in Embodiment 1 of this invention on the ScanRefer dataset;

[0044] Figure 4 The simulation results of the 3DDC task in Embodiment 1 of this invention on the ScanRefer dataset;

[0045] Figure 5 This is a schematic diagram of the device in Embodiment 2 of the present invention. Detailed Implementation

[0046] This invention provides a joint inference method and apparatus for 3D point cloud dense subtitle generation and visual localization. By adding a lightweight dense subtitle generation module to the 3DVG model and using text prompts carefully designed for the 3DDC task, the DETR-like architecture integrates the 3DVG and 3DDC tasks, and achieves efficient end-to-end training through a single-stage joint inference framework.

[0047] The overall concept of the technical solutions in the embodiments of the present invention is as follows:

[0048] This invention innovatively leverages the characteristic of 3DVG models to select target objects based on descriptive text. By using specially designed descriptive text to stimulate the 3DVG model to perform the 3DDC task, it naturally unites the two tasks into a unified DETR framework. Specifically, describing the descriptive text as a concatenation of object categories that may appear in a 3D scene can stimulate the 3DVG model to locate objects in the scene. Furthermore, by integrating a lightweight dense caption generation module onto the 3DVG model to construct a joint framework, the requirement to generate dense captions for all located objects can be met. Through this design, the method of this invention no longer depends on the performance of the object detector, enabling end-to-end model optimization and serving as a single-stage joint inference framework.

[0049] The following describes the joint framework constructed according to embodiments of the present invention, such as... Figure 1 As shown, this is an improved version of the state-of-the-art EDA model based on 3D visual localization (3DVG) models, mainly consisting of the following four parts:

[0050] (1) Visual text feature extraction network and text feature extraction network: Compared to using PointNet++ to extract visual features from the input point cloud, this embodiment uses PointMetaBase instead to achieve better visual feature extraction performance. A more powerful visual backbone network can better extract the attributes and relative positional relationships of objects in a 3D scene, which is very helpful for more granular interaction with language. Therefore, this embodiment replaces PointNet++ in EDA with PointMetaBase. Input point cloud A visual token will be generated after passing through a visual feature extraction network. Meanwhile, this embodiment uses Roberta-base as a text feature extraction network to process descriptive text and generate text tokens. .in, Indicates the number of visual tokens. Indicates the number of text tokens. Indicates the size of the point cloud. It is the feature dimension.

[0051] PointMetaBase's main contribution is abstracting the point cloud feature extraction network into a four-key module and exploring the optimal design of each module. However, due to the use of k-nearest neighbor queries in the neighborhood update module, the number of downsampled point clouds output by the backbone network is not fixed, and therefore cannot be directly used as a visual token. use.

[0052] To address this issue, this embodiment designs a parameterless visual token query module. First, the farthest point sampling (FPS) algorithm is used to sample a fixed number of n points from the original point cloud P as candidate points. Then, given the token output by PointMetaBase In this embodiment, a ball query with radius r is used. C Find each candidate point in V′ In coordinate space Find the nearest neighbor tokens, and then use max pooling to... The characteristics are assigned to them. Ultimately, visual tokens can be obtained. :

[0053]

[0054] (2) Cross-attention encoder: visual token and text tokens Integration is achieved through a complex bidirectional cross-encoding system. This system includes alternating self-attention layers and cross-attention layers, as well as feedforward neural network (FFN) layers. This process ultimately produces a dimensionality of [missing information]. Fusion visual features and text features Furthermore, these visual features are fed into a keypoint selection (KPS) module, which filters out... k The most relevant queries to the target object. These filtered queries It will then be used in object decoders and query scoring mechanisms.

[0055] (3) Object decoder: query the first k objects With text tags Together, they are fed into the object decoder, which consists of multiple stacked Transformer decoding layers, to generate the final query embedding. Based on these query embeddings The decoder consists of two main paths: one responsible for anchor box regression, and the other for aligning the anchor boxes with the text. The anchor box path continuously adjusts the anchor boxes during each Transformer decoding layer. It also provides embedded location information. The alignment path outputs a reference score. This is used to identify and determine the query that best matches the text, which is then used as the final output. Leveraging the localization potential of 3DVG, and with a well-designed dense caption generation module, all objects in the scene can be filtered out using citation scores, achieving scene target detection.

[0056] (4) Lightweight subtitle generation module

[0057] The key to generating unambiguous and detailed image descriptions lies in obtaining the relationship between the target object and its nearest neighbor environment. As mentioned in Vote2Cap-DETR, vote queries fail to provide sufficient attribute and spatial relationship information. To address this issue, they proposed a dense captioning generation module called Dual-CluedCaptioner (DCC), which introduces the k nearest local context token features of the vote query as its local environment and as keys for cross-attention. In the method of this embodiment, the KPS module generates the first k object queries... The same issues exist as those generated in the voting query in Vote2Cap-DETR. However, unlike Vote2Cap-DETR, which constructs voting queries based on spatial bias and content information to better generate descriptive captions, this embodiment's KPS-based method focuses on accurately locating key points of objects to simultaneously address 3DVG and 3DDC tasks. Therefore, this embodiment uses DCC as the dense captioning generation module and makes some modifications to accommodate this difference.

[0058] Specifically, this embodiment does not use voting queries. k Instead of performing cross-attention processing on the nearest neighbors, it introduces visual features that incorporate a broader scene context. As a carrier of information about the interaction and relationships between objects and their surroundings, it generates richer descriptions with more nuanced contextual details. DCC is a model based on a lightweight Transformer decoder, including two identical Transformer decoder blocks and sinusoidal position embeddings. PE (⋅) and a linear classification head.

[0059] In this embodiment, DCC processes dual visual cues in order to efficiently generate descriptions. First, when describing a proposal, the standard "Sequence Start Marker" (SOS) prefix is ​​used to describe the query within the query. QReplacement is performed to identify the object in focus. Then, for each query embedding, DDC generates its corresponding dense description. The Transformer's masking self-attention and cross-attention mechanisms decode the contextual relationship between the query and the visual scene, which can be represented as:

[0060]

[0061] Finally, the output of the cross-attention is processed by a feed-forward network (FFN) and a linear classification head, mapping the processed features to a descriptive vocabulary:

[0062]

[0063] This process ensures that the generated description is not only about the target object but also includes its relationship to its surrounding environment, which is crucial for generating specific and accurate descriptions. In this way, the model of this invention can better understand and express the details and context of objects in a 3D scene, thereby improving the quality and relevance of the description.

[0064] Compared with existing methods, the present invention has the following outstanding advantages:

[0065] First, it rethinks the role of textual prompts in 3DVG models: it provides a new perspective on the prompt-based localization capabilities of 3DVG models, transforming them into a dual-purpose tool that effectively facilitates the execution of 3DVG and 3DDC tasks.

[0066] Second, an advanced joint framework for 3DDC and 3DVG is proposed: by adding a lightweight dense caption generation module to the 3DVG model and using text prompts carefully designed for the 3DDC task, the DETR-like architecture integrates the 3DVG and 3DDC tasks, achieving efficient end-to-end training.

[0067] Third, it achieves state-of-the-art performance: The integration method of this invention not only produces outstanding results, but also sets a new benchmark in the field by mutually reinforcing the DC and VG tasks in end-to-end training, further establishing the framework's dominance in the field.

[0068] Example 1

[0069] This embodiment provides a joint reasoning method for 3D point cloud dense subtitle generation and visual positioning, such as... Figure 2 As shown, it includes:

[0070] Joint model construction process: The dual-cue description generator DCC is introduced as a dense captioning generation module into the 3DVG model based on the DETR architecture to form a joint model. DCC processes dual visual cues. Vc =( Q , V When describing a proposal, the standard "sequence start marker" prefix is ​​used to describe the query in the query. Q Replacement to identify objects in focus while introducing visual features. V As a carrier of information about the interaction and relationship between objects and their surrounding environment; among which, query Q Generated by an object decoder based on a 3DVG model using the DETR architecture, visual features V A cross-attention encoder based on visual tokens from a 3DVG model with a DETR architecture. V 0 and text tokens T 0 generated.

[0071] Joint Model Inference Process: The input point cloud and corresponding descriptive text are acquired, including 3DVG descriptive text and / or 3DDC descriptive text, where the 3DDC descriptive text consists of concatenated labels of the objects to be identified. Essentially, object detection can be viewed as a localization process for detecting cues; therefore, the design involves concatenating the labels of the objects to be identified, such as "cabinet, bed, chair, sofa." These concatenated labels are considered the localization descriptive text, with the goal of locating instances of all mentioned categories in the scene. This approach allows for visual localization and dense descriptive processing in a single stage. This integration enables end-to-end training within a single framework, fostering mutually reinforcing synergies between each task.

[0072] When the descriptive text includes 3DVG descriptive text (it can include both 3DVG and 3DDC descriptive text, or only 3DVG / 3DDC descriptive text), the Box Head output object box of the joint model... Then, the reference boxes are selected based on the reference scores. Specifically, in 3DVG, the target boxes described in the reference text are selected; in 3DDC, all the label boxes described in the description text are selected, and the Caption Head generates corresponding dense captions for each label box. This provides a new perspective on the cue-based localization capabilities of 3DVG models, transforming it into a dual-purpose tool that effectively facilitates the execution of 3DVG and 3DDC tasks.

[0073] In one possible implementation, the 3DVG model based on the DETR architecture is an improved EDA model, which replaces PointNet++, which extracts visual features from the input point cloud, with PointMetaBase, and uses a farthest point sampling algorithm to sample a fixed number of n points from the original point cloud P as candidate points. Then, given the token output by PointMetaBase Using a sphere with a radius of r The ball query is C Find each candidate point in In coordinate space Find the nearest neighbor tokens, and then use max pooling to... Features are assigned to them, ultimately resulting in visual tokens. :

[0074]

[0075] Among them, the input point cloud .

[0076] Before the joint model inference process, a joint model training process is also included: For the 3DVG task, this embodiment follows the EDA training scheme, which employs five loss functions for each layer of the object decoder: smoothing... loss (Used for the center coordinates of the prediction box), size loss (Using the size of the prediction box), GIoU loss Semantic alignment loss and position alignment loss . No. l The loss of the layer decoder is a weighted sum of these five loss terms:

[0077]

[0078] The average loss across all decoder layers forms the total 3DVG loss:

[0079]

[0080] For the 3DDC task, this embodiment follows the Vote2Cap-DETR approach, applying standard cross-entropy loss (maximum likelihood estimation training) and then fine-tuning with Self-Critical Sequence Training (SCST). This embodiment uses standard SCST in the model, with the CIDEr score as the reward function. In this embodiment, the caption loss is referred to as... Combined with the KPS loss used for query selection The final loss, expressed end-to-end, is as follows:

[0081]

[0082] In end-to-end training, the DC and VG tasks reinforce each other, setting a new benchmark for the field.

[0083] In one possible implementation, the 3DDC descriptive text is generated by concatenating labels from a training dataset. For example, a detector's object category library can be used to form cues by concatenating the labels of the objects to be identified, such as "cabinet, bed, chair, sofa." These concatenated labels are considered as localization descriptive text, with the goal of locating all instances of the mentioned categories in the scene. In another possible implementation, the 3DDC descriptive text can also accept user input.

[0084] The effects of the embodiments of the present invention are further presented through the following simulation experiments.

[0085] 1. Simulation conditions

[0086] This invention was developed on an NVIDIA V100 GPU environment, and the developed deep learning framework is based on PyTorch. The primary language used in this invention is Python.

[0087] 2. Simulation Content

[0088] Simulations were performed on the ScanRefer dataset. The ScanRefer dataset provides 51,583 text descriptions, with an average of 13.81 objects and 64.48 descriptions per scene. The 3DVG evaluation metric is Acc@6IoU, which measures the proportion of overlap between predicted and ground truth anchor boxes. Descriptions are categorized as "Unique" or "Multiple" if an object is the only representative of its category in the scene. The 3DDC evaluation metric is m@kIoU, where m can be any natural language generation metric, such as CIDEr (C), METEOR (M), BLEU-4 (B-4), and ROUGE-L (R). IoU thresholds are set to greater than 0.25 and 0.5, respectively. Figure 3 The simulation results of the 3DVG task on the ScanRefer dataset are as follows. Figure 4 The simulation results for the 3DDC task on the ScanRefer dataset are shown. It can be seen that the present invention achieves higher accuracy compared to other methods.

[0089] Based on the same inventive concept, this application also provides an apparatus corresponding to the method in Embodiment 1, as detailed in Embodiment 2.

[0090] Example 2

[0091] This embodiment provides a joint reasoning device for 3D point cloud dense subtitle generation and visual positioning, such as... Figure 5 As shown, it includes:

[0092] The joint model building module is used to incorporate the Dual Cue Description Generator (DCC) as a dense captioning generation module into a 3DVG model based on the DETR architecture to form a joint model. DCC processes dual visual cues. Vc =( Q , V When describing a proposal, the standard "sequence start marker" prefix is ​​used to describe the query in the query. Q Replacement to identify objects in focus while introducing visual features. V As a carrier of information about the interaction and relationship between objects and their surrounding environment; among which, query Q Generated by an object decoder based on a 3DVG model using the DETR architecture, visual features V A cross-attention encoder based on visual tokens from a 3DVG model with a DETR architecture. V 0 and text tokens T 0 generated;

[0093] The joint model inference module is used to acquire the input point cloud and the corresponding descriptive text, wherein the descriptive text includes 3DVG descriptive text and / or 3DDC descriptive text, wherein the 3DDC descriptive text consists of a series of labels of the object to be identified;

[0094] When the descriptive text includes 3DVG descriptive text, the Box Head of the joint model outputs object boxes, and then the target boxes described in the descriptive text are selected based on the referential score;

[0095] When the descriptive text includes 3DDC descriptive text, the Box Head of the joint model outputs object boxes, then selects all the label boxes described in the 3DDC descriptive text, and the Caption Head generates corresponding dense captions for each label box.

[0096] In one possible implementation, the joint model building module uses a DETR-based 3DVG model as an improved EDA model. PointNet++, which extracts visual features from the input point cloud, is replaced with PointMetaBase. A farthest-point sampling algorithm is used to sample a fixed number of n points from the original point cloud P as candidate points. Then, given the token output by PointMetaBase Using a sphere with a radius of r The ball query is C Find each candidate point in In coordinate space Find the nearest neighbor tokens, and then use max pooling to... Features are assigned to them, ultimately resulting in visual tokens. :

[0097]

[0098] Among them, the input point cloud .

[0099] In one possible implementation, prior to the joint model inference module, the following is also included:

[0100] Joint model training module: For 3DVG tasks, it employs five loss functions for each layer of the object decoder, including: smoothing... L 1. Loss, size loss, GIoU loss, semantic alignment loss, and position alignment loss; the average loss across all decoder layers forms the total 3DVG loss. ;

[0101] For the 3DDC task, the standard cross-entropy loss is applied, and then fine-tuned by training with self-critical sequences, with the reward function being the CIDEr score.

[0102] The total loss function of the joint model, expressed end-to-end, is as follows:

[0103]

[0104] in, Due to subtitle loss, This represents the KPS loss used for query selection.

[0105] In one possible implementation, the joint model building module uses Roberta-base as a text feature extraction network to process descriptive text and generate text tokens. ,in, Indicates the number of visual tokens. Indicates the number of text tokens. Indicates the size of the point cloud. It is the feature dimension.

[0106] In one possible implementation, in the joint model inference module, the 3DDC description text is generated by concatenating labels from the training dataset or by accepting user input.

[0107] Since the apparatus described in Embodiment 2 of the present invention is an apparatus used to implement the method of Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here. All apparatuses used in the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0108] This invention innovatively utilizes the characteristic of 3DVG models to select target objects based on descriptive text. By using specially designed descriptive text to stimulate the 3DVG model to perform the 3DDC task, it naturally unites the two tasks into a unified DETR framework. Furthermore, by integrating a lightweight dense caption generation module onto the 3DVG model, it can generate dense captions for all located objects. Through this design, the technical solution of this invention no longer depends on the performance of the object detector, enabling end-to-end model optimization and serving as a single-stage joint inference framework. In end-to-end training, the DC and VG tasks mutually reinforce each other, setting a new benchmark in the field.

[0109] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below.

Claims

1. A joint reasoning method for generating dense subtitles from 3D point clouds and for visual positioning, characterized in that, include: Joint model construction process: The dual-cue description generator DCC is introduced as a dense captioning generation module into the 3DVG model based on the DETR architecture to form a joint model. DCC processes dual visual cues. Vc =( Q , V When describing a proposal, the standard "sequence start marker" prefix is ​​used to describe the query within the query. Q Replacement to identify objects in focus while introducing visual features. V As a carrier of information about the interaction and relationship between objects and their surrounding environment; among which, query Q Generated by an object decoder based on a 3DVG model using the DETR architecture, visual features V A cross-attention encoder based on visual tokens from a 3DVG model with a DETR architecture. V 0 and text tokens T 0 generated; Joint model inference process: Obtain input point cloud and corresponding descriptive text, wherein the descriptive text includes 3DVG descriptive text and / or 3DDC descriptive text, wherein the 3DDC descriptive text consists of a series of labels of the object to be identified; When the descriptive text includes 3DVG descriptive text, the Box Head of the joint model outputs the object bounding box, and then selects the target bounding box described in the descriptive text based on the reference score; When the descriptive text includes 3DDC descriptive text, the Box Head of the joint model outputs the object box, then selects all the label boxes described in the 3DDC descriptive text, and the Caption Head generates corresponding dense captions for each label box; The 3DVG model based on the DETR architecture is an improved EDA model. It replaces PointNet++, which extracts visual features from the input point cloud, with PointMetaBase, and uses the farthest point sampling algorithm to sample a fixed number of n points from the original point cloud P as candidate points. Then, given the token output by PointMetaBase Using a sphere with a radius of r The ball query is C Find each candidate point in In coordinate space Find the nearest neighbor tokens, and then use max pooling to... Features are assigned to them, ultimately resulting in visual tokens. : Among them, the input point cloud , where n represents the number of visual tokens and N represents the size of the point cloud.

2. The method according to claim 1, characterized in that, Before the joint model inference process, it also includes: Joint model training process: For the 3DVG task, five loss functions are used for each layer of the object decoder, including: smoothing... L 1. Loss, size loss, GIoU loss, semantic alignment loss, and position alignment loss; the average loss across all decoder layers forms the total 3DVG loss. ; For the 3DDC task, the standard cross-entropy loss is applied, and then fine-tuned by training with self-critical sequences, with the reward function being the CIDEr score. The total loss function of the joint model, expressed end-to-end, is as follows: in, Due to subtitle loss, This represents the KPS loss used for query selection.

3. The method according to claim 1, characterized in that: Roberta-base was used as the text feature extraction network to process descriptive text and generate text tokens. ,in, Indicates the number of text tokens. It is the feature dimension.

4. The method according to claim 1, characterized in that: The 3DDC description text is generated by concatenating labels from the trained dataset or by accepting user input.

5. A joint reasoning device for generating dense subtitles from three-dimensional point clouds and for visual positioning, characterized in that, include: The joint model building module is used to incorporate the Dual Cue Description Generator (DCC) as a dense captioning generation module into a 3DVG model based on the DETR architecture to form a joint model. DCC processes dual visual cues. Vc =( Q , V When describing a proposal, the standard "sequence start marker" prefix is ​​used to describe the query within the query. Q Replacement to identify objects in focus while introducing visual features. V As a carrier of information about the interaction and relationship between objects and their surrounding environment; among which, query Q Generated by an object decoder based on a 3DVG model using the DETR architecture, visual features V A cross-attention encoder based on visual tokens from a 3DVG model with a DETR architecture. V 0 and text tokens T 0 generated; The joint model inference module is used to acquire the input point cloud and the corresponding descriptive text, wherein the descriptive text includes 3DVG descriptive text and / or 3DDC descriptive text, wherein the 3DDC descriptive text consists of a series of labels of the object to be identified; When the descriptive text includes 3DVG descriptive text, the Box Head of the joint model outputs the object bounding box, and then selects the target bounding box described in the descriptive text based on the reference score; When the descriptive text includes 3DDC descriptive text, the Box Head of the joint model outputs the object box, then selects all the label boxes described in the 3DDC descriptive text, and the Caption Head generates corresponding dense captions for each label box; In the joint model construction module, the 3DVG model based on the DETR architecture is an improved EDA model. PointNet++, which extracts visual features from the input point cloud, is replaced with PointMetaBase. The farthest point sampling algorithm is used to sample a fixed number of n points from the original point cloud P as candidate points. Then, given the token output by PointMetaBase Using a sphere with a radius of r The ball query is C Find each candidate point in In coordinate space Find the nearest neighbor tokens, and then use max pooling to... Features are assigned to them, ultimately resulting in visual tokens. : Among them, the input point cloud , where n represents the number of visual tokens and N represents the size of the point cloud.

6. The apparatus according to claim 5, characterized in that, Prior to the joint model inference module, it also includes: Joint model training module: For 3DVG tasks, it employs five loss functions for each layer of the object decoder, including: smoothing... L 1. Loss, size loss, GIoU loss, semantic alignment loss, and position alignment loss; the average loss across all decoder layers forms the total 3DVG loss. ; For the 3DDC task, the standard cross-entropy loss is applied, and then fine-tuned by training with self-critical sequences, with the reward function being the CIDEr score. The total loss function of the joint model, expressed end-to-end, is as follows: in, Due to subtitle loss, This represents the KPS loss used for query selection.

7. The apparatus according to claim 5, characterized in that: In the joint model construction module, Roberta-base is used as a text feature extraction network to process descriptive text and generate text tokens. ,in, Indicates the number of text tokens. It is the feature dimension.

8. The apparatus according to claim 5, characterized in that: In the joint model inference module, the 3DDC description text is generated by concatenating the labels of the training dataset or by accepting user input.

Images