A transformer-based multi-modal indoor three-dimensional scene understanding method

Abstract

The application provides a multi-modal indoor three-dimensional scene understanding method based on a Transformer, which is based on a newly proposed multi-modal indoor three-dimensional scene understanding mechanism, can combine geometric structure information in a three-dimensional scene and semantic constraint information in natural language, effectively filters and enhances the features of a problem-related target, and realizes the joint output of semantic understanding and spatial positioning in an indoor scene through a unified network, so as to improve the accuracy and robustness of indoor three-dimensional scene understanding. The problems that a three-dimensional detection model in the prior art is difficult to understand natural language semantics, a two-dimensional question and answer model is difficult to utilize three-dimensional spatial structure information, and a target and text are insufficiently associated in the prior art serial scheme are solved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D scene understanding and multimodal intelligent analysis technology, and in particular to a Transformer-based multimodal indoor 3D scene understanding method and system, electronic device and computer-readable storage medium. Background Technology

[0002] With the development of RGB-D cameras, depth sensors, LiDAR, and 3D reconstruction technologies, 3D scene understanding for indoor environments is gradually becoming a key technology in applications such as intelligent robots, augmented reality, digital twins, indoor navigation, and human-computer interaction. Compared with traditional 2D image understanding, 3D scene understanding can directly utilize the position, size, orientation, and relative relationships between targets in space, thus possessing stronger expressive capabilities in complex indoor environments.

[0003] In existing technologies, some methods primarily perform 3D object detection in indoor point cloud scenes. They can identify furniture, equipment, and other objects in the scene and output corresponding 3D bounding boxes. However, these methods typically focus on the detection task itself and lack the ability to understand natural language semantics. They struggle to handle complex semantic inputs that include descriptions of object categories, attributes, orientations, and relative spatial relationships, thus failing to meet the scene understanding requirements of indoor question answering, object querying, and semantic retrieval.

[0004] Another type of method uses visual question answering technology to process the semantic correspondence between images and text, and can output answers based on the user's input questions. However, most of these methods are based on two-dimensional images for reasoning. Since two-dimensional images lack a complete three-dimensional geometric structure, they often struggle to accurately identify targets that truly correspond to the semantics of the question in indoor environments with severe occlusion, dense targets, large changes in perspective, and the need to reason about spatial relationships such as front, back, left, right, up, and down, thus affecting the accuracy of the answers.

[0005] Some solutions attempt to concatenate 3D detection results with text-based question-answering models. However, these solutions typically only use the detection results as input to the subsequent question-answering module, lacking deep modeling of the relationship between candidate 3D targets and text semantics. This makes it difficult for the model to accurately focus on the target entity most relevant to the current question from multiple candidate targets. Furthermore, many existing solutions only output text answers without providing the corresponding spatial localization results, thus lacking spatial verifiability and exhibiting weak model interpretability. Summary of the Invention

[0006] To address the technical problems existing in the prior art, the present invention provides the following technical solution: On the one hand, a multimodal indoor 3D scene understanding method based on Transformer is provided, including the following steps: Acquire multimodal input data of an indoor scene, wherein the multimodal input data includes at least three-dimensional scene data and natural language text; The 3D scene data is input into a pre-trained TR3D 3D object detection model to generate multiple candidate 3D object boxes, and the geometric feature representations corresponding to each candidate 3D object box are extracted; the natural language text is encoded to obtain text context semantic features; Based on the semantic features of the text context, the geometric feature representations corresponding to the candidate 3D target boxes are modeled for relevance, resulting in problem-related target enhancement features. The problem-related target enhancement features and the text context semantic features are input into the Transformer cross-modal fusion network for interactive fusion to obtain a scene-language joint feature representation; Based on the scene-language joint feature representation, answer prediction and target localization are performed simultaneously, and scene understanding results are output. The scene understanding results include at least the question answer and one or more three-dimensional target boxes associated with the question answer.

[0007] Preferably, the three-dimensional scene data includes point cloud data, and each point in the point cloud also includes at least one of the following additional features: color features, normal vector features, relative height features, and multi-view projection features; The natural language text includes at least one of question text, query text, description text, or instruction text.

[0008] Preferably, the step of inputting 3D scene data into a pre-trained TR3D 3D object detection model to generate multiple candidate 3D object boxes, and extracting the geometric feature representations corresponding to each candidate 3D object box, includes: The point cloud data is normalized, denoised, voxelized, and sampled. Sparse convolutional networks are used to extract multi-level features from the voxelized point cloud to obtain voxel feature maps at different scales. The detection head outputs the center coordinates, size parameters, orientation parameters, category score, and confidence score of the candidate 3D target bounding boxes. For each candidate box, the RoI average pooling is performed on the coverage area of ​​the voxel feature map to obtain the initial geometric features. These features are then concatenated with the geometric parameters of the box through a layer of MLP encoding to obtain the final geometric feature representation of the candidate target.

[0009] Preferably, the pre-trained TR3D 3D target detection model is trained using indoor scene annotation data, which includes at least target category annotations and 3D directional bounding box annotations.

[0010] Preferably, the encoding of natural language text to obtain text context semantic features includes: The natural language text is segmented to obtain a token sequence; Map the word segmentation results to a sequence of word vectors; The word vector sequence is modeled in context using a bidirectional recurrent neural network, a pre-trained language model, or a Transformer text encoder to obtain a text context semantic feature sequence.

[0011] Preferably, the encoding of natural language text to obtain text context semantic features further includes a semantic saliency enhancement step: Pre-maintain word lists for four categories: directional words, color words, category words, and relational words; For each token segmentation, output the corresponding category one-hot label. A token can belong to multiple categories at the same time. The saliency of different types of token features is enhanced through a gating weighting mechanism: , in: For learnable class saliency weights, One-hot tags for the token category, Features of the original token For the enhanced token features, T represents transpose.

[0012] Preferably, the relevance modeling of the geometric feature representations corresponding to the candidate 3D target boxes based on text context semantic features to obtain problem-related target enhancement features includes: The semantic features of the text context and the geometric features of the candidate target are mapped to a unified feature space through two learnable projection matrices. Attention pooling is applied to text-side features to obtain sentence-level semantic representations; Calculate the matching score between each candidate 3D bounding box and the natural language text; Sort the matching scores in descending order and retain the top K candidate targets; Perform weighted enhancements on the retained candidate targets: , Where: β is the reinforcing strength To match scores, For original geometric features, For enhanced target features.

[0013] Preferably, the matching score is obtained by weighted fusion of four parts: , in: The dot product similarity between the sentence-level semantics of the text and the geometric features of the target in the common space; A score is awarded for the consistency between the candidate box category information and the target semantic description in the text. A score is given for the consistency between the spatial location of the candidate box and the location description in the text; The target confidence level output by the TR3D detection head; , , , These are the corresponding weighting coefficients.

[0014] Preferably, the Transformer cross-modal fusion network includes a text encoding sublayer, a target encoding sublayer, and a bidirectional cross-attention fusion sublayer, wherein: The text encoding sublayer uses the standard Transformer self-attention mechanism to model the semantic dependencies between words within the text. The target encoding sublayer is used to model the spatial and contextual relationships between candidate 3D targets; The bidirectional cross-attention fusion sublayer performs cross-attention computation from text to target and from target to text, respectively, to establish a bidirectional correspondence between natural language text semantics and candidate 3D targets.

[0015] Preferably, the spatial relationships between the candidate 3D targets include at least one of the following: relative distance relationship, left-right relationship, up-down relationship, front-back relationship, adjacency relationship, and containment relationship; and the spatial relationships can be encoded by constructing a target relationship graph and input together with the geometric feature representations corresponding to the candidate 3D target boxes into the Transformer cross-modal fusion network.

[0016] Preferably, the step of interactively fusing the question-related target enhancement features and the text context semantic features into the Transformer cross-modal fusion network includes: Using text context semantic features as keys and values, and question-related target enhancement features as queries, the first-way cross-attention computation is performed to obtain the target features after text semantic injection; Using question-related target enhancement features as keys and values, and text context semantic features as queries, a second-path cross-attention computation is performed to obtain text features after target space information injection. The cross-attention calculation result is added to the original feature and then processed by layer normalization and feedforward network to obtain the output of the layer; By stacking multiple layers of the above processing steps, the final output is a joint feature representation of the scene and language.

[0017] Preferably, the simultaneous execution of answer prediction and target localization based on scene-language joint feature representation includes: outputting the target answer category or free text answer from a preset answer set through the answer prediction branch; The target localization branch outputs the localization results or association scores of one or more candidate 3D target boxes related to the natural language text, where: The scene understanding results also include target category prediction results related to natural language text, which are used to help distinguish candidate targets with similar spatial relationships; One or more candidate 3D target boxes associated with the natural language text are used to spatially validate the answer to the question, thereby reducing incorrect answers caused by relying solely on textual statistical priors.

[0018] Preferably, a joint loss function is used to optimize the model during the training phase. This joint loss function includes at least answer prediction loss, target localization loss, target classification loss, and TR3D detection loss, and can be expressed as:

[0019] in, Predict the loss for the answer. Locate the loss to the target. Classification loss for the target For TR3D detection of loss, , , and These are the corresponding weighting coefficients.

[0020] On the other hand, an application of the above-described method is provided, which is applied to at least one of the following tasks: indoor 3D visual question answering task, indoor target referencing understanding task, indoor target spatial relationship reasoning task, or indoor scene semantic retrieval task.

[0021] On the other hand, a system for implementing the above-described method is provided, the system comprising: Multimodal data acquisition module: used to acquire multimodal input data of indoor scenes, wherein the multimodal input data includes at least three-dimensional scene data and natural language text; 3D data processing module: used to input the 3D scene data into the pre-trained TR3D 3D target detection model, generate multiple candidate 3D target boxes, and extract the geometric feature representations corresponding to each candidate 3D target box; NLP processing module: used to encode the natural language text to obtain text context semantic features; Target feature generation module: used to perform correlation modeling on the geometric feature representation corresponding to the candidate 3D target box based on the semantic features of the text context, to obtain problem-related target enhancement features; Feature fusion module: used to input the problem-related target enhancement features and the text context semantic features into the Transformer cross-modal fusion network for interactive fusion to obtain scene-language joint feature representation; Output module: Based on the scene-language joint feature representation, it performs answer prediction and target localization simultaneously, and outputs scene understanding results, which include at least the question answer and one or more 3D target boxes associated with the question answer.

[0022] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described above.

[0023] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above method.

[0024] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention utilizes a pre-trained TR3D 3D target detection model to generate candidate 3D target boxes from indoor 3D scenes and extracts the corresponding geometric feature representations, enabling subsequent scene understanding to be based on real 3D spatial targets, thereby making more effective use of the geometric structure information in the scene.

[0025] This invention guides the relevance modeling of candidate 3D targets through textual semantics, enabling the priority screening and enhancement of target entities most relevant to the question semantics among multiple candidate targets, reducing interference from irrelevant targets, and improving the accuracy of target selection.

[0026] This invention employs the Transformer cross-modal fusion mechanism to jointly model text semantics and 3D target geometric information, enabling simultaneous processing of multiple information such as target category, attribute description, and spatial relationships, thereby improving the understanding ability in complex indoor scenes.

[0027] This invention not only outputs the answer to the question, but also outputs one or more three-dimensional target boxes associated with the answer, making the scene understanding results spatially interpretable and verifiable, thereby reducing the risk of misjudgment caused by relying solely on language priors.

[0028] This invention simultaneously constrains the processes of 3D target detection, target association, answer prediction, and target localization through a multi-task joint optimization approach, which helps to improve the overall performance and robustness of the model. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the overall framework of a Transformer-based multimodal indoor 3D scene understanding method. Figure 2 A schematic diagram illustrating the generation of candidate 3D bounding boxes and extraction of geometric features of candidate targets for the TR3D 3D target detection model; Figure 3 This is a schematic diagram of the Transformer cross-modal fusion network structure; Figure 4 This is a schematic diagram of a two-branch structure that simultaneously performs answer prediction and target localization based on scene-language joint features; Figure 5 This is a schematic diagram of the training process; Figure 6 This is a schematic diagram illustrating an example of end-to-end inference. Figure 7 A schematic diagram showing the ROC curve and AUC annotation for the target branch. Detailed Implementation

[0031] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0032] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0033] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0034] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0035] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0036] like Figure 1 As shown, this invention generates candidate 3D target bounding boxes and their geometric feature representations by inputting indoor 3D scene data into a pre-trained TR3D 3D target detection model; then, it encodes natural language text to obtain text context semantic features; further, it uses text semantics to guide the relevance modeling of candidate targets, and combines the Transformer cross-modal fusion mechanism to generate scene-language joint feature representations; finally, it simultaneously outputs the question answer and one or more 3D target bounding boxes associated with the question answer, thereby improving the accuracy, interpretability, and spatial consistency of indoor 3D scene understanding.

[0037] The following will combine Figure 2-7 The implementation principles and effects of the present invention described above are explained in detail in the examples.

[0038] Example 1 This embodiment provides a multimodal indoor 3D scene understanding method based on Transformer.

[0039] Step S101: Acquire multimodal input data of the indoor scene. The multimodal input data includes at least three-dimensional scene data and natural language text.

[0040] (1) Acquisition and generation of 3D scene data. 3D scene data can be acquired by RGB-D cameras, depth cameras, LiDAR, or 3D reconstruction equipment based on multi-view images, preferably indoor scene point cloud data. The specific acquisition and preprocessing process is as follows: The first step is depth data acquisition. Taking an RGB-D camera as an example, the acquisition device takes pictures from multiple angles indoors to obtain aligned color images. With depth map , Indicates the viewpoint number. Indicates the height of the image. This represents the width of the image. Depth maps are obtained directly from the camera's built-in structured light, Time-of-Flight (ToF), or active stereo solutions. For pure RGB acquisition devices without a depth channel, stereo matching or a self-supervised monocular depth estimation network is used to recover the depth map from multi-view color images. For lidar acquisition, sparse point clouds are obtained directly. Skip the depth map backprojection step.

[0041] The second step is the back projection of the depth map onto the point cloud. For each pixel... The three-dimensional points in the camera coordinate system are obtained by back-projecting the pinhole camera model:

[0042] in For the camera intrinsic parameter matrix, For pixels The depth value at that location.

[0043] The third step is multi-view point cloud registration and fusion. This is achieved using camera extrinsic parameters. Point clouds from various perspectives are unified to the same world coordinate system, and then registered using ICP. Let be a rotation matrix. The vector is a translation vector; voxel downsampling is used to remove duplicates in overlapping regions to obtain the complete scene point cloud. .

[0044] The fourth step is to construct features point by point. For each point in the point cloud... In three-dimensional coordinates In addition, at least one of the following features are added: Color characteristics : Obtained by backprojecting the points back onto the color image sampled from the nearest viewpoint; Normal vector features :based on of Minimum eigenvector estimation of the nearest-neighbor local covariance matrix; Relative height features Estimating ground height based on scene (Obtained by fitting the maximum horizontal plane using RANSAC) as the benchmark; Multi-view projection features : point Projected to 2D feature maps from each viewpoint (extracted by a 2D CNN) are upsampled and then concatenated or pooled. mv The dimension of the multi-view projection feature vector;

[0045] Each point is ultimately represented as .

[0046] (2) Natural language text. Natural language text can be user-input question text, query text, description text, or instruction text, such as “What is on the floor between the TV and the wooden chair” or “Where is the blue suitcase located?”

[0047] Step S102 3D Target Detection and Geometric Feature Extraction Based on TR3D like Figure 2 As shown, after normalizing, denoising, voxelizing and sampling the point cloud data, it is input into the pre-trained TR3D 3D target detection model to generate candidate 3D target boxes and extract the geometric feature representation of the candidate targets.

[0048] (1) TR3D model structure. TR3D uses a sparse convolution (SparseConv) backbone network to perform multi-level feature extraction on the voxelized point cloud, obtaining voxel feature maps of different scales. ;F （l） This represents the voxel feature map of layer l. The detector head predicts at each candidate location: After non-maximum suppression (NMS), the output Candidate 3D target bounding boxes Geometric features are obtained by pooling the ROIs from the corresponding voxel feature map regions. B m Let g represent the m-th candidate 3D bounding box. m Let d represent the geometric feature vector of the m-th candidate target. g Indicates the dimensional size of the geometric feature.

[0049] (2) TR3D model training process. To ensure that this embodiment can be fully implemented by those skilled in the art, the complete training process is described below using the ScanNet-V2 dataset as an example; the method is the same when applied to other datasets (such as SUN RGB-D, S3DIS), only the class set and the division ratio are different.

[0050] The first step is training set preparation. A publicly available indoor scene dataset is used, with labeled data including point clouds, target category labels, and 3D oriented bounding boxes for each scene; taking ScanNet-V2 as an example, there are approximately 1513 scenes and 18 target categories.

[0051] The second step is data preprocessing. For each scene's point cloud, the following steps are performed sequentially: removing outliers outside the ground plane (statistical outlier removal, neighborhood...). Standard deviation multiple Coordinate centering (subtracting the scene centroid); Scale normalization (normalizing to the maximum axis range). Voxelization (voxel side length) ).

[0052] The third step is feature engineering. Within each voxel, the color, normal vector, relative height, and multi-view projection features of points falling into that voxel are averaged and aggregated to obtain the initial features of the voxel. , This indicates the dimension of the input feature; examples range from 6 to 64.

[0053] The fourth step is to divide the training data. Approximately 1201 / 312 scenarios are used as the training / validation set according to the official guidelines, while the test set uses the official online leaderboard. Alternatively, internal cross-validation can be performed on the training set at 80% / 20%.

[0054] Step 5, data augmentation. In each training iteration, perform the following on the point cloud with probability: [Further details about the data augmentation process are needed for accurate translation.] random rotation of axis Random translation Random scaling Random point sampling perturbation (Gaussian noise added to each point location) Randomly dropping points (probability of dropping points) ); Color perturbation (HSV space random jitter). Symbol This indicates that a variable is from a range A value randomly selected with equal probability.

[0055] Step 6, Training Process. The optimizer example uses AdamW with an initial learning rate. Weight decay The learning rate uses cosine annealing; the batch size is typically 8; the number of training epochs is typically 50; the loss function is the sum of TR3D's native classification Focal Loss and center / size / orientation IoU regression loss, denoted as... .

[0056] Step 7, Validation and Deployment. Each epoch evaluates mAP@0.25 and mAP@0.5 on the validation set; the optimal checkpoint is retained for downstream inference of this method; during deployment, the model parameters are frozen and embedded only as a forward candidate box generation module into the complete system.

[0057] (3) Geometric feature representation. For each candidate box... The RoI average pooling is performed on the coverage area of ​​its voxel feature map to obtain ( (This indicates that the dimension example here is 256), and is related to the geometric parameters of the box. After being encoded by an MLP layer and then concatenated, the final candidate target geometric features are obtained. ( Example: 512).

[0058] Step S103 Natural Language Text Encoding natural language text Execution: Token segmentation to obtain a token sequence The word vector sequence is obtained by looking up the embedding table. ;Will Input text encoding network to obtain contextual semantic feature sequence Text encoding networks can employ bidirectional long short-term memory (BiLSTM) networks, N q This represents the total number of tokens obtained after word segmentation of the natural language text Q.

[0059] Specific implementation methods for enhancing saliency.

[0060] The first step is to construct a semantic category dictionary. This involves pre-maintaining four categories of vocabulary. (Directional words: left, right, front, back, up, down, between, beside, ...) (Color words: red, blue, ...) (Category words: bed, chair, suitcase, ...) (Relational words: nearby, between, above, ...).

[0061] The second step is to process each token. Output category one-hot label (A token can belong to multiple categories at the same time).

[0062] The third step is to enhance saliency through gating weighting:

[0063] in The learnable class saliency weights are initialized to positive values ​​(e.g., ...). The training phase and downstream tasks jointly optimize the model, where T represents transpose. The technical principle of this mechanism is to assign greater feature magnitudes to the four categories of words that are most discriminative for target localization and answer prediction: location, color, category, and relationship. This forces the subsequent attention module to allocate more weights to these tokens during softmax normalization, thereby improving the model's ability to capture the semantic description of the target.

[0064] The final output is contextual semantic features. For the sake of brevity, the following text will still be referred to as... , The dimension of the feature.

[0065] Step S104: Relevance Modeling of Candidate 3D Targets Based on Textual Semantics This step will incorporate text context semantic features. Geometric features of candidate targets Map to a unified feature space and calculate the matching score between the two.

[0066] (1) Unified feature space mapping. This is achieved through two learnable projection matrices. , Project the features on both sides onto Public Space:

[0067] Attention pooling is then applied to the text to obtain sentence-level representations. These are the semantic features after projection. These are the geometric features after projection.

[0068]

[0069] in For learnable attention query vectors, For token Attention weights.

[0070] (2) Matching score calculation. Candidate targets The matching score between the text and the target text is a weighted fusion of four parts:

[0071] The components and their operating mechanisms are as follows: Attention Similarity The cosine / dot product similarity between the sentence-level semantics of the text and the geometric features of the target in the common space is used as a soft alignment component. Category Consistency ,in The semantic vector is the result of weighted averaging of the candidate box category distribution and the category noun vectors. Category words in the text (belonging to) The average word vectors reflect the consistency between category words such as "bed / chair / suitcase" and the predicted categories of candidate boxes; Spatial consistency ,in Let be the spatial parameter vector of the candidate boxes. For locative words in the text (belonging to) The orientation semantic vector obtained by BiLSTM encoding This indicates splicing; the MLP outputs a one-dimensional scalar score. Detection confidence This refers to the target confidence level output by the TR3D detection head.

[0072] Weighting coefficients that can be learned or manually set (example example) ).

[0073] (3) Filtering, reordering, and weighted enhancement. For Sort in descending order, keeping the first few. Candidates (example) ); and perform weighted enhancements on the retained candidates:

[0074] in To enhance strength (example) ). Result This refers to feature enhancement for problem-related objectives.

[0075] Step S105: Cross-modal fusion and output of scene-language joint features. Enhance the features of problem-related objectives Text context semantic features Common input Transformer cross-modal fusion network.

[0076] (1) Schematic diagram of the fusion mechanism. Fusion dimensions (hidden dimensions) Example: 512, multiple heads Example: 8, number of stacking layers Example 4) and data flow direction as follows Figure 3 As shown.

[0077] (2) Target encoding sublayer. Spatial relation bias is introduced on the basis of standard multi-head self-attention. It is obtained by encoding the target relationship graph according to six types of relationships: relative distance, left and right, up and down, front and back, adjacency, and containment.

[0078] in , , , To enhance the feature matrix of the stacked target, For learnable projection matrices, This is the multi-head relation bias tensor.

[0079] (3) Text encoding sub-layer. Standard Transformer self-attention pairs are used. Perform internal dependency modeling and output .

[0080] (4) Bidirectional cross-attention fusion sublayer. Let the target encoding output be... Text encoding output is Perform two-way cross-attention:

[0081]

[0082] in It is a learnable projection matrix. This represents the semantic injection of textual semantics into each candidate target. This represents the spatial-visual injection of candidate targets for each text token.

[0083] (5) Residual fusion and multi-layer stacking. The cross-attention output is added to the original features and normalized, then passed through a feedforward network (FFN) to obtain the output of this layer, and then stacked. layer:

[0084] Final output scenario - language joint feature representation .

[0085] Step S106: Answer Prediction and Target Location like Figure 4 As shown, after obtaining the scene-language joint features, the two branches are executed in parallel.

[0086] (1) Answer prediction branch. Attention pooling is performed on the joint features from the text side to obtain... :

[0087] The classification head is used when predicting the answer for a closed set.

[0088] in For a pre-defined set of answers, When generating open-ended answers, then... As the initial state of the decoder, the free text answer is generated by the autoregressive transformation of the Transformer decoder.

[0089] (2) Target localization branch. Joint features for each candidate target. Predicted association score:

[0090] in For the sigmoid function, , These are learnable parameters. Select... The highest one or more candidate bounding boxes are used as the localization results, and the bounding box offset can be output by a single MLP layer. Refine the geometric parameters.

[0091] (3) Training objectives. The overall joint loss is adopted as follows:

[0092] in, Predict the loss for the answer. Locate the loss to the target. Classification loss for the target For TR3D detection of loss, , , and For the corresponding weighting coefficients, this example uses... .

[0093] (4) Example. When the input text is "Where is the blue suitcase in front of the right side of the bed?", this embodiment can not only output the corresponding location description or target answer, but also output the three-dimensional bounding box of the blue suitcase in the indoor scene. This allows for the practical verification of the answer in three-dimensional space.

[0094] Example 2 This embodiment further illustrates the training process of the method described in Embodiment 1, providing feasible details from training data organization to model deployment, see reference. Figure 5 The training process is shown in this embodiment. The training scheme described in this embodiment is applicable to the overall network of Embodiment 1, and can also be used as an independent training protocol to optimize the TR3D 3D object detection module, text encoding module, and cross-modal fusion module separately.

[0095] Step S201 Training Data Preparation The training data described in this embodiment includes the following five categories, and the acquisition and organization methods for each category are as follows.

[0096] (1) Indoor scene point cloud data. Indoor scene point clouds reconstructed using public datasets (such as ScanNet-V2, SUN RGB-D, 3RScan, Matterport3D) or self-collected RGB-D sequences are denoted as . , For scene indexing, This represents the number of points in the scene. Each point... The additional features are constructed in the same way as in step S101 (1) of Example 1.

[0097] (2) Optional image features. When the training data contains multi-view color images aligned with the point cloud. At that time, stage-4 feature maps are extracted for each image using a pre-trained 2D visual backbone ResNet-50. The multi-view projection features are obtained by back-projecting the camera's intrinsic and extrinsic parameters onto the point cloud. , as an optional input channel.

[0098] (3) Natural language text. It appears in pairs with scenes and is denoted as... , This serves as a sample index for the specific scenario; the types include question text, query text, description text, or instruction text.

[0099] (4) Standard Answer. (and) One-to-one correspondence, denoted as It can be a pre-set set of answers. The category labels in the text (closed set) can also be free text strings (open set).

[0100] (5) Target box annotations associated with the text. Denoteed as... Each of them It also includes 3D oriented bounding box parameters and target category.

[0101] (6) Data partitioning. All data will be partitioned according to the official partitioning method or by scenario-specific partitioning. The samples are divided into training set, validation set, and test set, with a proportional sample size of [missing information]. To ensure that all questions in the same scenario do not appear simultaneously in the training and test sets, thus avoiding scenario leakage.

[0102] Step S202 Training Paradigm Selection This embodiment supports two interchangeable training paradigms, and users can choose one to execute based on the data scale and computing power.

[0103] (1) Two-stage training paradigm. In the first stage, the TR3D 3D object detection model is pre-trained using indoor scene data with 3D bounding boxes, and the target is optimized as follows: In the second stage, the TR3D backbone is frozen or fine-tuned with a small learning rate, and its output candidate 3D bounding boxes are compared with geometric features. The input consists of the downstream text encoding module, the correlation modeling module, and the Transformer cross-modal fusion network. The optimization objective is... .

[0104] (2) End-to-end joint training paradigm. The TR3D detection module and the subsequent cross-modal understanding module are placed together in the computation graph. The joint loss function simultaneously constrains 3D detection, correlation modeling, cross-modal fusion, answer prediction and target localization, and backpropagation continues all the way to the TR3D backbone. This paradigm is suitable for scenarios with sufficient data and computing power.

[0105] Regardless of the paradigm used, the forward process is consistent with steps S101-S106 of Embodiment 1, and the forward calculation will not be described again in this embodiment.

[0106] Step S203: The specific form of the joint loss function and its components During the training phase, a joint loss function is used to optimize the model. The joint loss function is expressed as follows:

[0107] in Predicting loss for the answer, Locating the loss for the target Classify loss for target For TR3D detection of loss, These are the corresponding weighting coefficients. The specific forms of each component are as follows.

[0108] (1) Answer prediction loss The closed-set answer is obtained using category cross-entropy:

[0109] in , For indicator functions, This is the standard answer. The open set answer uses token-by-token cross-entropy:

[0110] in The target answer is represented by the number of tokens. Take into account the actual task or .

[0111] (2) Target location loss It consists of two parts: the association score binary classification loss and the bounding box regression loss.

[0112] The association score loss uses binary cross-entropy: , From candidate boxes Does the IoU with the nearest ground truth bounding box exceed a threshold (example)? )Sure; This refers to the association score output in step S106 of Example 1. The bounding box regression loss uses a 3D-IoU + Smooth-L1 composite form:

[0113] in The set of positive samples; For the predicted center / size / orientation offset, This corresponds to the actual offset. Example and .

[0114] (3) Target classification loss Cross-entropy is used to constrain the "text-related target category" predicted after cross-modal fusion.

[0115] This loss corresponds to the target category prediction result “used to help distinguish candidate targets with similar spatial relationships”.

[0116] (4) TR3D detection loss Designed using TR3D native technology:

[0117] in For the category Focal Loss, The regression losses are for center, size, and orientation, respectively.

[0118] (5) Weighting coefficients The settings are as follows. By adjusting the weights of each loss term, the optimization objectives among answer prediction, target localization, auxiliary classification, and candidate target detection can be balanced. This example uses... When the task is more focused on positioning, the adjustment can be increased. (like When TR3D recall is insufficient in the training data, it can be increased. Dynamic weights can also be used to adaptively balance the various loss scales.

[0119] Step S204: Train hyperparticipation to optimize scheduling To make the training process reproducible, this embodiment provides a set of exemplary hyperparameter configurations, which can be slightly adjusted by those skilled in the art around the combination according to actual computing power and data scale.

[0120] The optimizer uses AdamW with an initial learning rate of (End-to-end paradigm) or (When training TR3D only), weight decay The learning rate is scheduled using linear warmup (first 5% of steps) + cosine annealing; the batch size is 8 (based on 24GB of VRAM) in the example, and the gradient accumulation steps are 2 in the example; the total number of training epochs is 50 in the example; gradient pruning threshold. Mixed precision training (FP16) is used to save GPU memory.

[0121] If the text encoding module uses a pre-trained model such as BiLSTM, a smaller learning rate can be applied (example). And enable layer-wise learning rate decay (decay coefficient) This avoids the large learning rate corrupting the pre-trained weights. Transformer cross-modal fusion networks employ dropout (example). It is configured with LayerNorm pre-norm for stable training.

[0122] Step S205 Data Augmentation During the training phase, data augmentation is performed on the 3D scene data, applied independently with the probabilities described in the table below:

[0123] When performing any geometric augmentation, the corresponding real 3D bounding box of the sample must be used. Apply identical geometric transformations to ensure consistent annotation; color and point-level perturbations do not affect the ground truth bounding boxes. No content enhancement is applied to the text; optional token-level dropout (probabilistic) can be applied. To improve robustness.

[0124] Step S206 Training Process (End-to-End Paradigm) like Figure 6 As shown, each iteration is performed in the order described above; at the end of each epoch, evaluation is conducted on the validation set, and the optimal checkpoint is retained according to the metrics described in S207 below. Optionally, an exponential moving average (EMA, decaying average) is maintained for the model weights. EMA weights are used during inference to improve stability.

[0125] Step S207 Verification, Early Stop and Deployment (1) Verification indicators. For example... Figure 7 As shown, for 3D detection: mAP@0.25, mAP@0.5; for target localization: Acc@0.25, Acc@0.5 (the proportion of predicted bounding boxes with IoU exceeding the threshold); for answer prediction: Top-1 accuracy (closed set) or BLEU-4 / CIDEr (open set); the comprehensive metric uses a weighted average, with weights set as an example. These correspond to location, answer, and test, respectively.

[0126] (2) Early stopping strategy. Monitor the validation set composite index and stop training if there is no improvement after 10 consecutive epochs; retain the checkpoint with the best validation set composite index during training.

[0127] (3) Deployment. Export the optimal checkpoint as an inference graph and perform operator fusion and FP16 quantization on the TR3D backbone; disable all data augmentation during inference, and perform token length truncation if a pre-trained language model is used on the text encoding side (example). To control latency.

[0128] Example 3 In some alternative implementations, the spatial relationships between the candidate 3D targets can not only serve as input to the target encoding sublayer, but can also be constructed into a target relationship graph. The relationship representation is then extracted through graph structure encoding and fed into the Transformer cross-modal fusion network along with the target geometric features to improve the understanding of complex spatial relationship descriptions such as "between", "near", "right front", and "above".

[0129] In some alternative implementations, the answer prediction adopts an open-ended generation approach, in which the scene-language joint feature representation is further input into the decoder network to generate free-form answers word by word, thereby adapting to the text output requirements of different expression forms.

[0130] In some alternative implementations, the text-guided relevance modeling may first filter the top K candidate targets based on matching scores, and then perform cross-modal fusion on the filtered candidate targets to reduce computational overhead and improve target focusing capability.

[0131] Example 4 Based on the application principles of embodiments one to three above, this embodiment proposes a system for implementing the above method, the system comprising: Multimodal data acquisition module: used to acquire multimodal input data of indoor scenes, wherein the multimodal input data includes at least three-dimensional scene data and natural language text; 3D data processing module: used to input the 3D scene data into the pre-trained TR3D 3D target detection model, generate multiple candidate 3D target boxes, and extract the geometric feature representations corresponding to each candidate 3D target box; NLP processing module: used to encode the natural language text to obtain text context semantic features; Target feature generation module: used to perform correlation modeling on the geometric feature representation corresponding to the candidate 3D target box based on the semantic features of the text context, to obtain problem-related target enhancement features; Feature fusion module: used to input the problem-related target enhancement features and the text context semantic features into the Transformer cross-modal fusion network for interactive fusion to obtain scene-language joint feature representation; Output module: Based on the scene-language joint feature representation, it performs answer prediction and target localization simultaneously, and outputs scene understanding results, which include at least the question answer and one or more 3D target boxes associated with the question answer.

[0132] For details on the specific execution flow and interactions of each module, please refer to Examples 1-3 above, which will not be repeated here.

[0133] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0134] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0135] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0136] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0138] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0139] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0140] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0141] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0142] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal indoor 3D scene understanding method based on Transformer, characterized in that, Includes the following steps: Acquire multimodal input data of an indoor scene, wherein the multimodal input data includes at least three-dimensional scene data and natural language text; The three-dimensional scene data is input into a pre-trained TR3D three-dimensional object detection model to generate multiple candidate three-dimensional object boxes, and the geometric feature representations corresponding to each candidate three-dimensional object box are extracted. The natural language text is encoded to obtain text context semantic features; Based on the semantic features of the text context, the geometric feature representations corresponding to the candidate 3D target boxes are modeled for relevance, resulting in problem-related target enhancement features. The problem-related target enhancement features and the text context semantic features are input into the Transformer cross-modal fusion network for interactive fusion to obtain a scene-language joint feature representation; Based on the scene-language joint feature representation, answer prediction and target localization are performed simultaneously, and scene understanding results are output. The scene understanding results include at least the question answer and one or more three-dimensional target boxes associated with the question answer.

2. The method according to claim 1, characterized in that, The three-dimensional scene data includes point cloud data, and each point in the point cloud also includes at least one of the following additional features: color feature, normal vector feature, relative height feature, and multi-view projection feature; The natural language text includes at least one of question text, query text, description text, or instruction text.

3. The method according to claim 1, characterized in that, The step of inputting the 3D scene data into a pre-trained TR3D 3D object detection model to generate multiple candidate 3D object boxes and extracting the geometric feature representations corresponding to each candidate 3D object box includes: The point cloud data is normalized, denoised, voxelized, and sampled. Sparse convolutional networks are used to extract multi-level features from the voxelized point cloud to obtain voxel feature maps at different scales. The detection head outputs the center coordinates, size parameters, orientation parameters, category score, and confidence score of the candidate 3D target bounding boxes. For each candidate box, the RoI average pooling is performed on the coverage area of ​​the voxel feature map to obtain the initial geometric features. These features are then concatenated with the geometric parameters of the box through a layer of MLP encoding to obtain the final geometric feature representation of the candidate target.

4. The method according to claim 1, characterized in that, The pre-trained TR3D 3D target detection model is trained using indoor scene annotation data, which includes at least target category annotations and 3D directional bounding box annotations.

5. The method according to claim 1, characterized in that, The process of encoding the natural language text to obtain text context semantic features includes: The natural language text is segmented to obtain a token sequence; Map the word segmentation results to a sequence of word vectors; The word vector sequence is modeled in context using a bidirectional recurrent neural network, a pre-trained language model, or a Transformer text encoder to obtain a text context semantic feature sequence.

6. The method according to claim 5, characterized in that, The process of encoding natural language text to obtain semantic features of text context also includes a semantic saliency enhancement step: Pre-maintain word lists for four categories: directional words, color words, category words, and relational words; For each token segmentation, output the corresponding category one-hot label. A token can belong to multiple categories at the same time. The saliency of different types of token features is enhanced through a gating weighting mechanism: ， in: For learnable class saliency weights, One-hot tags for token categories, Features of the original token For the enhanced token features, T represents transpose.

7. The method according to claim 1, characterized in that, The process of modeling the relevance of the geometric feature representations corresponding to the candidate 3D target boxes based on the semantic features of the text context to obtain problem-related target enhancement features includes: The semantic features of the text context and the geometric features of the candidate target are mapped to a unified feature space through two learnable projection matrices. Attention pooling is applied to text-side features to obtain sentence-level semantic representations; Calculate the matching score between each candidate 3D bounding box and the natural language text; Sort the matching scores in descending order and retain the top K candidate targets; Perform weighted enhancements on the retained candidate targets: ， Where: β is the reinforcing strength To match scores, For original geometric features, For enhanced target features.

8. The method according to claim 7, characterized in that, The matching score is obtained by weighted fusion of four parts: , in: The dot product similarity between the sentence-level semantics of the text and the geometric features of the target in the common space; A score is awarded for the consistency between the candidate box category information and the target semantic description in the text. A score is given for the consistency between the spatial location of the candidate box and the location description in the text; The target confidence level output by the TR3D detection head; , , , These are the corresponding weighting coefficients.

9. The method according to claim 1, characterized in that, The Transformer cross-modal fusion network includes a text encoding sublayer, a target encoding sublayer, and a bidirectional cross-attention fusion sublayer, wherein: The text encoding sublayer uses the standard Transformer self-attention mechanism to model the semantic dependencies between words within the text. The target encoding sublayer is used to model the spatial and contextual relationships between candidate 3D targets; The bidirectional cross-attention fusion sublayer performs cross-attention computation from text to target and from target to text, respectively, to establish a bidirectional correspondence between natural language text semantics and candidate 3D targets.

10. The method according to claim 9, characterized in that, The spatial relationships between the candidate 3D targets include at least one of the following: relative distance relationship, left-right relationship, up-down relationship, front-back relationship, adjacency relationship, and containment relationship; and the spatial relationships can be encoded by constructing a target relationship graph and input together with the geometric feature representations corresponding to the candidate 3D target boxes into the Transformer cross-modal fusion network.

11. The method according to claim 1, characterized in that, The step of inputting the question-related target enhancement features and the text context semantic features into the Transformer cross-modal fusion network for interactive fusion includes: S1. Using text context semantic features as keys and values, and question-related target enhancement features as queries, perform the first-way cross-attention computation to obtain the target features after text semantic injection; S2. Using the question-related target enhancement features as keys and values, and the text context semantic features as queries, perform the second-way cross-attention computation to obtain the text features after injecting target space information; S3. Add the cross-attention calculation result to the original feature and process it through layer normalization and feedforward network to obtain the output of this layer; The process involves stacking multiple layers of S1-S3, ultimately outputting a joint feature representation of the scene and language.

12. The method according to claim 1, characterized in that, The step of simultaneously performing answer prediction and target localization based on the scene-language joint feature representation includes: outputting the target answer category or free text answer from the preset answer set through the answer prediction branch; The target localization branch outputs the localization results or association scores of one or more candidate 3D target boxes related to the natural language text, where: The scene understanding results also include target category prediction results related to natural language text, which are used to help distinguish candidate targets with similar spatial relationships; One or more candidate 3D target boxes associated with the natural language text are used to spatially validate the answer to the question, thereby reducing incorrect answers caused by relying solely on textual statistical priors.

13. The method according to claim 1, characterized in that, During the training phase, a joint loss function is used to optimize the model. This joint loss function includes at least the answer prediction loss, the target localization loss, the target classification loss, and the TR3D detection loss, and can be expressed as: in, Predict the loss for the answer. Locate the loss to the target. Classification loss for the target, For TR3D detection loss, , , and These are the corresponding weighting coefficients.

14. The application of the method according to any one of claims 1-13, characterized in that, The method is applicable to at least one of the following tasks: indoor 3D visual question answering task, indoor target referencing understanding task, indoor target spatial relationship reasoning task, or indoor scene semantic retrieval task.

15. A system for implementing the method according to any one of claims 1-13, characterized in that, The system includes: Multimodal data acquisition module: used to acquire multimodal input data of indoor scenes, wherein the multimodal input data includes at least three-dimensional scene data and natural language text; 3D data processing module: used to input the 3D scene data into the pre-trained TR3D 3D target detection model, generate multiple candidate 3D target boxes, and extract the geometric feature representations corresponding to each candidate 3D target box; NLP processing module: used to encode the natural language text to obtain text context semantic features; Target feature generation module: used to perform correlation modeling on the geometric feature representation corresponding to the candidate 3D target box based on the semantic features of the text context, to obtain problem-related target enhancement features; Feature fusion module: used to input the problem-related target enhancement features and the text context semantic features into the Transformer cross-modal fusion network for interactive fusion to obtain scene-language joint feature representation; Output module: Based on the scene-language joint feature representation, it performs answer prediction and target localization simultaneously, and outputs scene understanding results, which include at least the question answer and one or more 3D target boxes associated with the question answer.

Images