Three-dimensional scene description generation model training method, three-dimensional scene description generation method and electronic device

Abstract

The embodiment of the application provides a kind of three-dimensional scene description generation model training method, three-dimensional scene description generation method, electronic equipment, it is related to visual model technical field, the method is by parameter learnable point cloud token encoder to the local point piece coding of three-dimensional scene point cloud sample and obtains scene token vector sequence, after splicing learnable task vector sequence, input parameter frozen pre-training two-dimensional image encoder, obtain scene feature vector.Scenario feature vector and scene token vector sequence are decoded using autoregressive decoder, and the output predicted description text is outputed.Meanwhile, the text feature vector is obtained by encoding the reference description text using the parameter frozen text encoder, the similarity of the feature vectors of the matching pair and the mismatching pair is calculated to construct the contrastive learning loss, and the learnable parameters are optimized in combination with the prediction loss.The embodiment of the application improves the accuracy of cross-modal semantic alignment based on the semantic prior knowledge of the pre-trained model and the reinforcement of the contrastive learning loss.

Description

Technical Field

[0001] This application relates to the field of visual model technology, and in particular to a training method for a 3D scene description generation model, a 3D scene description generation method, and an electronic device. Background Technology

[0002] 3D scene description generation aims to depict the content of 3D scenes using natural language. It is a key task at the intersection of computer vision and natural language processing, with significant application value in robotics, autonomous driving, and other fields. However, this task faces two major challenges: first, the inherent sparsity and chaotic distribution of objects in point cloud data make it difficult to accurately encode 3D geometric structures and spatial relationships; second, existing methods mostly adopt a two-stage "detection-description" paradigm, relying on external object detectors to generate candidate boxes, which is prone to error propagation and has weak cross-modal alignment capabilities, resulting in a disconnect between visual features and textual semantics. Although end-to-end Transformer models have emerged, attempting to simultaneously complete object localization and description, they still fall short in utilizing large-scale visual language pre-training priors and strengthening cross-modal semantic grounding, making it difficult to generate descriptive text that is both faithful to the details of 3D scenes and fluent and accurate. Summary of the Invention

[0003] This application provides a training method for a 3D scene description generation model, a 3D scene description generation method, and an electronic device to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, embodiments of this application provide a method for training a three-dimensional scene description generation model, including: Based on a pre-trained and parameter-frozen 2D image encoder, the scene feature vector of 3D scene point cloud samples is calculated. The reference description text is encoded by a parameter-freezing-based text encoder to obtain a text feature vector; the reference description text is a descriptive text used to describe the three-dimensional scene point cloud sample scene. For multiple 3D scene point cloud samples belonging to the same training iteration, a contrastive learning loss is calculated based on the feature vector similarity between the 3D scene point cloud samples belonging to the same sample pair and the reference description text, and the feature vector similarity between the 3D scene point cloud samples belonging to different sample pairs and the reference description text. The feature vector similarity is the vector similarity between the scene feature vector and the text feature vector. The contrastive learning loss is directly proportional to the feature vector similarity of the same sample pair and inversely proportional to the feature vector similarity of different sample pairs. The scene feature vector is decoded based on a parameter-learnable decoder, and descriptive text is generated as output. Based on the total loss calculated by considering the contrastive learning loss and the generation loss, each learnable parameter is optimized until the training convergence condition is met; the generation loss is determined based on the loss between the generated description text and the reference description text.

[0005] In some embodiments of this application, the calculation of scene feature vectors for 3D scene point cloud samples based on a pre-trained and parameter-frozen 2D image encoder includes: A parameter-learnable point cloud token encoder encodes multiple points extracted from the 3D scene point cloud sample to obtain a scene token vector sequence. The scene token vector sequence and the task vector sequence are concatenated and input into a pre-trained and parameter-frozen two-dimensional image encoder to obtain scene feature vectors; the task vector sequence consists of learnable parameters.

[0006] In some embodiments of this application, the parameter-learnable point cloud token encoder encodes multiple points extracted from the 3D scene point cloud sample to obtain a scene token vector sequence, including: Multiple representative center points are sampled from the three-dimensional scene point cloud sample; For each representative center point, extract multiple neighbor points around the representative center point to construct multiple local point patches that correspond one-to-one with the multiple representative center points; Based on the point-by-point encoder, the local point patches are encoded separately to obtain the scene token vector corresponding to each local point patch, and the scene token vector sequence is obtained by combining them; the point cloud token encoder includes the point-by-point encoder with learnable parameters.

[0007] In some embodiments of this application, the parameter-freezing-based text encoder encodes the reference description text to obtain a text feature vector, including: The reference description text is segmented to obtain a text token sequence; Based on a parameter-learnable embedding table, an embedding vector is assigned to each word in the text token sequence to obtain a text embedding sequence. The text embedding sequence is input into the text encoder to obtain the text feature vector.

[0008] In some embodiments of this application, the method further includes, before calculating the contrastive learning loss: Based on a scene modal projection head with learnable parameters, the scene feature vector is projected onto a shared embedding space to obtain the scene projection feature vector. Based on a parameter-learnable text modality projector, the text feature vector is projected onto the shared embedding space to obtain the text projection feature vector; The feature vector similarity is used to represent the vector similarity between the scene projection feature vector and the text projection feature vector.

[0009] In some embodiments of this application, the decoder is an autoregressive Transformer decoder. The parameter-learnable decoder decodes the scene feature vector and outputs descriptive text, including: Based on the self-attention layer of the decoder, the hidden state matrix is ​​updated by encoding according to the currently generated language token sequence; Based on the cross-modal attention layer of the decoder, cross-attention computation is performed on the query matrix, key matrix, and value matrix to update the hidden state matrix; the query matrix is ​​the updated hidden state matrix, and the key matrix and the value matrix are generated based on the scene feature vector; The next language token is output based on the updated hidden state matrix, and the output language token is added to the generated language token sequence. The generated description text is output based on the complete sequence of language tokens output in the iterative output.

[0010] In some embodiments of this application, the two-dimensional image encoder is a visual encoder pre-trained by contrastive learning in the CLIP model, and the text encoder is a text encoder pre-trained by contrastive learning in the CLIP model.

[0011] Secondly, embodiments of this application provide a method for generating a three-dimensional scene description, including: Based on a pre-trained and parameter-frozen 2D image encoder, the scene feature vector of the 3D scene point cloud is calculated. The scene feature vector is decoded using a decoder, and descriptive text is generated as output. Wherein, the decoder is trained according to the method described in any one of the first aspects, and the pre-trained two-dimensional image encoder is the same as the two-dimensional image encoder used in the method described in any one of the first aspects.

[0012] In some embodiments of this application, the calculation of scene feature vectors for three-dimensional scene point clouds based on a pre-trained and parameter-frozen two-dimensional image encoder includes: The point cloud token encoder encodes multiple points extracted from the 3D scene point cloud to obtain a scene token vector sequence. The scene token vector sequence and the task vector sequence are concatenated and input into the two-dimensional image encoder to obtain the scene feature vector; Wherein, the point cloud token encoder and the task vector sequence are trained according to the method described in any one of the first aspects.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0016] Based on the training method for the 3D scene description generation model, the 3D scene description generation method, and the electronic device described above, this application has at least the following beneficial effects or advantages: This application employs a pre-trained, parameter-frozen 2D image encoder and text encoder, fully utilizing the semantic prior knowledge accumulated by the pre-trained model to effectively reduce training complexity. For multiple samples in the same training iteration, a contrastive learning loss is calculated. By maximizing the feature vector similarity of the same sample pair and minimizing the feature vector similarity of different sample pairs, scene feature vectors and text feature vectors are aligned across modally in a shared embedding space, enhancing the cross-modal semantic alignment effect. This application does not rely on an external detector, avoiding the propagation of detection errors. The trained 3D scene description generation model can output generated descriptive text that not only fits the geometric structure and spatial relationships of the 3D scene point cloud but also has accurate and fluent semantics. Based on the semantic prior knowledge of the pre-trained model and the enhancement of contrastive learning loss, the accuracy of cross-modal semantic alignment is improved.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0018] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0019] Figure 1A flowchart illustrating a training method for a 3D scene description generation model provided in an embodiment of this application is shown. Figure 2 A flowchart of a three-dimensional scene description generation method provided in an embodiment of this application is shown; Figure 3 A schematic block diagram of a three-dimensional scene description generation model provided in an embodiment of this application is shown; Figure 4 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0020] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0021] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following related technologies are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this application, all of which fall within the protection scope of the embodiments of this application. It should be noted that the application scenarios or application examples provided in this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions.

[0022] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0023] See Figure 1 The flowchart shown illustrates the training method for a 3D scene description generation model. This method specifically includes the following steps: Step 101: Calculate the scene feature vector of the three-dimensional scene point cloud samples based on the pre-trained and parameter-frozen two-dimensional image encoder. Step 102: The reference description text is encoded by the parameter-frozen text encoder to obtain the text feature vector; the reference description text is the description text used to describe the three-dimensional scene point cloud sample scene. Step 103: For multiple 3D scene point cloud samples belonging to the same training iteration, calculate the contrastive learning loss based on the feature vector similarity between the 3D scene point cloud samples belonging to the same sample pair and the reference description text, and the feature vector similarity between the 3D scene point cloud samples belonging to different sample pairs and the reference description text. The feature vector similarity is the vector similarity between the scene feature vector and the text feature vector. The contrastive learning loss is directly proportional to the feature vector similarity of the same sample pair and inversely proportional to the feature vector similarity of different sample pairs. Step 104: Decode the scene feature vector based on the parameter learnable decoder and output the descriptive text. Step 105: Optimize each learnable parameter based on the total loss calculated by considering the contrastive learning loss and the generation loss, until the training convergence condition is met; the generation loss is determined based on the loss between the generated description text and the reference description text.

[0024] 3D scene point cloud samples are unstructured data containing additional features such as 3D coordinates and color normals. A pre-trained, parameter-frozen 2D image encoder can employ an architecture pre-trained on large-scale image-text data, with its weights remaining fixed during training. This allows for the reuse of existing semantic prior knowledge and avoids knowledge forgetting during training. The reference description text can be manually annotated and is natural language text used to describe the 3D scene point cloud samples.

[0025] In some implementations, the parameter-frozen text encoder and the 2D image encoder can originate from the same pre-trained model, such as the CLIP model, thereby converting the reference descriptive text into text feature vectors semantically aligned with the scene feature vectors. Specifically, the 2D image encoder can be a visual encoder pre-trained through contrastive learning within the CLIP model, and the text encoder can be a text encoder pre-trained through contrastive learning within the CLIP model.

[0026] In a single training iteration, multiple 3D scene point cloud samples and their corresponding reference descriptive texts are processed. A single sample pair refers to a 3D scene point cloud sample and its matching reference descriptive text, while different sample pairs are non-matching scene point clouds and texts. In some implementations, feature vector similarity can be obtained by calculating the cosine similarity between the scene feature vector and the text feature vector. For example, the contrastive learning loss can use the InfoNCE formula, with the distribution sharpness controlled by a temperature parameter to maximize the similarity of matched pairs and minimize the similarity of non-matched pairs.

[0027] The learnable decoder employs an autoregressive generation model, progressively generating descriptive text based on scene feature vectors. The generation loss measures the difference between the generated description and the reference description. The total loss is achieved by integrating the learning loss and the generation loss, updating the learnable parameters throughout the model based on the total loss. The training convergence condition can be set to loss stability or reaching a preset number of iterations.

[0028] This application employs a pre-trained, parameter-frozen 2D image encoder and text encoder, fully utilizing the semantic prior knowledge accumulated by the pre-trained model to effectively reduce training complexity. For multiple samples in the same training iteration, a contrastive learning loss is calculated. By maximizing the feature vector similarity of the same sample pair and minimizing the feature vector similarity of different sample pairs, scene feature vectors and text feature vectors are aligned across modally in a shared embedding space, enhancing the cross-modal semantic alignment effect. This application does not rely on an external detector, avoiding the propagation of detection errors. The trained 3D scene description generation model can output generated descriptive text that not only fits the geometric structure and spatial relationships of the 3D scene point cloud but also has accurate and fluent semantics. Based on the semantic prior knowledge of the pre-trained model and the enhancement of contrastive learning loss, the accuracy of cross-modal semantic alignment is improved.

[0029] In some embodiments of this application, the scene feature vector of a three-dimensional scene point cloud sample is calculated based on a pre-trained and parameter-frozen two-dimensional image encoder, including: encoding multiple points extracted from the three-dimensional scene point cloud sample based on a parameter-learnable point cloud token encoder to obtain a scene token vector sequence; concatenating the scene token vector sequence and the task vector sequence, and inputting them into the pre-trained and parameter-frozen two-dimensional image encoder to obtain the scene feature vector; the task vector sequence is a learnable parameter.

[0030] The parameter-learnable point cloud token encoder is used to process multiple points extracted from point cloud samples of a 3D scene. In some implementations, the point cloud token encoder can extract the local geometric and semantic features of each point through a multilayer perceptron (MLP) and output a sequence of scene token vectors. The sequence length is determined according to the number of samples extracted.

[0031] In some implementations, the task vector sequence consists of a set of learnable embedding vectors, initially set to index embeddings and continuously updated during training. This aggregates global semantic cues and guides the model to focus on describing the generation task. Concatenating the scene token vector sequence with the task vector sequence results in an input sequence that simultaneously contains local features and task context.

[0032] In some implementations, a 2D image encoder can process sequences through multi-layer self-attention and feedforward networks to output scene feature vectors that fuse geometric and semantic information. Freezing the weight parameters of the 2D image encoder can ensure that pre-trained knowledge is not lost.

[0033] For example, for an indoor scene point cloud sample containing 40,000 points, 1024 points are extracted by sampling from the farthest point. The point cloud token encoder encodes these points into a 1024-dimensional scene token vector sequence, and the task vector sequence is set to 16 learnable vectors. The two are concatenated into a sequence of length 1040, which is input into the frozen CLIP visual encoder, and after processing, a 512-dimensional scene feature vector is output.

[0034] The point cloud token encoder provided in this application converts unstructured point clouds into structured vector sequences, facilitating subsequent processing by a 2D image encoder. The task vector sequence is injected into the task context, improving the adaptability of the 3D scene description generation model to the description generation task. The concatenated vector sequence is input to the parameter-frozen 2D image encoder, which can fully reuse the semantic knowledge acquired during pre-training, effectively capturing the geometric and semantic information of the 3D scene while reducing training computational overhead, providing high-quality feature input for cross-modal alignment and text generation.

[0035] In some embodiments of this application, a parameter-learnable point cloud token encoder encodes multiple points extracted from a 3D scene point cloud sample to obtain a scene token vector sequence. This includes: sampling multiple representative center points in the 3D scene point cloud sample; for each representative center point, extracting multiple neighboring points around the representative center point to construct multiple local point patches that correspond one-to-one with the multiple representative center points; encoding the multiple local point patches separately based on a point-by-point encoder to obtain the scene token vector corresponding to each local point patch, and combining them to obtain a scene token vector sequence. The point cloud token encoder includes a parameter-learnable point-by-point encoder.

[0036] In some implementations, representative center points can be selected from 3D scene point cloud samples using the farthest point sampling (FPS) algorithm. This algorithm iteratively selects the point farthest from the selected point set, reducing redundancy in dense areas and preserving structure in sparse areas. The number of samples can be set according to the point cloud density and model requirements, such as 512 or 1024.

[0037] Furthermore, for each representative center point, in some implementations, 32 or 64 neighboring points can be extracted using the K-nearest neighbor algorithm to construct a local point patch. Each local point patch contains a set of spatially adjacent points, thereby reflecting the local geometric structure.

[0038] The point cloud token encoder includes a parameter-learnable point-by-point encoder. In some implementations, the point-by-point encoder may include a multilayer perceptron, which extracts deep features of each local point patch through linear transformation and nonlinear activation functions. The scene token vectors corresponding to each local point patch are combined in the sampling order representing the center point to form a scene token vector sequence, thereby taking into account both local details and global distribution.

[0039] For example, 512 representative center points are selected from the farthest point sampling of the 3D scene point cloud sample, and 32 neighboring points are extracted for each center point to construct a local point patch. The point-to-point encoder encodes each local point patch and outputs a 512-dimensional scene token vector. The vectors corresponding to the 512 local point patches are combined in the sampling order to obtain a scene token vector sequence of length 512.

[0040] Sampling at the farthest point ensures spatial coverage representing the center point, and constructing local point patches more accurately reflects the local geometric structure, reducing the processing difficulty caused by the sparsity of point clouds. By extracting deep geometric features of local point patches through a point-by-point encoder, the combined scene token vector sequence preserves the local details and global spatial relationships of the 3D scene, providing structured, high-dimensional feature support for subsequent feature fusion and cross-modal alignment, and improving the model's ability to capture scene geometry.

[0041] In some embodiments of this application, a text encoder based on parameter freezing encodes a reference description text to obtain a text feature vector, including: performing word segmentation on the reference description text to obtain a text token sequence; assigning a corresponding embedding vector to each word in the text token sequence based on a parameter-learnable embedding table to obtain a text embedding sequence; and inputting the text embedding sequence into the text encoder to obtain a text feature vector.

[0042] The word segmentation operation of the reference description text is performed by the sub-word tokenizer. The sub-word tokenizer can decompose registered words and unregistered words into known sub-word units. Each known sub-word unit is a text token. The tokens are combined to obtain a text token sequence. The length of the text token sequence is related to the text length.

[0043] A parameter-learnable embedding table is used to map each text token in a text token sequence to a fixed-dimensional embedding vector, each of which is a text embedding vector, forming a text embedding sequence. For example, the embedding table is typically 512 or 768-dimensional, and its parameters can be updated along with the training process of the 3D scene description model, thereby adaptively learning the semantic representation of the vocabulary.

[0044] In some implementations, to preserve word order information, positional encoding can be added to the text embedding sequence. Positional encoding can employ a sine function or parametric encoding. The text embedding sequence with added positional encoding is then input to a text encoder with frozen parameters. This text encoder is then processed through a multi-layer self-attention and feedforward network to extract global semantic features as the text feature vector.

[0045] This application embodiment uses a learnable embedding table to enable the embedding vectors to adapt to the semantic requirements of the 3D scene description generation task. By reusing pre-trained knowledge through a parameter-frozen text encoder, the generated text feature vectors can more accurately represent the global semantics of the reference description text.

[0046] In some embodiments of this application, before calculating the contrastive learning loss, the method further includes: projecting scene feature vectors onto a shared embedding space based on a parameter-learnable scene modal projector to obtain scene projection feature vectors; projecting text feature vectors onto a shared embedding space based on a parameter-learnable text modal projector to obtain text projection feature vectors; and using feature vector similarity to represent the vector similarity between scene projection feature vectors and text projection feature vectors.

[0047] In some implementations, both the scene modal projection head and the text modal projection head include a multilayer perceptron in their structure. For example, this may include a two-layer multilayer perceptron with learnable parameters. The scene modal projection head and the text modal projection head are used to project feature vectors from different modalities into the same shared embedding space. The scene modal projection head receives scene feature vectors and, in some implementations, maps them to an intermediate dimension through a first-layer linear transformation, introduces non-linearity through a ReLU activation function, and then maps them to the target dimension through a second-layer linear transformation. The text modal projection head can process text feature vectors using the same structure, ensuring that the output text projection feature vector has the same dimension as the scene projection feature vector.

[0048] In some implementations, the target dimension of the shared embedding space can be set to 512 dimensions, and the projected feature vectors can be L2 normalized so that the vectors lie on the unit hypersphere. At this time, the dot product of the two vectors can directly represent the cosine similarity, providing a unified metric for the calculation of contrastive learning loss.

[0049] This application embodiment achieves dimensional unification and alignment of scene feature vectors and text feature vectors in a shared embedding space through two projection heads, and the generated text projection feature vectors and scene projection feature vectors enhance cross-modal semantic relevance.

[0050] In some embodiments of this application, the decoder is an autoregressive Transformer decoder. Based on a parameter-learnable decoder, the scene feature vector is decoded, and descriptive text is generated. This includes: an autoattention layer based on the decoder, encoding the currently generated language token sequence and updating the hidden state matrix; a cross-modal attention layer based on the decoder, performing cross-attention calculations on the query matrix, key matrix, and value matrix, and updating the hidden state matrix; the query matrix is ​​the updated hidden state matrix, and the key matrix and value matrix are generated based on the scene feature vector; the next language token is output based on the updated hidden state matrix, and the output language token is added to the generated language token sequence; and descriptive text is generated based on the iteratively output complete language token sequence.

[0051] The autoregressive Transformer decoder iteratively outputs token vectors, each of which can be decoded into a subword based on an embedding table, ultimately yielding the complete generated descriptive text. The autoregressive Transformer decoder includes a self-attention layer and a cross-modal attention layer. The self-attention layer models the internal contextual dependencies of the language. The currently generated language token sequence is input into the self-attention layer, and a causal mask is applied using the Q matrix (query matrix), K matrix (key matrix), and V matrix (value matrix) to shield future tokens. The hidden state matrix is ​​then updated through similarity calculation and weighted aggregation. The query matrix of the cross-modal attention layer is the updated hidden state matrix. The key and value matrices are generated from scene feature vectors through two independent linear layers, storing scene semantic indexes and detailed information, respectively. Cross-attention calculation determines attention weights based on the similarity between the query and key matrices, and then weights and aggregates them with the value matrix, integrating scene information into the hidden state matrix. Based on the updated hidden state matrix, the probability distribution of the next language token is output through a linear layer and a softmax function. The token with the highest probability is added to the sequence, iterating until the final token is generated or the maximum length is reached.

[0052] For example, the decoder initially inputs the [BOS] token, and the self-attention layer updates the initial hidden state matrix. The cross-modal attention layer uses this matrix as a query, calculates the key-value matrix generated by the scene feature vector, updates the hidden state matrix, and outputs the first token A. A is added to the sequence, and the above process is repeated to generate tokens corresponding to sub-words such as compact, office, with, and desk, until the [EOS] token is output. Then, each token is converted into a sub-word to form the complete descriptive text.

[0053] In this embodiment, the interaction between the generated language and the 3D scene is realized through a cross-modal attention layer, so that the description text generated iteratively token by token not only conforms to the language syntax, but also fits the scene geometry and spatial relationships, effectively solving the problem of text and scene disconnection, and taking into account both the grammatical accuracy of the generated description text and the accuracy of scene description.

[0054] In some embodiments of this application, the two-dimensional image encoder is a visual encoder pre-trained by contrastive learning in the CLIP model, and the text encoder is a text encoder pre-trained by contrastive learning in the CLIP model.

[0055] Specifically, the 2D image encoder can employ a visual encoder pre-trained through contrastive learning within the CLIP model. This encoder uses a Transformer architecture, is pre-trained on a large-scale image-text dataset, and possesses powerful visual feature extraction and semantic understanding capabilities. Its input is a concatenated sequence of scene token vectors and task vector sequences, and the processed output is a scene feature vector containing 3D scene geometric and semantic information. During training, all parameters of this encoder are frozen to avoid losing pre-training knowledge and reduce computational overhead.

[0056] The text encoder uses a contrastive learning-pre-trained text encoder from the same CLIP model, sharing the pre-trained semantic space with the visual encoder to ensure that the scene feature vectors and text feature vectors can be directly calculated for similarity. The text encoder also freezes parameters, retaining the linguistic knowledge gained from large-scale pre-training, and accurately encodes the semantic information of the reference descriptive text.

[0057] For example, the ViT-B / 32 visual encoder of the CLIP model can be selected as a two-dimensional image encoder, all its weights are frozen, and the concatenated token sequence is processed to output a scene feature vector. The text Transformer encoder of the same CLIP model is used as a text encoder, and after freezing the parameters, the embedding sequence of the reference description text is converted into a text feature vector, with both having the same dimension and semantic alignment.

[0058] refer to Figure 2 This application also provides a method for generating a three-dimensional scene description, including: Step 201: Calculate the scene feature vector of the 3D scene point cloud based on the pre-trained and parameter-frozen 2D image encoder. Step 202: Decode the scene feature vector based on the decoder and output the descriptive text. The decoder is trained according to the training method of the three-dimensional scene description generation model provided in the embodiments of this application, and the pre-trained two-dimensional image encoder is the same as the two-dimensional image encoder used in the training method of the three-dimensional scene description generation model provided in the embodiments of this application.

[0059] The 3D scene description generation method uses a pre-trained 2D image encoder with frozen parameters from the training phase. This 2D image encoder is the same as the encoder used in the training phase, ensuring consistency in feature extraction logic between the generation and training processes. The 3D scene point cloud is the unstructured 3D data to be described, containing geometric and appearance information of the scene. The 2D image encoder processes the feature sequences related to the 3D scene point cloud, outputting a scene feature vector that maintains consistency with the feature distribution from the training phase.

[0060] The decoder is trained during the training phase, such as an autoregressive Transformer decoder. Its parameters are optimized using a joint loss based on contrastive learning loss and generation loss, enabling it to generate descriptive text based on scene feature vectors. The decoder generates natural language descriptive text token-by-token through iterative computation of self-attention and cross-modal attention based on scene feature vectors.

[0061] For example, the 3D point cloud of an indoor office scene is converted into a feature sequence and input into a frozen CLIP visual encoder to obtain a scene feature vector. This vector is then input into a trained autoregressive Transformer decoder, which iteratively generates tokens such as "A compact office with a desk and chairs" to form a complete scene description text.

[0062] This application employs a decoder trained using the training method of the 3D scene description model provided in this application, and a 2D image encoder identical to the one used in the training process of the 3D scene description model. This ensures that the generation process fully utilizes the cross-modal alignment capabilities and text generation knowledge learned during the training phase. The end-to-end generation process requires no external detector or additional preprocessing, balancing generation efficiency and accuracy. The generated descriptive text possesses both good fluency and accurately matches the geometric structure and semantic information of the 3D scene, effectively meeting the scene description needs in practical applications.

[0063] In some embodiments of this application, step 201 calculates the scene feature vector of the three-dimensional scene point cloud based on a pre-trained and parameter-frozen two-dimensional image encoder, including: encoding multiple points extracted from the three-dimensional scene point cloud based on a point cloud token encoder to obtain a scene token vector sequence; concatenating the scene token vector sequence and the task vector sequence and inputting them into the two-dimensional image encoder to obtain the scene feature vector; wherein, the point cloud token encoder and the task vector sequence are trained according to the training method of the three-dimensional scene description generation model provided in the embodiments of this application.

[0064] The point cloud token encoder is a three-dimensional scene description model trained by the training method of the three-dimensional scene description model provided in the embodiments of this application. The parameters of the point cloud token encoder have been optimized. Multiple points are extracted from the three-dimensional scene point cloud, and these points are encoded by the point cloud token encoder with optimized parameters to output a scene token vector sequence.

[0065] The task vector sequence is also a pre-optimized set of parameters, having learned the relevant contextual information during training. The scene token vector sequence and the task vector sequence are concatenated in a preset order to form a unified input sequence, which includes local features of the scene, global distribution, and task context.

[0066] For example, a trained point cloud token encoder is used to encode 1024 points extracted from the point cloud of an outdoor park 3D scene, resulting in a scene token vector sequence. This sequence is then concatenated with a trained 16-dimensional task vector sequence and input into a frozen CLIP visual encoder. After processing, a 512-dimensional scene feature vector is output for subsequent decoding and generation.

[0067] It is understood that the three-dimensional scene description generation method provided in this application embodiment uses the three-dimensional scene description generation model described in this application embodiment, and is trained based on the training method of the three-dimensional scene description generation model provided in this application embodiment. Therefore, the implementation methods and technical effects not described in detail in this embodiment can be referred to the training method of the three-dimensional scene description generation model provided in this application embodiment, and will not be described again.

[0068] The following example illustrates the training method and model structure of the 3D scene description generation model provided in the embodiments of this application.

[0069] refer to Figure 3 This is a sample 3D scene description generation model, which includes: a 3D scene encoder, a text encoder, and a decoder.

[0070] Early 3D description generation methods typically employed a two-stage detection-then-description paradigm, where target proposals were first detected from the point cloud, and then each target was described separately. While these approaches had some effectiveness, the two-stage pipeline could lead to a significant performance degradation. First, the detection stage often generates redundant bounding boxes, requiring adjustment using non-maximum suppression (NMS), which introduces additional hyperparameters and increases computational overhead. Second, the cascaded design of the detection-then-description process makes description generation highly dependent on the quality of the detection stage. Against this backdrop, exploring single-stage end-to-end 3D description generation models has attracted widespread attention. Some related techniques employ the Transformer framework, simultaneously locating and describing targets during a single forward propagation inference, improving efficiency and performance. Other techniques introduce bidirectional contextual attention mechanisms to separate target localization from contextual feature aggregation in a 3D scene, or employ late aggregation strategies, capturing local target details and global contextual information through novel aggregators. Despite some progress made by these technical solutions, 3D description generation still faces significant challenges, particularly in modeling spatial relationships and aligning 3D visual data with textual semantics. Describing complex spatial layouts requires models to understand 3D geometry and the relative positions of objects, and encoding and reasoning this information is no easy task. Bridging the gap between 3D modality and language is also challenging. Existing methods treat vision and language as separate stages, resulting in weak cross-modal interaction and poor alignment between visual and textual representations.

[0071] This application provides a 3D scene description model based on CoCa (Contrastive Captioner), such as... Figure 2 The diagram illustrates a specific example structure. The core idea is to train a 3D scene encoder and a text encoder together with a shared contrastive learning objective, while simultaneously training a multimodal decoder to generate descriptions. By coupling these tasks, the 3D CoCa model learns to align the 3D representation with the description depth in the joint feature space.

[0072] (1) 3D scene encoder The 3D scene encoder is responsible for converting unstructured 3D scene point clouds into structured semantic features: scene feature vectors. Specifically, representative points are first selected using FPS (farthest-point sampling), and then features of each local patch are extracted using MLP (Multilayer Perceptron) to obtain scene tokens. Multiple scenario tokens Composition of point cloud token vector sequence E pIt is concatenated with the task vector sequence (or task token) to guide the model to focus on describing the generated task.

[0073] After the 3D scene point cloud (3D scene point cloud samples during the training phase) is input into the 3D scene encoder, it first undergoes FPS sampling to select representative points, and then is processed by the MLP-based point-by-point encoder to obtain a scene token vector sequence. This sequence is concatenated with the task vector sequence, and the resulting point cloud token vector sequence E is generated. p Input CLIP image encoder, output scene feature vector Among them, the CLIP image encoder is a pre-trained two-dimensional image encoder whose parameters are frozen during this training process, while the parameters and task vector sequence of the MLP-based pointwise encoder are all learnable parameters during this training process.

[0074] The role of the 3D scene encoder is to transform unstructured point clouds into a set of latent tokens: scene token vectors, which capture the geometric and semantic content of the scene. The 3D scene encoder combines point-based 3D processing with a frozen 2D CLIP visual backbone network model to effectively capture geometric and semantic information. The 3D scene encoder comprises three components: (i) a point cloud tokenizer: which divides the raw point cloud into patch tokens; (ii) a task vector sequence: a set of learnable task tokens that injects 3D description generation context; and (iii) a frozen CLIP image encoder: a 2D CLIP visual Transformer encoder used to encode the stitched token sequence. The 3D scene encoder is used to transform unstructured 3D input into a structured representation suitable for multimodal reasoning.

[0075] (i) Point cloud tokenizer Given a 3D scene point cloud Each of the N points is represented by three-dimensional coordinates. The 3D scene point cloud is first converted into a discrete token sequence suitable for Transformer-based processing, described by F additional features (such as color, normal vector, height, or multi-view descriptors). To improve spatial coverage, farthest-point sampling (FPS) is used to select M representative points as patch centers. FPS achieves uniform sampling by reducing redundancy in dense regions while preserving structure in sparse regions. For each center, its K nearest neighbors are collected to construct a local patch, resulting in M ​​patches. Each patch contains K spatially adjacent points. Each patch extracts local geometric and appearance features through a lightweight pointwise encoder (i.e., a series of multilayer perceptrons), generating a sequence of M point tokens, each token represented as... Dimensional embedding:

[0076] in, This represents the embedding of the i-th patch. This tokenization strategy effectively transforms unstructured 3D input into a structured sequence, balancing local details (within each K-point patch) and global coverage (across M sampled patches), enabling the Transformer backbone network to perform efficient downstream processing.

[0077] (ii) Task vector sequence Although point cloud tokens encode local visual features, they lack explicit task-aware capabilities. To guide the model in 3D description generation, a set of learnable task tokens—embedding vectors—is introduced, which are pre-loaded into the point token sequence. Based on a cue-tuning strategy, these task tokens are initialized with different fixed values ​​(e.g., index embeddings) and updated during training. As high-level cues, the task tokens tune the model for description generation objectives. Through self-attention, they aggregate global semantic cues from the entire point cloud, such as scene layout and salient objects, which are crucial for generating descriptive language. In effect, they act as shared contextual anchors, biasing the encoder towards language-relevant features in the 3D scene.

[0078] (iii) Frozen CLIP image encoder Get M tokens and After obtaining the task tokens, concatenate them into a unified input sequence:

[0079] in, This represents the embedding of the j-th task token. Then, this length is... The sequence of tokens is fed into a frozen CLIP visual Transformer, which is adaptable from the original CLIP image encoder architecture. All CLIP weights are frozen to preserve their pre-trained visual representations and prevent catastrophic forgetting during training. This not only stabilizes the optimization process but also significantly reduces memory and computational overhead, as most parameters remain fixed. The CLIP encoder processes the token sequence and outputs a series of latent embeddings that jointly capture 3D geometry and task context. From these, a global scene representation is extracted. This representation serves as a semantically rich embedding based on a 3D scene describing the generated target. This feature vector is then used for subsequent comparison and alignment with the text representation.

[0080] (2) Text encoder 1) Generate text feature vectors The 3D scene encoder extracts visual features from the point cloud, while the text encoder transforms natural language descriptions into a semantically aligned embedding space. The text encoder employs a Transformer-based CLIP text encoder with its weights frozen to preserve the rich linguistic knowledge gained during large-scale pre-training. Freezing the text encoder ensures that all descriptions are assigned to the same semantic space as the CLIP visual representation, enabling effective alignment with the 3D scene embeddings.

[0081] The text encoder is used during the training phase to process the reference description text corresponding to the 3D scene point cloud samples, and is responsible for converting the input text into scene feature vectors. Semantic features of alignment: text feature vector The text tokenizer is used to break down the input text into sub-token tokens (such as compact, office, etc.) and add a [CLS] token. The parameter-frozen CLIP text encoder is used to embed the text into a sequence. Input a pre-trained CLIP text module (parameters frozen), output encoded text features. .

[0082] The reference description text (existing during the training phase) is input into the text token encoder to obtain a sequence of text tokens, which is then mapped to a sequence of text embeddings using a parameter-learnable embedding table. The input parameters are frozen in the CLIP text encoder, and the output is a text feature vector. Text feature vectors With scene feature vector They jointly participate in the calculation of contrastive learning loss. The CLIP text encoder and CLIP image encoder are from the same CLIP model and were obtained through the same pre-training.

[0083] For example, the point cloud data of a 3D scene, such as an indoor scene, corresponds to the matching reference description text of the same sample pair as "A compact office with a desk, chair and cabinet."

[0084] By using text feature vectors With scene feature vector Aligning to the same space, the contrastive learning loss is calculated, and the contrastive learning loss is used to constrain the parameter optimization of the training process, enabling the 3D scene description model to learn semantic matching between the 3D scene and the text.

[0085] (i) Text tokenizer Given an input sentence T, tokenize it into a sequence of L word tokens using a subtotal tokenizer. This tokenizer ensures robustness to out-of-vocabulary (OV) words by breaking them down into known units. Each token... An embedded table is assigned a learned one. From the dimensional vector, we obtain the text token sequence:

[0086] in, This represents the embedding of the i-th token. A special sequence start token is used as the sentence-level aggregator, and positional encoding is added to preserve word order, which is crucial for capturing the syntactic and semantic structures in natural language.

[0087] (ii) Frozen CLIP text encoder Text Embedded Sequence Processed by CLIP text Transformer encoder, which contains Multi-head self-attention and feedforward networks. Let... Given the input; the hidden state matrix is ​​updated as follows:

[0088] Each block includes self-attention, layer normalization, and an MLP sublayer. All weights in the text encoder are kept frozen to preserve the language knowledge gained from large-scale image-text pre-training. This also helps prevent overfitting, especially given the limited size of the 3D description generation dataset.

[0089] From the last layer, extract the hidden state corresponding to the special [CLS] token as the global text representation: This vector encoding describes the overall semantics and serves as a language-side embedding in the contrastive learning module. This is achieved by utilizing a fixed CLIP text encoder. Located in the same embedding space as CLIP visual features, it can be integrated with 3D scene feature vectors. Align directly.

[0090] 2) Align scene feature vectors with text feature vectors To align the heterogeneous modalities of 3D point clouds and text, a contrastive learning objective is employed to map 3D scene feature vectors and text feature vectors to a shared embedding space. In this space, matched 3D-text pairs are brought closer together, while unmatched pairs are pushed apart. This strategy borrows from the CLIP training paradigm, promoting cross-modal associations through discriminative alignment.

[0091] (i) Feature alignment Before calculating similarity, two learnable projectors are used to project the 3D scene feature vector and the text feature vector into a shared embedding space, respectively. Specifically, each projector applies two parallel two-layer MLPs:

[0092]

[0093] in, and These represent the projection heads for processing the 3D modality and the text modality, respectively. Each projection head consists of the following components in sequence: a first linear layer, a ReLU activation function, and a second linear layer. These modules not only unify the feature dimensions but also adapt the embeddings to achieve optimal cross-modal alignment. To facilitate similarity calculation, L2 normalization is applied to the two projection vectors:

[0094]

[0095] This ensures that the features lie on a unit hypersphere, so their similarity can be calculated as the cosine of the angle between them, which is a key step in contrastive loss.

[0096] (ii) Contrastive loss function Utilizing projected and normalized features, a contrastive learning objective is applied to align paired 3D scenes and descriptions. Following the InfoNCE formula widely used in CLIP, a training batch of N 3D-text pairs is considered, and the pairwise cosine similarity between all scene-description combinations is calculated. For the i-th scene and j-th text in a batch, the similarity is defined as:

[0097] This corresponds to the dot product of unit-normalized vectors. The contrastive loss maximizes the similarity between matched pairs (i=j) while minimizing the similarity between unmatched pairs (i≠j). For each scene i, the loss is:

[0098] Here, τ is a learnable temperature parameter controlling the sharpness of the softmax distribution. This loss encourages the model to assign high similarity to correct 3D-text pairs while suppressing alignment with incorrect matches. Through training with this objective, the model learns a shared embedding space where the 3D scene and its corresponding description are closely aligned, thereby promoting effective cross-modal understanding.

[0099] (3) Decoder The final component of 3D CoCa is the multimodal fusion decoder, which is responsible for generating natural language descriptions based on the input 3D scene. It is implemented as an autoregressive Transformer decoder, equipped with a cross-attention mechanism, integrating the 3D scene context in each generation step. Specifically, the decoder can be called a caption decoder, based on the encoded scene feature vectors. By using a cross-attention layer, autoregressive descriptions are generated (e.g., outputting "A compact office with a desk, chair, storage cabinet, and accents of greenery"), and the generation quality is optimized using a captioning loss.

[0100] As a conditional language model, the decoder generates descriptions token-by-token. Starting with a special sequence of initial tokens, at each time step t, descriptions are generated based on previously generated tokens via causal self-attention. Model intra-sentence dependencies.

[0101] At the same time, the decoder incorporates visual context into each prediction by embedding 3D scenes across attentional points. By leveraging the semantically aligned 3D features obtained during the contrast phase, the decoder generates descriptions that are not only grammatically coherent but also faithful to the 3D content. The integration of causal attention and cross-modal attention ensures that the output is linguistically fluent and visually accurate.

[0102] Scene feature vector The linear layer of the input decoder generates the K and V matrices. Based on the language token sequence currently generated by the decoder, the Q matrix is ​​obtained after processing. The Q matrix, K matrix, and V matrix are processed by the cross-modal attention layer to perform cross-attention calculation. The hidden state matrix is ​​iteratively updated by combining the feedforward network and the residual normalization layer. The output language token sequence is then used to generate the descriptive text. At the same time, the language token sequence and the reference descriptive text participate in the generation loss calculation.

[0103] To integrate 3D visual context into the description generation process, the decoder includes a cross-modal attention layer. Within each decoder block, the cross-attention module enables the decoder to focus on scene feature vectors generated by the scene encoder. In form, let's set This represents the query matrix derived from the current hidden state of the decoder. , These are the key and value matrices derived from the 3D scene embedding. The cross-attention computation is as follows:

[0104] in, This is the key dimension. This operation allows each location in the decoder to selectively integrate relevant 3D information, guided by attention weights. By adjusting the decoder based on scene features at each generation step, the attention mechanism ensures that the output captures the global context and fine-grained spatial details of the 3D input. The generated description is based on visual content and reflects the structural layout of the scene.

[0105] To balance linguistic coherence and visual accuracy, the cross-attention layers and self-attention layers within the decoder are interleaved. This design allows for dynamic interaction between textual dependencies and visual context, enabling the model to generate fluent and semantically accurate descriptions.

[0106] During the training phase, a combined loss of contrastive learning loss and generation loss is used to train and optimize the decoder. Optimizing by combining the two losses enhances cross-modal alignment and improves generation quality. Contrastive learning loss This is applied at the encoder level to align 3D and textual features in a shared embedding space. This alignment provides a robust initialization for the decoder and guides it to focus on semantically relevant visual regions across attention spans.

[0107] The decoder itself is supervised by the standard cross-entropy loss that describes the generation task. Given a predicted sequence of generated descriptive text... and the corresponding reference description text sequence The generation loss is defined as:

[0108] In summary, the algorithm processing of the 3D scene description model provided in this application embodiment can be described as follows: Input: Point cloud data P, text input T Output: Generate descriptive text

[0109] 1. Point cloud and text input processing: 2. ← Point cloud tokenizer (P) {converts the input point cloud tokens into a sequence} 3. ← Text tokenizer (T) {tokenizes the input text into a sequence} 4. Feature encoding using a frozen CLIP encoder: 5. ← {Frozen CLIP Visual Encoder} 6. ← {Frozen CLIP Text Encoder} 7. Feature alignment and contrastive learning: 8. ← Feature alignment and normalization ,

[0110] 9. ← {Comparative loss between matched and unmatched pairs} 10. Use Update alignment layer 11. Multimodal Decoding and Description Generation: 12. ← {right Perform cross-attention, autoregressive generation. 13. Joint optimization objective: 14. ← {Describe the generation loss} 15. ←

[0111] The overall training objective incorporates two types of loss:

[0112] Here, λ is a weighted hyperparameter that controls the trade-off between alignment and generation. In practice, λ is set to give both approximately equal importance.

[0113] This joint training scheme generates mutually reinforcing learning signals: the alignment process of contrastive learning ensures that the scene features generated by the encoder are readable and informative to the decoder, while descriptive supervision encourages the encoder to capture fine-grained, text-related details. Therefore, the descriptions generated by 3D CoCa learning are not only fluent and descriptive but also accurately based on the 3D scene content, as described in the algorithmic processing above.

[0114] Figure 4 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 4As shown, the electronic device includes a memory 401 and a processor 402. The memory 401 stores a computer program that can run on the processor 402. When the processor 402 executes the computer program, it implements the method described in the above embodiments. The number of memories 401 and processors 402 can be one or more. In a specific implementation, the electronic device may also include a communication interface 403 for communicating with external devices and performing data exchange and transmission.

[0115] In practical implementation, if the memory 401, processor 402, and communication interface 403 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0116] Optionally, in a specific implementation, if the memory 401, processor 402 and communication interface 403 are integrated on a single chip, the memory 401, processor 402 and communication interface 403 can communicate with each other through an internal interface.

[0117] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0118] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0119] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0120] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0121] It should be understood that the aforementioned processor can be a CPU (Central Processing Unit), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0122] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0123] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0124] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0125] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0126] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0127] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0128] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0129] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0130] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a 3D scene description generation model, characterized in that, include: Based on a pre-trained and parameter-frozen 2D image encoder, the scene feature vector of 3D scene point cloud samples is calculated. The reference description text is encoded by a parameter-freezing-based text encoder to obtain a text feature vector; the reference description text is a descriptive text used to describe the three-dimensional scene point cloud sample scene. For multiple 3D scene point cloud samples belonging to the same training iteration, a contrastive learning loss is calculated based on the feature vector similarity between the 3D scene point cloud samples belonging to the same sample pair and the reference description text, and the feature vector similarity between the 3D scene point cloud samples belonging to different sample pairs and the reference description text. The feature vector similarity is the vector similarity between the scene feature vector and the text feature vector. The contrastive learning loss is directly proportional to the feature vector similarity of the same sample pair and inversely proportional to the feature vector similarity of different sample pairs. The scene feature vector is decoded based on a parameter-learnable decoder, and descriptive text is generated as output. Based on the total loss calculated by considering the contrastive learning loss and the generation loss, each learnable parameter is optimized until the training convergence condition is met; the generation loss is determined based on the loss between the generated description text and the reference description text.

2. The method according to claim 1, characterized in that, The pre-trained and parameter-frozen 2D image encoder calculates the scene feature vector of the 3D scene point cloud samples, including: A parameter-learnable point cloud token encoder encodes multiple points extracted from the 3D scene point cloud sample to obtain a scene token vector sequence. The scene token vector sequence and the task vector sequence are concatenated and input into a pre-trained and parameter-frozen two-dimensional image encoder to obtain scene feature vectors; the task vector sequence consists of learnable parameters.

3. The method according to claim 2, characterized in that, The parameter-learnable point cloud token encoder encodes multiple points extracted from the 3D scene point cloud sample to obtain a scene token vector sequence, including: Multiple representative center points are sampled from the three-dimensional scene point cloud sample; For each representative center point, extract multiple neighbor points around the representative center point to construct multiple local point patches that correspond one-to-one with the multiple representative center points; Based on the point-by-point encoder, the local point patches are encoded separately to obtain the scene token vector corresponding to each local point patch, and the scene token vector sequence is obtained by combining them; the point cloud token encoder includes the point-by-point encoder with learnable parameters.

4. The method according to claim 1, characterized in that, The parameter-freezing-based text encoder encodes the reference description text to obtain a text feature vector, including: The reference description text is segmented to obtain a text token sequence; Based on a parameter-learnable embedding table, an embedding vector is assigned to each word in the text token sequence to obtain a text embedding sequence. The text embedding sequence is input into the text encoder to obtain the text feature vector.

5. The method according to claim 1, characterized in that, Before calculating the contrastive learning loss, the method further includes: Based on a scene modal projection head with learnable parameters, the scene feature vector is projected onto a shared embedding space to obtain the scene projection feature vector. Based on a parameter-learnable text modality projector, the text feature vector is projected onto the shared embedding space to obtain the text projection feature vector; The feature vector similarity is used to represent the vector similarity between the scene projection feature vector and the text projection feature vector.

6. The method according to claim 1, characterized in that, The decoder is an autoregressive Transformer decoder. The parameter-learnable decoder decodes the scene feature vector and outputs descriptive text, including: Based on the self-attention layer of the decoder, the currently generated language token sequence is encoded, and the hidden state matrix is ​​updated; Based on the cross-modal attention layer of the decoder, cross-attention computation is performed on the query matrix, key matrix, and value matrix to update the hidden state matrix; the query matrix is ​​the updated hidden state matrix, and the key matrix and the value matrix are generated based on the scene feature vector; The next language token is output based on the updated hidden state matrix, and the output language token is added to the generated language token sequence. The generated description text is output based on the complete sequence of language tokens output in the iterative output.

7. The method according to claim 1, characterized in that, The two-dimensional image encoder is a visual encoder pre-trained by contrastive learning in the CLIP model, and the text encoder is a text encoder pre-trained by contrastive learning in the CLIP model.

8. A method for generating a three-dimensional scene description, characterized in that, include: Based on a pre-trained and parameter-frozen 2D image encoder, the scene feature vector of the 3D scene point cloud is calculated. The scene feature vector is decoded using a decoder, and descriptive text is generated as output. Wherein, the decoder is trained by the method according to any one of claims 1-7, and the pre-trained two-dimensional image encoder is the same as the two-dimensional image encoder used in the method according to any one of claims 1-7.

9. The method according to claim 8, characterized in that, The pre-trained and parameter-frozen 2D image encoder calculates the scene feature vector of the 3D scene point cloud, including: The point cloud token encoder encodes multiple points extracted from the 3D scene point cloud to obtain a scene token vector sequence. The scene token vector sequence and the task vector sequence are concatenated and input into the two-dimensional image encoder to obtain the scene feature vector; The point cloud token encoder and the task vector sequence are trained using the method described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-7 or 8-9.

Images