A method and apparatus for scene representation that enhances three-dimensional spatial understanding of large language models

Abstract

This application provides a scene representation method and apparatus for enhancing the 3D spatial understanding of large language models, relating to the field of data processing technology. Multiple visual references are constructed, and the spatial information of points within the corresponding local regions of each visual reference is learned to obtain a first-level 3D spatial representation. A message passing mechanism facilitates global spatial distribution modeling among different visual references, enabling each visual reference to not only capture its local features but also understand its global spatial relationships with adjacent references, forming a second-level 3D spatial representation. An attention mechanism enables information interaction between visual references and the global scene, and a positional fine-tuning network refines the localization of visual references, resulting in a third-level 3D spatial representation. Thus, by employing a progressive 3D spatial representation from the first to the third level, a spatial representation with enhanced positional information is captured, improving the spatial understanding and reasoning capabilities of large language models in processing 3D visual language tasks.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a scene representation method and apparatus for enhancing the three-dimensional spatial understanding of large language models. Background Technology

[0002] 3D scene understanding is a crucial pathway for robots to perceive the rich semantic and spatial information of the real world. It aims to solve the task of understanding specific objects or spaces in natural language queries by using a specific scene as context. Currently, large language models have made some progress in 3D scene understanding, but they still lack effective perception of 3D spatial relationships and precise location generation. Existing models often rely on overall scene information or specific regions when understanding 3D scenes, limiting the richness of 3D scene representation and hindering the comprehensive perception of 3D space by large language models. Summary of the Invention

[0003] In view of this, this application provides a method and apparatus for enhancing the scene representation of large language models in three-dimensional space, aiming to enhance the scene representation of large language models in three-dimensional space.

[0004] Firstly, this application provides a scene representation method to enhance the three-dimensional spatial understanding of large language models, including:

[0005] The scene point cloud representation is obtained by sampling and position offset to obtain multiple subset points. Based on the multiple subset points, the scene point cloud representation is clustered to construct multiple visual references. The multiple visual references serve as the first-level three-dimensional representation of the corresponding local region in the scene point cloud representation.

[0006] The multiple visual references are modeled in a global spatial distribution through a message propagation mechanism to obtain the second-level three-dimensional representation of each visual reference.

[0007] The second-level 3D representations of the multiple visual references and the scene point cloud representations are processed through a multi-layer attention mechanism and a position fine-tuning network to obtain a third-level 3D representation. The position fine-tuning network is a neural network that adjusts the spatial positioning of each visual reference based on real coordinates.

[0008] Optionally, the step of obtaining multiple subsets of points from the scene point cloud representation through sampling and position offset, and clustering the scene point cloud representation based on the multiple subsets of points to construct multiple visual references includes:

[0009] The scene point cloud representation is sampled using the farthest point sampling algorithm to obtain a first number of sampling points with uniform spatial distribution;

[0010] By aligning each sampling point with the center of the nearest object using a first feedforward neural network, a subset of points corresponding to the sampling points is obtained.

[0011] Based on the first number of subset points, the scene point cloud representation is clustered and pooled to form the first number of visual references.

[0012] Optionally, the step of modeling the global spatial distribution of the multiple visual references through a message propagation mechanism to obtain the second-level three-dimensional representation of each visual reference includes:

[0013] The first number of visual references are input into a graph convolutional network. The feature information of the visual references is used to model graph nodes. The spatial distance between each pair of visual references is used to construct edge information, thereby obtaining the second-level three-dimensional representation of each visual reference.

[0014] Optionally, the step of processing the second-level 3D representation of the plurality of visual references and the scene point cloud representation through a multi-layer attention mechanism and a position fine-tuning network to obtain a third-level 3D representation includes:

[0015] The second-level three-dimensional representation of the visual reference is input into a self-attention network, and the processing result output by the self-attention network and the scene point cloud representation are input into a cross-attention network to update the third-level three-dimensional representation of the visual reference.

[0016] The third-level 3D representation is input into the position fine-tuning network to update the third-level 3D representation.

[0017] Optionally, the method further includes:

[0018] The updated third-level 3D representation is aligned to the language text space using a visual-language bridge to obtain visual cues;

[0019] The visual cues are input into a large language model so that the large language model can understand and process the visual cues.

[0020] Optionally, the method further includes:

[0021] Collect point clouds of the target scene to obtain scene point clouds;

[0022] The scene point cloud is encoded by a point cloud encoder to obtain a scene point cloud representation.

[0023] Optionally, the method further includes:

[0024] Configure a center loss function and a pair space constraint loss function for the position fine-tuning network to achieve feedback adjustment of the neural network corresponding to the third-level three-dimensional representation generated from the second-level three-dimensional representation, the neural network corresponding to the second-level three-dimensional representation generated from multiple visual references, and the neural network corresponding to multiple visual references generated from the scene point cloud representation.

[0025] The location fine-tuning network configuration center loss function and the inter-pair spatial constraint loss function;

[0026] The central loss function:

[0027]

[0028] Where M represents the number of visual reference objects, The coordinates of the predicted visual reference object, The coordinates of the actual visual reference object;

[0029] The space constraint loss function between pairs:

[0030]

[0031] Where N represents the number of visual reference pairs, It is the predicted distance between the visual reference pair (i,j). It is the distance between the actual visual reference pairs (i,j).

[0032] Optionally, the method further includes:

[0033] An overall optimization loss function is set based on the center loss function, the inter-space constraint loss function, and the generation loss function of the large language model. Feedback optimization is applied to the large language model, the vision-language bridge, the neural network generating the third-level 3D representation from the second-level 3D representation, the neural network generating the second-level 3D representation from multiple visual references, and the neural network generating multiple visual references from scene point cloud representations. The formula for the overall optimization loss function is:

[0034] L total =L LLM +α1L center +α2L psc

[0035] Among them, L LLM Let α1 represent the coefficient of the central loss function and α2 represent the coefficient of the inter-space constraint loss function.

[0036] Secondly, this application provides a scene representation device for enhancing the three-dimensional spatial understanding of large language models, the device comprising:

[0037] The clustering abstraction module is used to obtain multiple subset points from the scene point cloud representation through sampling and position offset, and to cluster the scene point cloud representation based on the multiple subset points to construct multiple visual references. The multiple visual references serve as the first-level three-dimensional representation of the corresponding local region in the scene point cloud representation.

[0038] The message passing module is used to perform global spatial distribution modeling of the multiple visual references through a message propagation mechanism to obtain the second-level three-dimensional representation of each visual reference.

[0039] The interaction module is used to process the second-level three-dimensional representations of the multiple visual references and the scene point cloud representations through a multi-layer attention mechanism and a position fine-tuning network to obtain a third-level three-dimensional representation. The position fine-tuning network is a neural network that adjusts the spatial positioning of each visual reference based on real coordinates.

[0040] Optionally, the device further includes:

[0041] A visual-language bridge is used to align the updated third-level 3D representation to the language text space to obtain visual cues;

[0042] A large language model is used to understand and process the visual cues.

[0043] Thirdly, this application provides an apparatus comprising a memory and a processor, the memory for storing instructions or code, and the processor for executing the instructions or code to cause the apparatus to perform a scene representation method for enhancing three-dimensional spatial understanding of large language models as described in any of the first aspects above.

[0044] Fourthly, this application provides a computer storage medium storing code, wherein when the code is executed, a device running the code implements a scene representation method for enhancing three-dimensional spatial understanding of a large language model as described in any of the first aspects above.

[0045] This application provides a scene representation method and apparatus to enhance the 3D spatial understanding of large language models. In executing the method, the scene point cloud representation is first sampled and offset to obtain multiple subsets of points. Based on these subsets, the scene point cloud representation is clustered to construct multiple visual references. Then, these visual references are modeled for global spatial distribution through a message propagation mechanism to obtain a second-level 3D representation for each visual reference. Next, the second-level 3D representations of the multiple visual references and the scene point cloud representation are processed through a multi-layer attention mechanism and a positional fine-tuning network to obtain a third-level 3D representation. Thus, multiple visual references are first randomly constructed, and the spatial information of points within the corresponding local regions of the visual references is learned to obtain a first-level 3D spatial representation. Then, through a message propagation mechanism, global spatial distribution modeling between different visual references is promoted, enabling each visual reference to not only capture its local features but also understand the global spatial relationships with adjacent references, forming a second-level 3D spatial representation. Finally, an attention mechanism is used to achieve information interaction between the visual references and the global scene, and a positional fine-tuning network is added to refine the localization of the visual references, resulting in a third-level 3D spatial representation. Thus, by adopting a progressive three-dimensional spatial representation with enhanced positional information from the first to the third level, spatial representation with enhanced positional information is captured, better spatial positional perception is achieved, and the spatial understanding and reasoning ability of large language models in processing three-dimensional visual language tasks is improved. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating a scene representation method for enhancing three-dimensional spatial understanding of a large language model, provided in an embodiment of this application;

[0048] Figure 2 This is a schematic diagram of the structure of a scene representation device for enhancing the three-dimensional spatial understanding of a large language model, provided in an embodiment of this application. Detailed Implementation

[0049] Currently, 3D vision-language modeling (LLM) technology, which combines 3D scenes with large-scale language models (3D scenes), has explored various spatial learning paradigms for perceiving and understanding the 3D world. The architecture of these technologies typically includes a 3D visual encoder, a vision-language bridge, and a large language model. Some methods are primarily limited to understanding object attributes and lack precise 3D spatial location perception. Other methods first use a scene encoder to extract scene point cloud features; then, they employ a vision-language bridge to extract instruction-specific visual information and align it with text space; finally, they input the information into an LLM to complete the 3D scene understanding task. However, this type of technology is often limited by the vision-language bridge's ability to fit the task, making it unable to handle complex tasks in scenes, such as spatial layout understanding, object distance estimation, and spatial layout editing, and lacking effective perception of 3D spatial relationships and precise location generation.

[0050] To address the aforementioned issues, this application provides a scene representation method and apparatus for enhancing the 3D spatial understanding of large language models. The method first randomly constructs multiple visual references. By learning the spatial information of points within the local regions corresponding to the visual references, a first-level 3D spatial representation is obtained. Next, a message passing mechanism facilitates global spatial distribution modeling among different visual references, enabling each visual reference to not only capture its local features but also understand its global spatial relationships with adjacent references, forming a second-level 3D spatial representation. Finally, an attention mechanism enables information interaction between visual references and the global scene, and a positional fine-tuning network refines the localization of the visual references, resulting in a third-level 3D spatial representation. Thus, by employing a progressive position-enhanced 3D spatial representation from the first to the third level, the method effectively captures and enhances spatial representations with improved positional information, better achieving spatial position perception and improving the spatial understanding and reasoning capabilities of large language models in processing 3D visual language tasks.

[0051] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0052] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0053] Unless otherwise stated, the term "multiple" means two or more.

[0054] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0055] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0057] See Figure 1 , Figure 1 A flowchart illustrating a scene representation method for enhancing three-dimensional spatial understanding of a large language model, provided in this application embodiment, includes:

[0058] S101. The scene point cloud representation is sampled and offset to obtain multiple subset points. Based on the multiple subset points, the scene point cloud representation is clustered to construct multiple visual references. The multiple visual references serve as the first-level three-dimensional representation of the corresponding local region in the scene point cloud representation.

[0059] Scene point cloud representation is obtained by encoding the point cloud of the target scene using a point cloud encoder.

[0060] A point cloud is a collection of discrete points in three-dimensional space. Each point carries three-dimensional coordinate information, which may also include other attributes such as color and reflectivity.

[0061] Scene point cloud representation is the process of extracting meaningful features from point clouds and converting them into a form that computers can understand and process, so that they can be used for subsequent tasks such as scene understanding, object recognition, and 3D reconstruction. Scene point cloud representation can include the position of points in the point cloud and the distance between points, shape and contour (which can be represented by normal vectors, curvature, etc.), spatial distribution relationship (describing the relative positional relationship between objects in the scene), attribute information (color, reflectivity), etc.

[0062] The above method obtains multiple sub-points evenly distributed in space through sampling. These sub-points are then aligned with the center of the nearest object through positional offset, resulting in a subset of points representing the object or a part of it. Based on these subsets, the scene point cloud representation is clustered to construct multiple visual references. Clustering divides the scene point cloud representation into regions corresponding to these subsets, resulting in multiple sub-regions. Finally, pooling is used to compress the point sets of each sub-region, forming the visual references.

[0063] This allows us to obtain spatial information about points within a local area, thus obtaining a first-level three-dimensional spatial representation.

[0064] S102. The multiple visual references are modeled in a global spatial distribution through a message propagation mechanism to obtain the second-level three-dimensional representation of each visual reference.

[0065] Optionally, global spatial distribution modeling can model graph nodes using the feature information of multiple visual references, construct edge information through the spatial distance between each pair of visual references, and promote global spatial distribution modeling between different visual references through a message passing mechanism.

[0066] In this way, we can focus on the implicit relationships between visual references, capture both the local features of visual references and the global spatial relationship semantics of adjacent visual references. Through a message propagation mechanism, we learn the enhanced representation of the spatial relationships of each visual reference, thus obtaining a second-level three-dimensional spatial representation.

[0067] S103. The second-level three-dimensional representations of the multiple visual references and the scene point cloud representations are processed through a multi-layer attention mechanism and a position fine-tuning network to obtain a third-level three-dimensional representation. The position fine-tuning network is a neural network that adjusts the spatial positioning of each visual reference based on real coordinates.

[0068] Optionally, self-attention can be used to process the second-level three-dimensional representation of the visual reference, and then combined with the scene point cloud representation to realize information interaction between the visual reference and the global scene through cross-attention to obtain the third-level three-dimensional representation. The third-level three-dimensional representation is then updated through a position fine-tuning network.

[0069] Based on steps S101-S103 above, multiple visual references are first randomly constructed. By learning the spatial information of points within the local regions corresponding to the visual references, a first-level 3D spatial representation is obtained. Next, a message passing mechanism facilitates global spatial distribution modeling among different visual references, enabling each visual reference to not only capture its local features but also understand its global spatial relationships with adjacent references, forming a second-level 3D spatial representation. Finally, an attention mechanism enables information interaction between the visual references and the global scene, and a positional fine-tuning network refines the localization of the visual references, resulting in a third-level 3D spatial representation. Thus, this progressive positional enhancement 3D spatial representation, from the first to the third level, effectively captures and enhances spatial representations of positional information, better achieving spatial position perception and improving the spatial understanding and reasoning capabilities of large language models in processing 3D visual language tasks.

[0070] Optionally, before step S101 above, a scene point cloud representation can also be obtained. The specific steps are as follows:

[0071] First, the point cloud of the target scene is collected to obtain the scene point cloud.

[0072] For example, a large collection of points on the surface of an object in three-dimensional space can be obtained through devices such as scanners and cameras to form a scene point cloud.

[0073] Then, the scene point cloud is encoded by a point cloud encoder to obtain a scene point cloud representation.

[0074] A point cloud encoder primarily compresses, extracts, or transforms raw scene point cloud data to form a scene point cloud representation.

[0075] Optionally, after step S103 above, a third-level three-dimensional representation based on enhanced large language model three-dimensional spatial understanding can be used to solve the understanding task of a specified object or space in natural language queries, such as ScanQA, ScanRefer, and Scan2Cap. This may include:

[0076] First, the third-level three-dimensional representation output in step S103 is aligned to the language text space to obtain visual cues through a visual-language bridge.

[0077] The visual-language bridge aligns the enhanced scene representation to the third-level 3D representation in the language text space as a visual cue, enabling the subsequent large language model to process 3D spatial information and natural language input simultaneously, enhancing multimodal understanding capabilities and facilitating accurate scene analysis.

[0078] Then, the visual cues are input into a large language model so that the large language model can understand and process the visual cues to obtain results for the understanding task of a specified object or space in a natural language query.

[0079] In the embodiments of this application, the above Figure 1 The aforementioned step S101 can be implemented in various ways, which will be described in detail below. It should be noted that the implementation methods given in the following description are merely illustrative examples and do not represent all implementation methods of the embodiments of this application.

[0080] In step S101, the scene point cloud representation is sampled and offset to obtain multiple subsets of points. Based on these subsets of points, the scene point cloud representation is clustered to construct multiple visual references. Specific steps may include:

[0081] First, the scene point cloud representation is sampled using the farthest point sampling algorithm to obtain a first number of sampling points with uniform spatial distribution.

[0082] The aforementioned farthest point sampling algorithm can obtain a first number of spatially evenly distributed sampling points from the scene point cloud representation. In this way, representative sampling points are selected from a given set of point clouds (scene point cloud representation), and these sampling points can reflect the overall distribution characteristics of the point cloud to a certain extent.

[0083] Then, each sampling point is aligned with the center of the nearest object using a first feedforward neural network to obtain the subset points corresponding to the sampling points.

[0084] The offset between the current sampling point and the center of the nearest object is obtained through the first feedforward neural network, thereby aligning each sampling point with the center of the nearest object.

[0085] Finally, the scene point cloud representation is clustered and pooled based on the first number of subset points to form a first number of visual references.

[0086] Optionally, the above clustering can employ the K-Nearest Neighbors algorithm, an algorithm used for classification and regression tasks. KNN selects multiple points in the scene point cloud representation that are closest to each subset of points to form a sub-region point set.

[0087] Then, pooling is performed on each sub-region point set to compress the sub-region point set, and the information of multiple points included in the sub-region point set is aggregated to obtain the point set features with rich local regional information corresponding to the sub-region point set, forming the first-level three-dimensional representation of the corresponding local region in the scene point cloud representation, that is, forming the visual reference of the sub-region point set.

[0088] In the embodiments of this application, the above Figure 1The aforementioned step S102 can be implemented in various ways, which will be described in detail below. It should be noted that the implementation methods given in the following description are merely illustrative examples and do not represent all implementation methods of the embodiments of this application.

[0089] In step S102, the multiple visual references are modeled in a global spatial distribution using a message propagation mechanism to obtain the second-level three-dimensional representation of each visual reference. Specific steps include:

[0090] The first number of visual references are input into a graph convolutional network. The feature information of the visual references is used to model graph nodes. The spatial distance between each pair of visual references is used to construct edge information, thereby obtaining the second-level three-dimensional representation of each visual reference.

[0091] The graph convolutional network described above is a deep learning model specifically designed for processing graph-structured data. A graph is a data structure where nodes represent entities and edges represent relationships between entities; it allows modeling of a set of nodes and their relationships (edges).

[0092] By effectively extracting features from graph structure data through graph convolutional networks, this application models graph nodes using feature information from multiple visual references and constructs edge information based on the spatial distance between each pair of visual references to achieve global spatial distribution modeling. In this way, it captures the local features of visual references and the global spatial relationship features of adjacent visual references, learns the spatial relationship enhancement representation of each visual reference, and obtains the second-level three-dimensional representation.

[0093] In the embodiments of this application, the above Figure 1 The aforementioned step S103 can be implemented in various ways, which will be described in detail below. It should be noted that the implementation methods given in the following description are merely illustrative examples and do not represent all implementation methods of the embodiments of this application.

[0094] In step S103, the second-level 3D representation of the multiple visual references and the scene point cloud representation are processed through a multi-layer attention mechanism and a position fine-tuning network to obtain the third-level 3D representation. Specific steps include:

[0095] First, the second-level 3D representation of the visual reference is input into a self-attention network. The processing result output by the self-attention network and the scene point cloud representation are input into a cross-attention network to update the third-level 3D representation of the visual reference.

[0096] Self-attention networks are better able to capture long-range dependencies and important information in the second-level 3D representations of various visual references. Cross-attention networks, on the other hand, can fuse the processing results from the self-attention networks with the correlation information between the scene point cloud representations, enabling information interaction between each visual reference and the global scene. In this way, the inherent features of visual references are effectively combined with information from the global scene, giving the visual references the attribute of being perceived in a comprehensive spatial environment.

[0097] The third-level 3D representation is input into the position fine-tuning network to update the third-level 3D representation.

[0098] For example, the location fine-tuning network can employ an FFN containing multiple layers.

[0099] The spatial positioning of the visual reference is adjusted by minimizing the relative distance between the visual reference and the real coordinates, so that the visual reference is closer to the coordinate center of the object.

[0100] In one specific implementation, to achieve steps S101-S103 above, a spatial awareness model can be set up and trained for the neural network corresponding to steps S101-S103. Specifically:

[0101] The spatial awareness model may include a clustering abstraction module, a message passing module, and an interaction module;

[0102] The clustering abstraction module is used to execute step S101 above. Specifically, this module may include a farthest point sampling algorithm, a feedforward neural network layer, a clustering layer (which may use the K-nearest neighbor algorithm), and a pooling layer. The message passing module is used to execute step S102 above. Specifically, this module may include a graph convolutional network. The interaction module is used to execute step S103 above. Specifically, this message passing module is configured with a self-attention layer, a cross-attention layer, and a positional fine-tuning network layer.

[0103] In training this spatial perception model, a multi-layer feedforward neural network can be configured in the position fine-tuning network layer, along with a center loss function and an inter-space constraint loss function. The spatial perception model is trained on a training set containing scene point cloud representations (which can be a training set containing scene point clouds encoded by a point cloud encoder). The loss function penalizes the Euclidean distance between the predicted coordinates and the true coordinates of visual references obtained through the spatial perception model, encouraging the model to predict more accurate visual reference positions.

[0104] The central loss function:

[0105]

[0106] Where M represents the number of visual reference objects, The coordinates of the predicted visual reference object, The coordinates of the actual visual reference object;

[0107] The space constraint loss function between pairs:

[0108]

[0109] Where N represents the number of visual reference pairs, It is the predicted distance between the visual reference pair (i,j). It is the distance between the actual visual reference pairs (i,j).

[0110] Furthermore, in order to better utilize the third-dimensional spatial features in large language models, the aforementioned spatial perception model can be further trained by combining a vision-language bridge with a large language model. The specific steps are as follows:

[0111] First, a site cloud representation from the training set is input into the aforementioned spatial perception model to obtain the third-level 3D representation output by the spatial perception model. The spatial perception model is then trained based on the center loss function and the inter-pair spatial constraint loss function.

[0112] The training set can be composed of various visual-language understanding and visual-language localization tasks. The training set also includes point cloud scene representations or point cloud scenes corresponding to various visual-language understanding and visual-language localization tasks. If it is a point cloud scene, it needs to be encoded by a point cloud encoder to obtain the corresponding point cloud scene representation.

[0113] Secondly, the third-level three-dimensional representation is input into the visual-language bridge, and the third-level three-dimensional representation is aligned to the language text space to obtain the visual cues output by the visual-language bridge.

[0114] Then, the visual cue is input into the large language model, enabling the model to understand and process it to obtain results for various visual-language understanding and visual-language localization tasks corresponding to the point cloud scene representation. The spatial perception model, visual-language bridge, and large language model are then optimized by comprehensively optimizing the loss function.

[0115] The overall optimization loss function is: L total =L LLM +α1L center +α2L psc

[0116] Among them, L LLMLet α1 represent the coefficient of the central loss function and α2 represent the coefficient of the inter-space constraint loss function.

[0117] The above are some specific implementations of a scene representation method for enhancing the three-dimensional spatial understanding of large language models provided in this application. Based on this, this application also provides a corresponding device. The device provided in this application will be described below from the perspective of functional modularity.

[0118] See Figure 2 The diagram shows a scene representation device for enhancing three-dimensional spatial understanding of large language models. The scene representation device includes:

[0119] The clustering abstraction module 201 is used to obtain multiple subset points from the scene point cloud representation through sampling and position offset, and to cluster the scene point cloud representation based on the multiple subset points to construct multiple visual references. The multiple visual references serve as the first-level three-dimensional representation of the corresponding local region in the scene point cloud representation.

[0120] The message passing module 202 is used to perform global spatial distribution modeling of the multiple visual references through a message propagation mechanism to obtain the second-level three-dimensional representation of each visual reference.

[0121] The interaction module 203 is used to process the second-level three-dimensional representation of the multiple visual references and the scene point cloud representation through a multi-layer attention mechanism and a position fine-tuning network to obtain a third-level three-dimensional representation. The position fine-tuning network is a neural network that adjusts the spatial positioning of each visual reference based on real coordinates.

[0122] This application, through the aforementioned apparatus, randomly constructs multiple visual references via a clustering abstraction module 201. By learning the spatial information of points within the local regions corresponding to the visual references, a first-level 3D spatial representation is obtained. Next, a message passing module 202 facilitates global spatial distribution modeling among different visual references, enabling each visual reference to not only capture its local features but also understand its global spatial relationships with adjacent references, forming a second-level 3D spatial representation. Finally, an interaction module 203 implements information interaction between the visual references and the global scene through a multi-layer attention mechanism and adds a position fine-tuning network to refine the localization of the visual references, resulting in a third-level 3D spatial representation. Thus, by employing a progressive position-enhanced 3D spatial representation from the first to the third level, spatial information containing positional information is effectively captured, better spatial position perception is achieved, and the spatial understanding and reasoning capabilities of large language models in processing 3D visual language tasks are improved.

[0123] Optionally, the device further includes: a data acquisition module for acquiring point clouds of the target scene to obtain scene point clouds.

[0124] Optionally, the device further includes: a point cloud encoder for encoding the scene point cloud to obtain a scene point cloud representation.

[0125] Optionally, the device further includes:

[0126] A visual-language bridge is used to align the updated third-level 3D representation to the language text space to obtain visual cues;

[0127] A large language model is used to understand and process the visual cues.

[0128] In one possible implementation, the clustering abstraction module 201 is specifically used to sample the scene point cloud representation using the farthest point sampling algorithm to obtain a first number of sampling points with uniform spatial distribution; to align each sampling point with the center of the nearest object using a first feedforward neural network to obtain a subset of points corresponding to the sampling points; and to cluster and pool the scene point cloud representation based on the first number of subset points to form a first number of visual references.

[0129] In one possible implementation, the message passing module 202 is specifically used to input the first number of visual references into a graph convolutional network, model graph nodes using the feature information of the visual references, construct edge information using the spatial distance between each pair of visual references, and obtain a second-level three-dimensional representation of each visual reference.

[0130] In one possible implementation, the interaction module 203 is specifically used to input the second-level three-dimensional representation of the visual reference into a self-attention network, input the processing result output by the self-attention network and the scene point cloud representation into a cross-attention network to update the third-level three-dimensional representation of the visual reference; and input the third-level three-dimensional representation into a position fine-tuning network to update the third-level three-dimensional representation.

[0131] In one possible implementation, the aforementioned clustering abstraction module, message passing module, and interaction module form a spatial perception model. Point cloud scene representations are input into the spatial perception model to obtain scene representations that enhance the 3D spatial understanding of large language models. The training method for this spatial perception model can be:

[0132] This spatial perception model may include;

[0133] The training is based on a training set containing scene point cloud representations (which can be a training set containing scene point clouds encoded by a point cloud encoder). A loss function is used to penalize the Euclidean distance between the predicted coordinates and the true coordinates of the visual reference obtained by the spatial perception model, thereby encouraging the spatial perception model to predict the position of the visual reference more accurately. The loss function is configured with an interactive module for outputting the third-level 3D representation. The loss function includes a center loss function and an inter-space constraint loss function.

[0134] The central loss function can be:

[0135]

[0136] Where M represents the number of visual reference objects, The coordinates of the predicted visual reference object, The coordinates of the actual visual reference object;

[0137] The space constraint loss function can be:

[0138]

[0139] Where N represents the number of visual reference pairs, It is the predicted distance between the visual reference pair (i,j). It is the distance between the actual visual reference pairs (i,j).

[0140] Furthermore, in order to better apply the third-dimensional spatial features to large language models, the aforementioned spatial perception model can be trained in conjunction with a vision-language bridge and a large language model. The specific steps are as follows:

[0141] First, a site cloud representation from the training set is input into the aforementioned spatial perception model to obtain the third-level three-dimensional representation output by the spatial perception model. The spatial perception model is then trained based on the center loss function and the inter-pair spatial constraint loss function.

[0142] The training set can be composed of various visual-language understanding and visual-language localization tasks. The training set also includes point cloud scene representations or point cloud scenes corresponding to various visual-language understanding and visual-language localization tasks. If it is a point cloud scene, it needs to be encoded by a point cloud encoder to obtain the corresponding point cloud scene representation.

[0143] Secondly, the third-level three-dimensional representation is input into the visual-language bridge, and the third-level three-dimensional representation is aligned to the language text space to obtain the visual cues output by the visual-language bridge.

[0144] Then, the visual cue is input into the large language model, enabling the model to understand and process it to obtain results for various visual-language understanding and visual-language localization tasks corresponding to the point cloud scene representation. The spatial perception model, visual-language bridge, and large language model are then optimized by comprehensively optimizing the loss function.

[0145] The overall optimization loss function is: L total =L LLM +α1L center +α2L psc

[0146] Among them, L LLM Let α1 represent the coefficient of the central loss function and α2 represent the coefficient of the inter-space constraint loss function.

[0147] This application also provides corresponding devices and computer storage media for implementing the solutions provided in this application.

[0148] The device includes a memory and a processor. The memory stores instructions or code, and the processor executes the instructions or code to enable the device to perform a scene representation method for enhancing three-dimensional spatial understanding of a large language model as described in any embodiment of this application.

[0149] The computer storage medium stores code, and when the code is executed, the device running the code implements a scene representation method for enhancing three-dimensional spatial understanding of a large language model as described in any embodiment of this application.

[0150] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0151] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0152] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0153] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.

Claims

1. A scene representation method to enhance the three-dimensional spatial understanding of large language models, characterized in that, include: The scene point cloud representation is obtained by sampling and position offset to obtain multiple subset points. Based on the multiple subset points, the scene point cloud representation is clustered to construct multiple visual references. The multiple visual references serve as the first-level three-dimensional representation of the corresponding local region in the scene point cloud representation. The multiple visual references are modeled in a global spatial distribution through a message propagation mechanism to obtain the second-level three-dimensional representation of each visual reference. The second-level three-dimensional representations of the multiple visual references and the scene point cloud representations are processed through a multi-layer attention mechanism and a position fine-tuning network to obtain a third-level three-dimensional representation. The position fine-tuning network is a neural network that adjusts the spatial positioning of each visual reference based on real coordinates. The process involves obtaining multiple subsets of points from the scene point cloud representation through sampling and position offset, and then clustering the scene point cloud representation based on these subsets to construct multiple visual references, including: The scene point cloud representation is sampled using the farthest point sampling algorithm to obtain a first number of sampling points with uniform spatial distribution; The sampling points are aligned with the center of the nearest object by a first feedforward neural network to obtain a subset of points corresponding to the sampling points; the scene point cloud representation is clustered and pooled based on the first number of subset points to form a first number of visual references.

2. The method according to claim 1, characterized in that, The step of modeling the global spatial distribution of the multiple visual references through a message propagation mechanism to obtain the second-level three-dimensional representation of each visual reference includes: The first number of visual references are input into a graph convolutional network. The feature information of the visual references is used to model graph nodes. The spatial distance between each pair of visual references is used to construct edge information, thereby obtaining the second-level three-dimensional representation of each visual reference.

3. The method according to claim 2, characterized in that, The process of combining the second-level 3D representation of the multiple visual references with the scene point cloud representation through a multi-layer attention mechanism and a position fine-tuning network to obtain the third-level 3D representation includes: The second-level three-dimensional representation of the visual reference is input into a self-attention network, and the processing result output by the self-attention network and the scene point cloud representation are input into a cross-attention network to update the third-level three-dimensional representation of the visual reference. The third-level 3D representation is input into the position fine-tuning network to update the third-level 3D representation.

4. The method according to claim 3, characterized in that, The method further includes: The updated third-level 3D representation is aligned to the language text space using a visual-language bridge to obtain visual cues; The visual cues are input into a large language model so that the large language model can understand and process the visual cues.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Collect point clouds of the target scene to obtain scene point clouds; The scene point cloud is encoded by a point cloud encoder to obtain a scene point cloud representation.

6. The method according to claim 4, characterized in that, The method further includes: Configure a center loss function and a pair space constraint loss function for the position fine-tuning network to achieve feedback adjustment of the neural network corresponding to the third-level three-dimensional representation generated from the second-level three-dimensional representation, the neural network corresponding to the second-level three-dimensional representation generated from multiple visual references, and the neural network corresponding to multiple visual references generated from the scene point cloud representation. The location fine-tuning network configuration center loss function and the inter-pair spatial constraint loss function; The central loss function: ， Where M represents the number of visual reference objects, The coordinates of the predicted visual reference object, The coordinates of the actual visual reference object; The space constraint loss function between pairs: ， Where N represents the number of visual reference pairs, It is the predicted distance between the visual reference pair (i, j). It is the distance between the actual visual reference pair (i, j).

7. The method according to claim 6, characterized in that, The method further includes: An overall optimization loss function is set based on the center loss function, the inter-space constraint loss function, and the generation loss function of the large language model. Feedback optimization is applied to the large language model, the visual-language bridge, the neural network generating the third-level 3D representation from the second-level 3D representation, the neural network generating the second-level 3D representation from multiple visual references, and the neural network generating multiple visual references from scene point cloud representations. The formula for the overall optimization loss function is: ， in, This represents the generation loss function of a large language model. The coefficients represent the central loss function. The coefficients of the space constraint loss function are represented. Represents the central loss function. This represents the space constraint loss function between pairs.

8. A scene representation device for enhancing three-dimensional spatial understanding of large language models, characterized in that, The device includes: The clustering abstraction module is used to obtain multiple subset points from the scene point cloud representation through sampling and position offset, and to cluster the scene point cloud representation based on the multiple subset points to construct multiple visual references. The multiple visual references serve as the first-level three-dimensional representation of the corresponding local region in the scene point cloud representation. The message passing module is used to perform global spatial distribution modeling of the multiple visual references through a message propagation mechanism to obtain the second-level three-dimensional representation of each visual reference. An interaction module is used to process the second-level three-dimensional representations of the multiple visual references and the scene point cloud representations through a multi-layer attention mechanism and a position fine-tuning network to obtain a third-level three-dimensional representation. The position fine-tuning network is a neural network that adjusts the spatial positioning of each visual reference based on real coordinates. The clustering abstraction module is specifically used to sample the scene point cloud representation using the farthest point sampling algorithm to obtain a first number of sampling points with uniform spatial distribution; to align each sampling point with the center of the nearest object using a first feedforward neural network to obtain a subset of points corresponding to the sampling points; and to cluster and pool the scene point cloud representation based on the first number of subset points to form a first number of visual references.

9. The apparatus according to claim 8, characterized in that, The device further includes: A visual-language bridge is used to align the updated third-level 3D representation to the language text space to obtain visual cues; A large language model is used to understand and process the visual cues.

Images