Three-dimensional scene segmentation method and device, equipment and storage medium

Abstract

The invention provides a three-dimensional scene segmentation method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence and computer vision technologies. The method comprises the following steps: inputting an RGB rendered image into a coder-decoder module, and processing the input RGB rendered image through a two-dimensional visual language model and a coder to obtain Gaussian semantic features; fusing the Gaussian semantic features with other Gaussian attribute features of the three-dimensional Gaussian rendering model to obtain all Gaussian attribute information, inputting the Gaussian attribute information and text information into a large language model, and outputting segmentation feature vectors; inputting the Gaussian semantic feature and the segmentation feature vector into a segmentation information output module to generate a semantic segmentation mask; and based on the semantic segmentation mask, outputting a segmentation result of the target object in the three-dimensional scene. According to the invention, automatic identification and region segmentation of the target object can be realized in a complex three-dimensional environment, so that the intelligent level of a three-dimensional multi-mode interaction and application system is improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and computer vision technology, and in particular to a three-dimensional scene segmentation method, apparatus, device and storage medium. Background Technology

[0002] 3D scene understanding has important applications in virtual reality, augmented reality, and robotics, with object segmentation being a core task. Traditional point cloud or voxel methods are computationally intensive and have limited accuracy, while multi-view methods are susceptible to occlusion and missing viewpoints. 3D Gaussian Rendering (3DGS) can efficiently represent dense 3D scenes and achieve differentiable rendering, but it lacks semantic guidance and struggles to distinguish different objects in complex scenes. Large Language Models (LLMs), such as ChatGPT, have demonstrated powerful capabilities in natural language understanding, converting text instructions into semantic information to guide 3D segmentation. How to fuse LLM semantics with 3DGS geometric information to achieve cross-modal alignment and accurate segmentation is a pressing technical problem that needs to be solved.

[0003] Existing segmentation methods do not fully utilize large language models for text reasoning, and their fusion of linguistic information remains insufficient. They lack effective modeling and utilization of natural language descriptions, making it difficult to achieve high accuracy and good scalability in complex text scenarios. This results in poor performance in human-computer interaction and complex text-driven scene understanding tasks, failing to meet practical application needs. Furthermore, the text scenarios that large language models can understand are more complex and have higher information dimensions than 2D language models, making it difficult for 3D models to directly understand them across modalities. Therefore, it is impossible to directly transfer large language models to 3D models to achieve cross-modal information alignment. Summary of the Invention

[0004] This application provides a three-dimensional scene segmentation method, apparatus, device, and storage medium. By fusing semantic information generated by language with three-dimensional geometric representation, it achieves accurate positioning and segmentation of target objects in complex scenes, while improving segmentation efficiency and overcoming the shortcomings of existing methods such as large computational load, insufficient semantic understanding, and strong viewpoint dependence. It has great practical application significance.

[0005] Firstly, this application provides a three-dimensional scene segmentation method, including: An RGB rendered image is obtained, which is generated by a pre-trained 3D Gaussian rendering model. The RGB rendered image is input into an encoder-decoder module, which includes a 2D visual language model and an encoder. The 2D visual language model and the encoder process the input RGB rendered image to obtain Gaussian semantic features. The Gaussian semantic features are fused with other Gaussian attribute features of the 3D Gaussian rendering model to obtain all Gaussian attribute information. The all Gaussian attribute information and text information are then input into a large language model, which outputs a segmentation feature vector. The text information is the natural language command input by the user. The Gaussian semantic features and the segmentation feature vector are input into the segmentation information output module. The segmentation information output module performs cross-modal fusion on the Gaussian semantic features and the segmentation feature vector to generate a semantic segmentation mask. Based on the semantic segmentation mask, the segmentation result of the target object in the 3D scene is output.

[0006] In one possible design, the process of processing the input RGB rendered image using the two-dimensional visual language model and encoder to obtain Gaussian semantic features is represented as follows: In the formula, For high-dimensional semantic information, For a two-dimensional visual language model, Rendering images for RGB Gaussian semantic features for Encoder.

[0007] In one possible design, the method further includes training the encoder-decoder module based on reconstruction loss and language alignment loss, and obtaining Gaussian semantic features based on the trained encoder-decoder module; wherein the reconstruction loss and language alignment loss are expressed as: In the formula, To rebuild the losses, Mean square error, The features after encoder compression, For language alignment loss, This represents the semantic alignment loss function between linguistic features, used to compute Gaussian semantic features. With high-dimensional semantic information The loss error between them.

[0008] In one possible design, the large language model outputs segmentation feature vectors in the following manner: In the formula, For feature embedding containing segmentation information, For text information, Gaussian semantic features Other Gaussian property features of the 3D Gaussian rendering model, This contains all the attribute information for Gauss. For large language models, To segment feature vectors, This represents the last dimension of the output result from the large language model.

[0009] In one possible design, the method by which the segmentation information output module performs cross-modal fusion of the Gaussian semantic features and the segmentation feature vector is as follows: In the formula, Gaussian semantic features for Decoder For pre-trained The high-dimensional semantic information restored by the decoder This is a mapping layer used to map high-dimensional semantic information. Mapping to a unified dimension to align features. The semantic information after dimensional alignment by the mapping layer is used as The input at one end of the decoder, For semantic segmentation mask, for Decoder To segment feature vectors.

[0010] In one possible design, the method further includes training the segmentation information output module based on a predetermined loss function, and generating a semantic segmentation mask based on the trained segmentation information output module; wherein, the predetermined loss function is expressed as: In the formula, Total loss; The text semantic constraint loss is used to measure the degree of matching between the semantic features generated by the large language model and the text description; The segmentation mask supervision loss is used to measure the deviation between the segmentation result predicted by the model and the true mask of the target region. , which is the weighting coefficient for text semantic loss, used to adjust the contribution ratio of language alignment to the overall optimization; These are the weighting coefficients for the mask loss, used to balance the impact of segmentation accuracy on the total loss.

[0011] In one possible design, it is determined by the following formula. and : In the formula, Cross-entropy loss is used to measure the difference between the features generated by LLM and the real text labels; Embedding features containing segmentation information For target text labels; The weights for the Binary Cross-Entropy (BCE) loss are used to control the strength of the influence of the binary cross-entropy loss (BCE) on the segmentation loss. Binary cross-entropy loss; The Dice loss weights are used to control the proportion of the Dice loss term. For Dice coefficient loss; For the final The decoder's output mask. This is the true mask used to represent the standard answer for the segmentation task.

[0012] Secondly, this application provides a three-dimensional scene segmentation device, the device comprising: The first feature extraction module is configured to acquire an RGB rendered image, which is generated by a pre-trained three-dimensional Gaussian rendering model. The RGB rendered image is input to an encoder-decoder module, which includes a two-dimensional visual language model and an encoder. The two-dimensional visual language model and the encoder process the input RGB rendered image to obtain Gaussian semantic features. The second feature extraction module is configured to fuse the Gaussian semantic features with other Gaussian attribute features of the 3D Gaussian rendering model to obtain all Gaussian attribute information, and input the all Gaussian attribute information and text information into a large language model, and output a segmentation feature vector through the large language model; wherein, the text information is a natural language command input by the user; The segmentation information determination module is configured to input the Gaussian semantic features and the segmentation feature vectors into the segmentation information output module, and perform cross-modal fusion of the Gaussian semantic features and the segmentation feature vectors through the segmentation information output module to generate a semantic segmentation mask. The target segmentation module is configured to output the segmentation result of the target object in the 3D scene based on the semantic segmentation mask.

[0013] Thirdly, embodiments of this application provide an electronic device, including: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the three-dimensional scene segmentation method as described in the first aspect and various possible designs of the first aspect.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the three-dimensional scene segmentation method described in the first aspect and various possible designs of the first aspect.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the three-dimensional scene segmentation method described in the first aspect and various possible designs of the first aspect.

[0016] The three-dimensional scene segmentation method, apparatus, device, and storage medium provided in this application have at least the following beneficial effects: This application introduces a multimodal large language model, deeply integrating three-dimensional geometric features with language semantic reasoning capabilities, enabling the system to understand and execute complex natural language instructions. Supported by a cross-modal fusion mechanism, the model not only possesses stronger semantic understanding and spatial reasoning capabilities but also exhibits higher robustness and scalability in open semantic environments, further enhancing the system's human-computer interaction capabilities and practical application value.

[0017] The encoder-decoder module proposed in this application effectively solves the mismatch between semantic feature dimensions and Gaussian representation. On the one hand, it reduces the computational burden of high-dimensional semantic features and improves rendering and inference efficiency; on the other hand, it ensures that low-dimensional Gaussian features still have sufficient semantic expressive power, thus achieving a balance between computational efficiency and semantic carrying capacity. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] Figure 1 A flowchart illustrating a three-dimensional scene segmentation method provided in this application embodiment; Figure 2 A training flowchart for a three-dimensional scene segmentation method provided in an embodiment of this application; Figure 3 This is a structural diagram of a three-dimensional scene segmentation device provided in an embodiment of this application.

[0020] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0022] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0023] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0024] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0025] This application provides a three-dimensional scene segmentation method, such as... Figure 1 As shown, the 3D scene segmentation method includes the following steps S10-S40.

[0026] S10: Obtain the RGB rendered image. The RGB rendered image is generated by a pre-trained 3D Gaussian rendering model. The RGB rendered image is input into the encoder-decoder module, which includes a 2D visual language model and an encoder. The 2D visual language model and the encoder process the input RGB rendered image to obtain Gaussian semantic features.

[0027] In the encoder-decoder module, a two-dimensional knowledge transfer mechanism is introduced. The RGB rendered image generated by the pre-trained 3D Gaussian rendering (3DGS) model is used as input, and the two-dimensional visual language model extracts semantic information, thus fully utilizing existing two-dimensional semantic understanding capabilities. Since the semantic features obtained from the two-dimensional visual language model have high dimensionality and complex distribution, directly fusing them with the features of the large language model would cause dimensionality mismatch and information redundancy. Therefore, this embodiment designs a cross-modal feature alignment structure based on the encoder-decoder to perform dimensionality adjustment on the extracted semantic features, ensuring consistency in representation space with the semantic features output by the large language model. Specifically, this embodiment uses CLIP as the two-dimensional visual language model. Leveraging its powerful cross-modal representation capabilities, it achieves accurate semantic parsing of the rendered image and efficiently aligns and fuses it with the semantic information of the large language model, thereby providing robust semantic support for target segmentation in 3D scenes.

[0028] In some embodiments, let This represents an RGB rendered image generated by a pre-trained 3D Gaussian rendering (3DGS) model. To align with the semantic features of a Large Language Model (LLM), this method employs CLIP as a 2D visual language model to extract language-related semantic features from the rendered image. After passing through a mapping layer, high-dimensional semantic information is obtained. Subsequently, high-dimensional semantic information It is fed into the encoder and performs the following operations: In the formula, For high-dimensional semantic information, For a two-dimensional visual language model, Rendering images for RGB Gaussian semantic features for Encoder.

[0029] The above process compresses high-dimensional semantic information into a low-dimensional latent space, which serves as Gaussian semantic features. The method employs a supervisory signal. Furthermore, to ensure that semantic information is not lost during the mapping process, a decoder is designed to reconstruct the feature. In this embodiment, the decoder remaps the low-dimensional feature back to the original CLIP feature space, thereby forcing the Gaussian feature to maintain consistency with the original semantics during reconstruction, ensuring that the Gaussian point cloud possesses an interpretable semantic representation.

[0030] In some embodiments, the encoder-decoder module is trained based on reconstruction loss and language alignment loss, and Gaussian semantic features are obtained based on the trained encoder-decoder module. Specifically, this embodiment uses feature reconstruction loss. To train and update Encoder, making The encoder compresses features that closely resemble the original spatial features and also employs language alignment loss to allow Gaussian to learn Gaussian semantic features. The two-dimensional visual language model is embedded and aligned after compression by the L encoder.

[0031] Feature Reconstruction Loss Represented as: In the formula, Mean square error, These are the features after compression by the encoder.

[0032] This embodiment optimizes using the mean square error (MSE). Encoder.

[0033] Language alignment loss Represented as: In the formula, This represents the semantic alignment loss function between linguistic features, used to compute Gaussian semantic features. With high-dimensional semantic information The loss error between them.

[0034] This embodiment applies language alignment loss. Let Gaussian semantic features Learning high-dimensional semantic information extracted from language models .

[0035] S20: The Gaussian semantic features are fused with other Gaussian attribute features of the 3D Gaussian rendering model to obtain all Gaussian attribute information. All Gaussian attribute information and text information are input into the large language model, and the large language model outputs segmentation feature vectors. Among them, the text information is the natural language command input by the user.

[0036] In this embodiment, the Gaussian features are first dimensionally mapped and semantically enhanced using an encoder-decoder module to train a Gaussian semantic representation aligned with the feature dimensions of the Large Language Model (LLM). To further achieve cross-modal reasoning and 3D segmentation, a LLM output module is designed. This module takes the aligned Gaussian semantic features and other Gaussian features as input to guide the LLM in outputting cross-modal information. During inference, the LLM combines the input Gaussian features with the existing semantic context to output a representation specifically designed for the segmentation task. <seg>This vector, located at the end of the output sequence of the large language model, is used to explicitly represent the segmentation result associated with the input Gaussian features. In this embodiment, this vector is used during the inference phase. <seg>Vectors are extracted and decoded to tightly integrate linguistic semantics with 3D spatial features, achieving efficient and accurate 3D scene target segmentation. This embodiment also employs LoRA for fine-tuning of the large language model.

[0037] All of Gauss's attribute information , and text information As input to a large language model, the large language model outputs feature embeddings containing segmentation information. : In the formula, Other Gaussian property features of the 3D Gaussian rendering model, For large language models.

[0038] After this stage, the large language model generates a vector in the last dimension of its output sequence that specifically represents the segmentation task: In the formula, To segment feature vectors, This represents the last dimension of the output result from the large language model.

[0039] In the embedding of information from the large language model output, the last layer is the segmentation result from the Gaussian features of the reference. (Segmentation feature vector) This is the segmentation result corresponding to the input Gaussian features.

[0040] S30: Input the Gaussian semantic features and segmentation feature vectors into the segmentation information output module. The segmentation information output module performs cross-modal fusion of the Gaussian semantic features and segmentation feature vectors to generate a semantic segmentation mask.

[0041] In this embodiment, a decoder structure similar to the LISA model is adopted in the segmentation inference stage. The Gaussian semantic features learned by the encoder-decoder module are jointly modeled with other Gaussian features and inferred from the large language model. <seg>Vectors are aligned and fused for feature integration. This cross-modal fusion process effectively integrates linguistic semantic information with 3D geometric features, forming a unified high-dimensional feature representation. Based on this, the differentiable rendering capability of 3DGS is further utilized to map the fused features to pixel space, ultimately generating a semantic mask corresponding to the input scene, achieving accurate segmentation of the target object. This design not only ensures consistency between semantic and spatial features but also significantly improves the interpretability and accuracy of the segmentation results.

[0042] In the segmentation information output module, this embodiment adopts a decoder structure similar to the LISA model, jointly modeling the Gaussian semantic features learned by the encoder-decoder module with other Gaussian features, and then combining them with the inference results from the large language model. <seg>Cross-modal feature alignment and fusion of vectors: Among them, decoder This step does not participate in parameter updates; it is only used to remap the feature dimensions back to the original feature space. The mapped features are then passed through mapping layer F and cross-modal aligned with the output of the large language model. The calculation formula is as follows: In the formula, for decoder (such as) Figure 2 (as shown) For pre-trained The high-dimensional semantic information restored by the decoder This is a mapping layer used to map high-dimensional semantic information. Mapping to a unified dimension to align features. The semantic information after dimensional alignment by the mapping layer is used as The input at one end of the decoder.

[0043] Will Input to the final The decoder, by mixing Gaussian features with the results of a large language model, obtains a semantic segmentation mask: In the formula, For semantic segmentation mask, For the final decoder (such as) Figure 2 (as shown) To segment feature vectors.

[0044] Through the aforementioned fusion mechanism, this embodiment effectively integrates linguistic semantics and 3D geometric features to form a unified high-dimensional representation. Finally, leveraging the differentiable rendering capabilities of 3DGS, the fused features are mapped to pixel space, outputting a semantic mask corresponding to the input scene, thus achieving accurate segmentation of the target object. This design not only ensures consistency between semantic and spatial features but also significantly improves the interpretability, stability, and accuracy of the segmentation results.

[0045] In some embodiments, the segmentation information output module is trained based on a set loss function, and a semantic segmentation mask is generated based on the trained segmentation information output module. Specifically, the text loss is first calculated using the following formula: In the formula, Cross-entropy loss is used to measure the difference between the features generated by LLM and the real text labels; For target text labels; The text semantic constraint loss measures the degree of matching between the semantic features generated by the large language model and the text description.

[0046] Then, the mask loss is calculated using the following formula: In the formula, The weights for the BCE loss are used to control the influence of the binary cross-entropy loss (BCE) on the segmentation loss. Binary cross-entropy loss; The Dice loss weights are used to control the proportion of the Dice loss term. For Dice coefficient loss; For the final The decoder's output mask. This is the true mask, used to represent the standard answer for the segmentation task.

[0047] pass and Hybrid loss is used to constrain the prediction results of the mask. The total loss is jointly constrained by the text loss and the mask loss. The formula for calculating the total loss is as follows: In the formula, For the total loss, For text semantic constraint loss, The segmentation mask supervision loss is used to measure the deviation between the segmentation result predicted by the model and the true mask of the target region. , which is the weighting coefficient for text semantic loss, used to adjust the contribution ratio of language alignment to the overall optimization; These are the weighting coefficients for the mask loss, used to balance the impact of segmentation accuracy on the total loss.

[0048] S40: Based on semantic segmentation mask, output the segmentation result of the target object in the 3D scene.

[0049] In some embodiments, combined with Figure 2 As shown, this embodiment details the training process of a 3D scene segmentation method. This process aims to train a system capable of understanding complex natural language instructions and outputting accurate 3D segmentation results. The training process mainly includes three stages: 2D semantic feature extraction and compression, cross-modal reasoning using a large language model, and segmentation mask generation and supervision.

[0050] In the two-dimensional semantic feature extraction and compression stage, an RGB rendered image generated by a pre-trained 3D Gaussian rendering (3DGS) model is acquired. This rendered image is input into a pre-trained two-dimensional visual language model CLIP to extract high-dimensional semantic features. The semantic features extracted by CLIP are then fed into an L encoder after passing through a mapping layer. The function of the L encoder is to compress the high-dimensional semantic features into a low-dimensional latent space, forming compact Gaussian semantic features. This compression process aims to reduce the subsequent computational burden and facilitate fusion with the Gaussian attributes of 3DGS (such as color, position, covariance, etc.). To ensure that semantic information is not lost during compression, this embodiment introduces an L decoder to reconstruct the compressed low-dimensional features into the high-dimensional CLIP feature space. The encoder-decoder training is supervised by calculating the dimension alignment loss (Loss) between the reconstructed features and the original CLIP features, forcing the low-dimensional Gaussian semantic features to retain sufficient semantic information.

[0051] In the cross-modal reasoning stage of the large language model, the aligned low-dimensional Gaussian semantic features are concatenated with the original 3DGS Gaussian attribute features to jointly construct a fused feature representation of the 3D scene. Simultaneously, the user inputs a natural language command, such as: "What food has the highest vitamin C content in the image? Please provide the segmented output." The fused 3D scene features and the text command are input together into a large language model (LLM). To efficiently adapt the LLM to this task, this embodiment uses LoRA technology to fine-tune the LLM. Based on its powerful semantic understanding and reasoning capabilities, the large language model processes the input cross-modal information and generates a specific segmentation identifier at the end of its output sequence. Figure 2 As shown, the final output of LLM is "The output is <seg>",in h seg That is, the last one produced <seg>A vector that encodes the segmentation intent corresponding to the text instruction.

[0052] In the segmentation mask generation and supervision stage, the low-dimensional Gaussian semantic features output by the L encoder in the first stage are input into another... L Upsampling is performed in the decoder to obtain the mapped features. f' Through a F The decoder will upsample the features f' Segmentation vectors output by large language models h seg Cross-modal fusion is performed to generate a binary segmentation mask. This segmentation mask is then compared with the real image labels rendered by the 3DGS model, and the training of the entire model is supervised by a combined loss function (collectively referred to as Loss in the diagram). This loss function typically includes constraints on the accuracy of the segmentation mask and on the degree of semantic alignment.

[0053] It should be noted that in the entire training architecture, L Encoder, used for feature fusion F The decoder and the LoRA adapter of the large language model are labeled as trainable parameters. However, the backbone parameters of the CLIP model, the Gaussian properties of 3DGS, and the LLM of the large language model are fixed and do not participate in training, thereby ensuring the stability of pre-trained knowledge and effectively controlling training costs.

[0054] In summary, the 3D scene segmentation method provided in this application, compared with existing segmentation methods, has the following significant advantages: (1) Cross-modal semantic fusion: This method transfers the semantic information generated by the 2D visual language model to the 3DGS, so that Gaussian not only contains the original geometric color and other information, but also carries the semantic information extracted by the 2D model, which is used to align with the information of the large language model, thus solving the problem that Gaussian cannot directly understand the cross-modal information of the large language model.

[0055] (2) Gaussian Semantic Information Learning: In 3DGS semantic segmentation, the key is how to make Gaussian features have stable and interpretable semantic expressions. The encoder-decoder structure designed in this method uses a 2D visual language model as a medium to effectively solve the mismatch between the semantic feature dimension and the Gaussian representation: on the one hand, feature compression reduces the computational burden of high-dimensional semantic information and improves rendering efficiency; on the other hand, feature reconstruction and alignment mechanisms ensure that low-dimensional Gaussian features can still fully preserve and express the original semantic information, thereby achieving a balance between semantic carrying capacity and computational efficiency.

[0056] (3) Large Language Model Information Fusion: This method innovatively introduces a multimodal large language model, which deeply integrates three-dimensional Gaussian features with text semantic reasoning capabilities, enabling the system to achieve accurate segmentation of three-dimensional scenes under complex natural language instructions. Through this cross-modal fusion mechanism, the model not only possesses stronger semantic understanding and spatial reasoning capabilities, but also maintains good robustness and adaptability in open semantic environments, thereby significantly improving the accuracy and scalability of segmentation tasks.

[0057] Furthermore, with the rapid development of Large Language Models (LLMs) in intelligent question answering, semantic understanding, and multimodal reasoning, their cross-modal fusion capabilities have received significant attention from both academia and industry. This application's method fully leverages the semantic parsing and reasoning advantages of LLMs, combined with the efficient 3D representation capabilities of 3D Gaussian Rendering (3DGS). In emerging interactive fields such as virtual reality and augmented reality, it enables natural language-based 3D scene segmentation and interaction; in intelligent robots and autonomous driving systems, it can improve the accuracy of environmental perception and target recognition through language guidance; and in digital twins, 3D reconstruction, and film and game production, it can utilize semantically driven segmentation results for rapid modeling and editing. With the continuous iteration and upgrading of Large Language Models and the expanding application scope, this application's method possesses excellent adaptability and scalability, enabling it to play a greater role in future multimodal intelligent application scenarios.

[0058] This application also provides a three-dimensional scene segmentation device, such as... Figure 3 As shown, the 3D scene segmentation device includes: The first feature extraction module 301 is configured to acquire an RGB rendered image, which is generated by a pre-trained three-dimensional Gaussian rendering model. The RGB rendered image is input to an encoder-decoder module, which includes a two-dimensional visual language model and an encoder. The two-dimensional visual language model and the encoder process the input RGB rendered image to obtain Gaussian semantic features. The second feature extraction module 302 is configured to fuse the Gaussian semantic features with other Gaussian attribute features of the 3D Gaussian rendering model to obtain all attribute information of Gaussian, and input the all attribute information of Gaussian and text information into a large language model, and output a segmentation feature vector through the large language model; wherein, the text information is a natural language command input by the user; The segmentation information determination module 303 is configured to input the Gaussian semantic features and the segmentation feature vector to the segmentation information output module, and perform cross-modal fusion of the Gaussian semantic features and the segmentation feature vector through the segmentation information output module to generate a semantic segmentation mask. The target segmentation module 304 is configured to output the segmentation result of the target object in the three-dimensional scene based on the semantic segmentation mask.

[0059] This application provides an electronic device. The electronic device may include a processor and a memory, wherein the processor and the memory can communicate; exemplarily, the processor and the memory communicate via a communication bus.

[0060] The processor executes computer execution instructions stored in memory, causing the processor to perform the scheme in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0061] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0062] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0063] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the technical solution of the three-dimensional scene segmentation method described in the above embodiments.

[0064] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the three-dimensional scene segmentation method in the above embodiments.

[0065] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0066] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0067] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0068] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0069] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0070] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0071] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Architecture (EISA) buses, etc. Buses can be categorized into address buses, data buses, control buses, etc.

[0072] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0073] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0074] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.< / seg> < / seg> < / seg> < / seg> < / seg> < / seg>

Claims

1. A three-dimensional scene segmentation method, characterized in that, The method includes: An RGB rendered image is obtained, which is generated by a pre-trained 3D Gaussian rendering model. The RGB rendered image is input into an encoder-decoder module, which includes a 2D visual language model and an encoder. The 2D visual language model and the encoder process the input RGB rendered image to obtain Gaussian semantic features. The Gaussian semantic features are fused with other Gaussian attribute features of the 3D Gaussian rendering model to obtain all Gaussian attribute information. The all Gaussian attribute information and text information are then input into a large language model, which outputs a segmentation feature vector. The text information is the natural language command input by the user. The Gaussian semantic features and the segmentation feature vector are input into the segmentation information output module. The segmentation information output module performs cross-modal fusion on the Gaussian semantic features and the segmentation feature vector to generate a semantic segmentation mask. Based on the semantic segmentation mask, the segmentation result of the target object in the 3D scene is output.

2. The three-dimensional scene segmentation method according to claim 1, characterized in that, The process of processing the input RGB rendered image using the two-dimensional visual language model and encoder to obtain Gaussian semantic features is expressed as follows: In the formula, For high-dimensional semantic information, For a two-dimensional visual language model, Rendering images for RGB Gaussian semantic features for Encoder.

3. The three-dimensional scene segmentation method according to claim 2, characterized in that, The method further includes training the encoder-decoder module based on reconstruction loss and language alignment loss, and obtaining Gaussian semantic features based on the trained encoder-decoder module; wherein, the reconstruction loss and language alignment loss are expressed as: In the formula, For reconstruction losses; The mean squared error is used to calculate the error loss between the compressed semantic information and the high-dimensional semantic information. These are the features after encoder compression; For language alignment loss; This represents the semantic alignment loss function between linguistic features, used to compute Gaussian semantic features. With high-dimensional semantic information The loss error between them.

4. The three-dimensional scene segmentation method according to claim 1, characterized in that, The large language model outputs segmentation feature vectors in the following manner: In the formula, For feature embedding containing segmentation information, For text information, Gaussian semantic features Other Gaussian property features of the 3D Gaussian rendering model, This contains all the attribute information for Gauss. For large language models, To segment feature vectors, This represents the last dimension of the output result from the large language model.

5. The three-dimensional scene segmentation method according to claim 1, characterized in that, The method by which the segmentation information output module performs cross-modal fusion of the Gaussian semantic features and the segmentation feature vector is as follows: In the formula, Gaussian semantic features for Decoder For pre-trained The high-dimensional semantic information restored by the decoder This is a mapping layer used to map high-dimensional semantic information. Mapping to a unified dimension to align features. The semantic information after dimensional alignment by the mapping layer is used as The input at one end of the decoder, For semantic segmentation mask, for Decoder To segment feature vectors.

6. The three-dimensional scene segmentation method according to claim 5, characterized in that, The method further includes training the segmentation information output module based on a set loss function, and generating a semantic segmentation mask based on the trained segmentation information output module; wherein, the set loss function is expressed as: In the formula, Total loss; The text semantic constraint loss is used to measure the degree of matching between the semantic features generated by the large language model and the text description; The segmentation mask supervision loss is used to measure the deviation between the segmentation result predicted by the model and the true mask of the target region. , which is the weighting coefficient for text semantic loss, used to adjust the contribution ratio of language alignment to the overall optimization; These are the weighting coefficients for the mask loss, used to balance the impact of segmentation accuracy on the total loss.

7. The three-dimensional scene segmentation method according to claim 6, characterized in that, Determined by the following formula and : In the formula, Cross-entropy loss is used to measure the difference between the features generated by LLM and the real text labels; Embedding features containing segmentation information; For target text labels; The weights for the BCE loss are used to control the strength of the influence of the binary cross-entropy loss (BCE) on the segmentation loss. Binary cross-entropy loss; The Dice loss weights are used to control the proportion of the Dice loss term. For Dice coefficient loss; For the final Decoder output mask; This is the true mask, used to represent the standard answer for the segmentation task.

8. A three-dimensional scene segmentation device, characterized in that, The device includes: The first feature extraction module is configured to acquire an RGB rendered image, which is generated by a pre-trained three-dimensional Gaussian rendering model. The RGB rendered image is input to an encoder-decoder module, which includes a two-dimensional visual language model and an encoder. The two-dimensional visual language model and the encoder process the input RGB rendered image to obtain Gaussian semantic features. The second feature extraction module is configured to fuse the Gaussian semantic features with other Gaussian attribute features of the 3D Gaussian rendering model to obtain all Gaussian attribute information, and input the all Gaussian attribute information and text information into a large language model, and output a segmentation feature vector through the large language model; wherein, the text information is a natural language command input by the user; The segmentation information determination module is configured to input the Gaussian semantic features and the segmentation feature vectors into the segmentation information output module, and perform cross-modal fusion of the Gaussian semantic features and the segmentation feature vectors through the segmentation information output module to generate a semantic segmentation mask. The target segmentation module is configured to output the segmentation result of the target object in the 3D scene based on the semantic segmentation mask.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes the computer execution instructions stored in the memory to implement the three-dimensional scene segmentation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the three-dimensional scene segmentation method as described in any one of claims 1-7.

Images