Online segmentation method and device based on three-dimensional Gaussian, equipment and storage medium

Abstract

The embodiment of the invention provides an online segmentation method and device based on three-dimensional Gaussian, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a three-dimensional Gaussian model generated based on original camera viewpoints, selecting an interactive camera viewpoint from the original camera viewpoints in response to a viewpoint selection instruction, acquiring a three-dimensional world position of a target object obtained based on the three-dimensional Gaussian model under the interactive camera viewpoint, determining a virtual sphere radius based on the three-dimensional world position, and displaying the virtual sphere radius. The method comprises the following steps of: constructing a three-dimensional world position, determining a ground normal vector, constructing a virtual spherical coordinate system by taking the three-dimensional world position as a sphere center, taking a virtual sphere radius as a radius and taking the ground normal vector as an axial vector, sampling based on the virtual spherical coordinate system to obtain a plurality of virtual camera viewpoints, and carrying out online segmentation on a target object according to the virtual camera viewpoints and interactive camera viewpoints. And in combination with a virtual camera viewpoint, the segmentation model obtains multi-dimensional information of the object at the same time, so that segmentation errors caused by missing of single-view-angle information are avoided, and the accuracy of a segmentation result is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to online segmentation methods, apparatus, devices and storage media based on three-dimensional Gaussian. Background Technology

[0002] The advantage of online 3D Gaussian segmentation technology lies in its ability to directly adapt to diverse application needs without requiring pre-training for specific scenarios. The specific process is as follows: First, based on the 3D Gaussian model generated by 3D Gaussian sputtering, the 2D segmentation results corresponding to each virtual view are obtained; then, view frustum filtering is performed on these 2D results in 3D space, and through precise intersection operations, the final 3D segmentation result of the target object is obtained.

[0003] In related technologies, sampling is often performed on the original camera array, or only a limited transformation (such as a small rotation) is made to the original camera array before sampling. This approach to planning multiple viewpoints has significant limitations: firstly, in most cases, it fails to provide good coverage of the target object, easily creating blind spots; secondly, the planned multiple viewpoints may not have good texture continuity, leading to poor cross-view tracking performance, and it has low adaptability to large objects. Many viewpoints may only be able to observe a portion of the object, making it difficult to achieve high-quality 3D segmentation through 2D segmentation, ultimately resulting in poor 3D segmentation results. Summary of the Invention

[0004] The main objective of this application is to propose an online segmentation method, apparatus, device, and storage medium based on three-dimensional Gaussian to improve the accuracy of online three-dimensional Gaussian segmentation results.

[0005] To achieve the above objectives, a first aspect of this application proposes an online segmentation method based on three-dimensional Gaussian, comprising: Obtain a 3D Gaussian model generated based on the original camera viewpoint; In response to a viewpoint selection command, an interactive camera viewpoint is selected from the original camera viewpoints, and the three-dimensional world position of the target object is obtained based on the three-dimensional Gaussian model under the interactive camera viewpoint. The radius of the virtual sphere is determined based on the three-dimensional world position, and the ground normal vector is determined. A virtual sphere coordinate system is constructed with the three-dimensional world position as the center of the sphere, the radius of the virtual sphere as the radius, and the ground normal vector as the axis vector. Based on the virtual spherical coordinate system, multiple virtual camera viewpoints are obtained, and the target object is segmented online based on the virtual camera viewpoints and the interactive camera viewpoints.

[0006] In some embodiments, obtaining the 3D world position of the target object based on the 3D Gaussian model from the viewpoint of the interactive camera includes: A two-dimensional mask of the target object is obtained based on the viewpoint of the interactive camera; The target object's three-dimensional world position in the world coordinate system is obtained based on the camera parameters corresponding to the two-dimensional mask and the interactive camera viewpoint.

[0007] In some embodiments, acquiring a two-dimensional mask of the target object based on the interactive camera viewpoint includes: The interactive view is obtained by rendering the interactive camera viewpoint based on the three-dimensional Gaussian model. Obtain prompt words related to the target object, input the prompt words and the interactive view into the visual basic segmentation model for image processing, and obtain the two-dimensional mask.

[0008] In some embodiments, obtaining the three-dimensional world position of the target object in the world coordinate system based on the camera parameters corresponding to the two-dimensional mask and the interactive camera viewpoint includes: Calculate the geometric centroid of the foreground region in the two-dimensional mask to obtain the two-dimensional position of the target object in the two-dimensional view; Obtain the interactive depth map rendered from the viewpoint of the interactive camera based on the three-dimensional Gaussian model, and query the interactive depth map to obtain the depth information corresponding to the two-dimensional position; The three-dimensional world position is obtained based at least on the depth information, the two-dimensional position, and the camera parameters.

[0009] In some embodiments, the camera parameters include an intrinsic parameter matrix and an extrinsic parameter matrix, and obtaining the three-dimensional world position based at least on the depth information, the two-dimensional position, and the camera parameters includes: The three-dimensional coordinates of the target object are obtained based on the two-dimensional position and the depth information; The 3D camera position in the camera coordinate system is obtained based on the intrinsic parameter matrix, the depth information, and the 3D coordinates. The 3D world position is obtained based on the extrinsic parameter matrix and the 3D camera position.

[0010] In some embodiments, determining the virtual sphere radius based on the three-dimensional world position includes: obtaining the camera world coordinates corresponding to the interactive camera's viewpoint, and obtaining the virtual sphere radius based on the distance between the camera world coordinates and the three-dimensional world position.

[0011] In some embodiments, the axis vector indicates a pitch angle of zero, and the sampling based on the virtual spherical coordinate system to obtain multiple virtual camera viewpoints includes: Set the preset pitch angle corresponding to the virtual camera to obtain the sampling plane; On the sampling plane, sampling is performed within a preset sampling range to obtain multiple sampling points. These sampling points are then used as the virtual camera viewpoints to obtain the camera parameters corresponding to each virtual camera viewpoint.

[0012] To achieve the above objectives, a second aspect of this application proposes an online segmentation device based on three-dimensional Gaussian, comprising: Data acquisition module: used to acquire a 3D Gaussian model generated based on the original camera viewpoint; Interactive selection module: used to select an interactive camera viewpoint from the original camera viewpoint in response to the viewpoint selection command, and obtain the three-dimensional world position of the target object based on the three-dimensional Gaussian model under the interactive camera viewpoint; Spherical coordinate system construction module: used to determine the radius of the virtual sphere based on the three-dimensional world position, and to determine the ground normal vector, and to construct a virtual sphere coordinate system with the three-dimensional world position as the center of the sphere, the radius of the virtual sphere as the radius, and the ground normal vector as the axis vector; Virtual sampling module: used to sample based on the virtual spherical coordinate system to obtain multiple virtual camera viewpoints, and to perform online segmentation of the target object based on the virtual camera viewpoints and the interactive camera viewpoints.

[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a storage medium that stores a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0015] The online segmentation method, apparatus, device, and storage medium based on 3D Gaussian proposed in this application acquire a 3D Gaussian model generated from the original camera viewpoint. Responding to a viewpoint selection command, an interactive camera viewpoint is selected from the original camera viewpoints. The 3D world position of the target object, obtained from the 3D Gaussian model under the interactive camera viewpoint, is acquired. The radius of a virtual sphere is determined based on the 3D world position, and the ground normal vector is also determined. A virtual sphere coordinate system is constructed with the 3D world position as the sphere center, the virtual sphere radius as the radius, and the ground normal vector as the axis vector. Sampling is performed based on the virtual sphere coordinate system to obtain multiple virtual camera viewpoints. Online segmentation of the target object is then performed based on the virtual camera viewpoints and the interactive camera viewpoint. This application embodiment first reflects the actual coordinates of the target object in 3D space by selecting an interactive camera viewpoint, thereby reducing position calculation errors caused by viewpoint deviations. Next, a virtual spherical coordinate system is constructed around the target object. The radius of the virtual sphere matches the actual size of the target object, avoiding the inclusion of irrelevant background areas due to an excessively large radius, or the omission of object edge views due to an excessively small radius. The ground normal vector ensures that the virtual spherical coordinate system maintains consistency with the spatial orientation of the target object, avoiding viewpoint sampling offsets caused by coordinate system tilt. By constructing this coordinate system, the virtual camera viewpoints obtained through subsequent sampling can form a uniform and comprehensive viewpoint distribution around the target object, covering key areas such as the sides and back of the object that may not be covered by the interactive camera viewpoint, thus solving the problem of incomplete coverage by a single or a few original viewpoints. Compared to relying solely on the original viewpoint or viewpoints with limited transformations, combining the virtual camera viewpoint and the interactive camera viewpoint can capture the appearance features and spatial relationships of the target object from different directions. This allows the segmentation model to simultaneously acquire multi-dimensional information such as the front, sides, and detailed textures of the object, avoiding inaccurate segmentation boundaries or missegmentation of the object's internal structure due to missing information from a single viewpoint, ultimately improving the accuracy of online 3D Gaussian segmentation results. Attached Figure Description

[0016] Figure 1 This is a flowchart of the online segmentation method based on three-dimensional Gaussian provided in the embodiments of this application.

[0017] Figure 2 This is a flowchart provided in an embodiment of the present application for obtaining the three-dimensional world position of a target object based on a three-dimensional Gaussian model from the viewpoint of an interactive camera.

[0018] Figure 3 This is a flowchart of a two-dimensional mask for acquiring a target object based on an interactive camera viewpoint, provided in an embodiment of this application.

[0019] Figure 4 This is a flowchart provided in this application embodiment for obtaining the three-dimensional world position of the target object in the world coordinate system based on the camera parameters corresponding to the two-dimensional mask and the interactive camera viewpoint.

[0020] Figure 5 This is a flowchart provided in the embodiments of this application for obtaining the three-dimensional world position based at least on depth information, two-dimensional position and camera parameters.

[0021] Figure 6 This is a flowchart of sampling based on a virtual spherical coordinate system to obtain multiple virtual camera viewpoints, provided in an embodiment of this application.

[0022] Figure 7 This is a schematic diagram of a virtual camera viewpoint provided in an embodiment of this application.

[0023] Figure 8 This is a structural block diagram of an online segmentation device based on three-dimensional Gaussian provided in another embodiment of this application.

[0024] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0028] First, let's analyze some of the terms used in this application: Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0029] 3D Gaussian Splatting (3DGS), as one of the latest mainstream technologies for 3D reconstruction, is an important part of the popularization of 3D media technology. Against the backdrop of technological trends such as the metaverse, virtual reality, mixed reality, and spatial intelligence, 3DGS has become one of the mainstream carriers. However, 3DGS itself lacks semantic information, exhibiting significant shortcomings in interactivity, editability, and scene semantic understanding. The advantage of online 3D Gaussian segmentation technology lies in its ability to directly adapt to diverse application needs without requiring pre-training for specific scenes. Its specific process is as follows: First, based on the 3D Gaussian model generated by 3D Gaussian sputtering, the 2D segmentation results corresponding to each virtual view are obtained; then, these 2D results are subjected to view frustum filtering in 3D space, and through precise intersection operations, the final 3D segmentation result of the target object is obtained.

[0030] In related technologies, sampling is often performed on the original camera array, or only a limited transformation (such as a small rotation) is made to the original camera array before sampling. This approach to planning multiple viewpoints has significant limitations: firstly, in most cases, it fails to provide good coverage of the target object, easily creating blind spots; secondly, the planned multiple viewpoints may not have good texture continuity, leading to poor cross-view tracking performance, and it has low adaptability to large objects. Many viewpoints may only be able to observe a portion of the object, making it difficult to achieve high-quality 3D segmentation through 2D segmentation, ultimately resulting in poor 3D segmentation results.

[0031] Based on this, embodiments of this application provide an online segmentation method, apparatus, device, and storage medium based on 3D Gaussian. By selecting an interactive camera viewpoint to reflect the actual coordinates of the target object in 3D space, position calculation errors caused by viewpoint deviations are reduced. Next, a virtual spherical coordinate system is constructed around the target object. The radius of the virtual sphere matches the actual size of the target object, avoiding the inclusion of irrelevant background areas due to an excessively large radius, or the omission of object edge views due to an excessively small radius. The ground normal vector ensures that the virtual spherical coordinate system maintains consistency with the spatial orientation of the target object, avoiding viewpoint sampling offsets caused by coordinate system tilt. By constructing this coordinate system, the subsequently sampled virtual camera viewpoints can form a uniform and comprehensive viewpoint distribution around the target object, covering key areas such as the sides and back of the object that the interactive camera viewpoint may not have covered, thereby solving the problem of incomplete coverage by a single or a small number of original viewpoints. Compared to relying solely on the original viewpoint or viewpoints with limited transformations, combining virtual camera viewpoints and interactive camera viewpoints can capture the appearance features and spatial relationships of target objects from different directions. This allows the segmentation model to simultaneously acquire multi-dimensional information such as the front, side, and detailed textures of the object, avoiding inaccurate segmentation boundaries or missegmentation of the object's internal structure due to the lack of information from a single perspective, ultimately improving the accuracy of online 3D Gaussian segmentation results.

[0032] This application provides an online segmentation method, apparatus, device, and storage medium based on three-dimensional Gaussian, which are specifically described through the following embodiments. First, the online segmentation method based on three-dimensional Gaussian in this application embodiment is described.

[0033] This application's embodiments can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0034] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0035] The online segmentation method based on 3D Gaussian provided in this application relates to the field of image processing technology. This online segmentation method based on 3D Gaussian can be applied to a terminal, a server, or a computer program running on either the terminal or the server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client supporting online segmentation based on 3D Gaussian, i.e., a program that only needs to be downloaded to a browser environment to run; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. The online segmentation method based on 3D Gaussian can be executed by the terminal or the server, or by the terminal and the server working together.

[0036] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. Additionally, the terminal can also be a smart in-vehicle device. This smart in-vehicle device uses the online segmentation method based on three-dimensional Gaussian in this embodiment to provide related services and improve the driving experience. The server can be an independent server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The terminal and server can connect via Bluetooth, Universal Serial Bus (USB), or network communication methods; this embodiment does not impose any limitations.

[0037] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0038] The following describes the online segmentation method based on three-dimensional Gaussian in the embodiments of this application.

[0039] Figure 1 This is an optional flowchart of the online segmentation method based on 3D Gaussian provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 110 to 140. It is also understood that this embodiment... Figure 1 The order of steps 110 to 140 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0040] Step 110: Obtain a 3D Gaussian model generated based on the original camera viewpoint.

[0041] In one embodiment, for the target scene, an image acquisition device can be deployed in a 360-degree panoramic surround layout to achieve synchronous acquisition. This surround layout can cover the target scene from all directions, ensuring no angular information is missed, thus enabling the acquisition of data from all angles. Each image acquisition device corresponds to a specific viewpoint, which is called the original camera viewpoint, similar to observing the same object from different positions, where the images seen from different positions differ. At each viewpoint, the image acquisition device continuously captures corresponding video sequences. The video sequence contains multiple video frames recorded at different acquisition times. At each acquisition time, sparse point cloud data of the target scene is generated based on the corresponding video frames. Next, COLMAP is used to calibrate the camera system corresponding to the image acquisition device, thereby acquiring the camera's intrinsic and extrinsic parameters. The intrinsic parameters mainly include the camera's focal length, principal point position, and lens distortion parameters, which reflect the camera's internal imaging characteristics. The extrinsic parameters relate to the camera's position and attitude information in space, such as the camera's rotation angle and translation vector, aiming to determine the spatial relationship between the camera and the captured target scene. After COLMAP calibration is completed, the camera's intrinsic and extrinsic parameters can be used to calibrate the sparse point cloud data, thereby obtaining a 3D Gaussian model corresponding to each acquisition time. This 3D Gaussian model includes multiple 3D Gaussian primitives. The attributes of these primitives are trained and adjusted through a 3D Gaussian splashing training process. These attributes can include: center position coordinates, color, opacity, rotation, scaling, normal vector, and visibility value. Finally, a reconstructed 3D Gaussian model based on the original camera viewpoint is obtained.

[0042] Step 120: In response to the viewpoint selection command, select the interactive camera viewpoint from the original camera viewpoint, and obtain the 3D world position of the target object based on the 3D Gaussian model under the interactive camera viewpoint.

[0043] In one embodiment, a user can select an original camera viewpoint as the interactive camera viewpoint through a visual interface. The user's selection action on the visual interface generates a viewpoint selection command. Upon responding to this command, the user-selected original camera viewpoint is used as the interactive camera viewpoint. Then, the 3D world position of the target object is determined based on this interactive camera viewpoint.

[0044] In one embodiment, reference is made to Figure 2 , Figure 2 This is a flowchart of obtaining the 3D world position of a target object based on a 3D Gaussian model from the viewpoint of an interactive camera, provided in an embodiment of this application. The flowchart specifically includes the following steps: Step 210: Obtain a two-dimensional mask of the target object based on the interactive camera viewpoint.

[0045] In one embodiment, reference is made to Figure 3 , Figure 3 This is a flowchart of a method for acquiring a 2D mask of a target object based on an interactive camera viewpoint, provided in an embodiment of this application. The flowchart specifically includes the following steps: Step 310: Render the interactive camera viewpoint based on the 3D Gaussian model to obtain the interactive view.

[0046] In one embodiment, since the 3D Gaussian model can already characterize the 3D geometry, surface texture, and spatial relationships of the target object in the target scene, after clarifying the observation angle using the interactive camera viewpoint, the 3D Gaussian model can be used for rasterized rendering of differentiable tiles on the interactive camera viewpoint. During the rendering process, using the interactive camera viewpoint as the observation reference, the projection position of each 3D Gaussian primitive in the 3D Gaussian model onto the 2D image plane is calculated. Combined with parameters such as weights and covariance, the appearance details and spatial occlusion relationships of the target object are restored, ultimately generating a 2D image similar to the effect of a real camera shot as an interactive view. For example, if the interactive camera viewpoint is the frontal view of the target object, the rendered interactive view will clearly present the frontal outline, surface texture, and boundary between the background and the target object.

[0047] Furthermore, since the mean parameter of each 3D Gaussian primitive in the 3D Gaussian model already contains its 3D position in the world coordinate system when representing the target scene, and depth information is essentially the distance of that 3D position relative to the interactive camera viewpoint along the viewing direction, during the rendering process, the camera's viewing direction is determined based on the camera extrinsic parameters of the interactive camera viewpoint. For each 3D Gaussian primitive in the 3D Gaussian model, the straight-line distance from its mean coordinates to the camera's optical center is calculated; the resulting distance is the depth value corresponding to that 3D Gaussian primitive. When the 3D Gaussian primitive is projected onto a pixel position on the 2D image plane, the calculated depth value is assigned to that pixel. If multiple Gaussian primitives are projected onto the same pixel, the depth value corresponding to the 3D Gaussian primitive with the largest contribution is taken, ensuring the uniqueness and accuracy of the pixel depth.

[0048] The final result is an interactive depth map, in which the value of each pixel directly corresponds to its distance in 3D space from the interactive camera's viewpoint. For example, if the interactive camera viewpoint is capturing a table and a water glass in front of it, the pixels in the depth map with smaller depth values ​​for the water glass area are closer to the camera, while those in the table area have larger depth values ​​and are farther from the camera. The depth map can clearly distinguish the spatial distance relationships between different objects in the target scene.

[0049] The above process can be represented as follows:

[0050] in, Represents a three-dimensional Gaussian model. The camera parameters represent the viewpoint of the interactive camera. These camera parameters include the intrinsic parameter matrix. and extrinsic parameter matrix In the extrinsic parameter matrix Represents the rotation matrix. Represents the translation matrix. This indicates the rendering process. Represents an interactive view. This represents the interaction depth map.

[0051] Step 320: Obtain prompts related to the target object, input the prompts and interactive view into the visual basic segmentation model for image processing, and obtain a two-dimensional mask.

[0052] In one embodiment, to achieve online object segmentation, it is necessary to first clarify the specific location of the object to be segmented. Therefore, this embodiment defines the semantic requirements of the segmentation task by obtaining prompt words related to the target object. The prompt words here can be descriptions manually entered by the user in the visual interface, such as "segmenting the red car in the image" or "extracting the sofa"; or they can be relevant prompt words automatically generated by the system based on the position of the target object after the user selects it by box selection or clicking.

[0053] Next, after obtaining the prompt words, the prompt words and interactive view are input into the visual basic segmentation model for image processing to obtain a two-dimensional mask. The visual basic segmentation model in this embodiment can be a SAM model, CLIP-Decoder model, etc., which has the ability to understand prompt words and match image features. After inputting the prompt words and interactive view, the visual basic segmentation model first parses the semantics of the prompt words, then extracts the visual features of the interactive view. Through precise matching of semantic features and visual features, it locates all pixel regions that match the description of the prompt words, and finally outputs a two-dimensional mask.

[0054] Taking the SAM model as the basic visual segmentation model as an example, the calculation process of the two-dimensional mask is as follows:

[0055] in, Represents a two-dimensional mask. Indicates a prompt word.

[0056] Understandably, a 2D mask is presented as a binary or grayscale image, defining the 2D pixel range of a target object within the interactive view. Pixels corresponding to the target object are marked as foreground, with a pixel value of 1, while background and other areas are marked as background, with a pixel value of 0. In other words, a 2D mask is a matrix of 0s and 1s. For example, if the prompt is "segmenting pedestrians in the image," the visual basic segmentation model will accurately identify the pedestrian pixel region in the interactive view and generate a 2D mask that retains only the pedestrian's outline.

[0057] Step 220: Based on the camera parameters corresponding to the 2D mask and the interactive camera viewpoint, obtain the 3D world position of the target object in the world coordinate system.

[0058] In one embodiment, reference is made to Figure 4 , Figure 4 This is a flowchart illustrating how to obtain the 3D world position of a target object in the world coordinate system based on camera parameters corresponding to a 2D mask and an interactive camera viewpoint, as provided in this application embodiment. The flowchart specifically includes the following steps: Step 410: Calculate the geometric centroid of the foreground region in the 2D mask to obtain the 2D position of the target object in the 2D view.

[0059] Step 420: Obtain the interactive depth map rendered from the interactive camera viewpoint based on the 3D Gaussian model, and query the interactive depth map to obtain the depth information corresponding to the 2D position.

[0060] Step 430: Obtain the three-dimensional world position based at least on depth information, two-dimensional position, and camera parameters.

[0061] In one embodiment, the geometric centroid of the portion of the two-dimensional mask with a pixel value of 1 can be calculated to obtain the two-dimensional position of the target object in the two-dimensional view. Next, query the interaction depth graph. This will give you the depth information of the corresponding two-dimensional location. .

[0062] Next, refer to Figure 5 , Figure 5 This is a flowchart provided in an embodiment of the present application for obtaining a three-dimensional world position based at least on depth information, two-dimensional position, and camera parameters, specifically including the following steps: Step 510: Obtain the three-dimensional coordinates of the target object based on the two-dimensional position and depth information.

[0063] In one embodiment, the three-dimensional coordinates of the target object are represented as follows: .

[0064] Step 520: Obtain the 3D camera position in the camera coordinate system based on the intrinsic parameter matrix, depth information, and 3D coordinates.

[0065] In one embodiment, the position of the three-dimensional camera Represented as:

[0066] in, The intrinsic parameter matrix representing the camera parameters of the interactive camera viewpoint.

[0067] Step 530: Obtain the 3D world position based on the extrinsic parameter matrix and the 3D camera position.

[0068] In one embodiment, three-dimensional world position Represented as:

[0069] in, The extrinsic parameter matrix represents the camera parameters for the interactive camera viewpoint.

[0070] This yields the three-dimensional world position of the target object's geometric centroid in the world coordinate system. Next, the virtual sphere radius is determined based on this three-dimensional position, including: obtaining the camera world coordinates corresponding to the interactive camera's viewpoint, and calculating the virtual sphere radius based on the distance between the camera world coordinates and the three-dimensional world position.

[0071] The calculation process is expressed as follows:

[0072]

[0073] in, The camera world coordinates, representing the viewpoint of the interactive camera, are calculated from the camera's extrinsic parameter matrix. This represents the radius of the virtual sphere obtained by the Euclidean distance between the camera's world coordinates and its 3D world position.

[0074] Step 130: Determine the radius of the virtual sphere based on the 3D world position and determine the ground normal vector. Construct a virtual sphere coordinate system with the 3D world position as the center, the virtual sphere radius as the radius, and the ground normal vector as the axis vector.

[0075] In one embodiment, the Random Sample Consensus (RANSAC) algorithm is used to determine the ground normal vector. RANSAC is a robust method for calculating normal vectors by extracting ground plane parameters from noisy or outlier 3D point cloud data. Therefore, inputting a 3D Gaussian model into the RANSAC algorithm yields the ground normal vector. , represented as Then, based on the three-dimensional world position... Centered on the sphere, with the radius of the virtual sphere With radius and ground normal vector A virtual spherical coordinate system can be constructed using the axis vectors. Here, the ground normal vector... The corresponding axis vector indicates that the pitch angle is zero.

[0076] Step 140: Sampling is performed based on the virtual spherical coordinate system to obtain multiple virtual camera viewpoints. Online segmentation of the target object is then performed based on the virtual camera viewpoints and the interactive camera viewpoints.

[0077] In one embodiment, reference is made to Figure 6 , Figure 6 This is a flowchart of sampling based on a virtual spherical coordinate system to obtain multiple virtual camera viewpoints, provided in an embodiment of this application. The flowchart specifically includes the following steps: Step 610: Set the preset pitch angle corresponding to the virtual camera to obtain the sampling plane.

[0078] In one embodiment, reference is made to Figure 7 , Figure 7 This is a schematic diagram of a virtual camera viewpoint provided in an embodiment of this application. Figure 7 3D world position of the target object in China and Israel With the center of the sphere as the center and the radius of the virtual sphere as the radius of the virtual sphere With radius as the ground normal vector Indicates pitch angle A virtual spherical coordinate system is constructed using axis vectors with a value of zero. Next, a preset pitch angle (e.g., 45 degrees) is set for the virtual cameras. This results in a sampling plane where all virtual cameras are neither directly above the target object nor on a horizontal plane, but rather at an angle, allowing for a clearer view of the target object's top and sides. Based on this, a ring of virtual camera viewpoints is planned around the target object to be segmented, ensuring that these viewpoints uniformly and completely cover the target, and that all virtual camera viewpoints lie on the same sampling plane.

[0079] Step 620: On the sampling plane, sample within the preset sampling range to obtain multiple sampling points. Use the sampling points as virtual camera viewpoints to obtain the camera parameters corresponding to each virtual camera viewpoint.

[0080] In one embodiment, a preset sampling range is set (e.g., a complete 360-degree circle or a certain sector interval), and uniform sampling is performed within the sampling plane to obtain k angle values ​​as sampling points. These sampling points are then used as virtual camera viewpoints to achieve a set of... A virtual camera For example, if the preset sampling range is 360 degrees and k=12, then a virtual camera viewpoint is sampled every 30 degrees. (See reference...) Figure 7 Sampling is performed on the sampling plane, and the resulting virtual camera viewpoints can all be considered as images of the target object taken at a preset pitch angle. Next, based on the position of the virtual camera viewpoint in the world coordinate system, a corresponding extrinsic parameter matrix is ​​generated, with the default value set as the intrinsic parameter matrix. The camera parameters for each virtual camera viewpoint are then obtained from the intrinsic and extrinsic parameter matrices. Therefore, for each sampled virtual camera viewpoint, the spherical coordinate transformation formula is used to determine its pitch and yaw angles. This allows us to calculate its position on the sphere in the virtual spherical coordinate system. The virtual camera's orientation is automatically set to face the center of the sphere, and they are evenly distributed on the circular track where the sampling plane is located.

[0081] As can be seen, this embodiment of the application ensures that the virtual camera can see the target object from every possible angle by uniformly sampling on the sphere. Regardless of the object's complex shape (e.g., with concave structures), it can generate virtual camera viewpoints covering its entire surface, meaning that almost every 3D Gaussian element in the 3D Gaussian model can be seen from multiple virtual perspectives. Simultaneously, the fixed sphere radius and pitch angle ensure that all virtual cameras operate at a scale that allows viewing the overall structure of the target, solving the problem of blind spots. Therefore, it can improve the accuracy of subsequent online segmentation results.

[0082] In one embodiment, online segmentation of the target object is performed based on a common viewpoint formed by the virtual camera viewpoint and the interactive camera viewpoint. During segmentation, it is not necessary to actually take photos from these angles; instead, a pre-trained 3D Gaussian model is used to directly render at the positions of these k+1 common viewpoints, generating corresponding virtual images and depth maps. This virtual rendering method has extremely low cost and can generate images from any angle infinitely, solving the problems of high data acquisition costs and time-consuming processes in the real world. Each virtual image is then input into the visual basic segmentation model, and segmentation is performed based on prompts, thus obtaining a segmentation mask corresponding to each virtual image. The goal is to accurately segment the same target object in different virtual images. Next, for each segmentation mask, the corresponding pixel position in 3D space is calculated using the depth map and camera parameters corresponding to that viewpoint. Then, the 3D Gaussian primitives that contribute the most to this position are found in the 3D Gaussian model, resulting in a view frustum.

[0083] Understandably, these virtual camera viewpoints are actively planned and generated by the system, with uniform and comprehensive viewpoint distribution, representing a high-quality dataset specifically tailored for the current segmentation task. This method eliminates the need for real-world shooting, saving significant time, equipment, and manpower costs. Furthermore, the camera parameters are precisely known during rendering, resulting in accurate 2D-to-3D backprojection without requiring additional estimation.

[0084] This application's embodiments consider the potential unreliability of 2D segmentation results from a single viewpoint, such as the influence of occlusion, lighting interference, or model misjudgment. If the 3D segmentation result is directly inferred from only one or two views, errors at the 2D level can easily be amplified to the 3D level. Therefore, the solution uses multi-view voting to select multiple 3D Gaussian primitives ultimately labeled as target objects: Multi-view 2D segmentation is performed using multiple generated virtual camera viewpoints. The same 3D Gaussian primitive in the 3D Gaussian model can be observed by multiple virtual camera viewpoints, thus receiving segmentation votes from multiple perspectives. A majority voting or probability fusion strategy is then employed (for example, if a 3D Gaussian primitive is identified as a "target object" in 8 out of 10 virtual camera views, it is ultimately classified as the target object). This process effectively filters noise and errors from single-view segmentation, resulting in more consistent and robust 3D segmentation results.

[0085] Furthermore, for large objects, close-up cameras can only observe a portion of them. Based on this partial view, the 2D segmentation process struggles to fully understand the object's overall appearance, easily leading to 3D segmentation failure. However, the embodiments of this application determine the center of the virtual spherical coordinate system by the geometric center of the target object and set the sphere radius according to the object's specific location. All virtual cameras are located on the surface of this fixed-radius sphere, ensuring that each virtual camera observes the target object at a consistent scale and can observe a relatively complete object. This design ensures that all viewpoints cover the complete target, thereby more accurately understanding the overall structure of the target object, which is particularly beneficial for achieving complete segmentation of large objects.

[0086] Therefore, the online segmentation method based on 3D Gaussian in this application does not rely on expensive and imperfect real data acquisition, but instead utilizes the capabilities of computer graphics to actively generate data under ideal conditions, thereby achieving high-quality 3D segmentation.

[0087] The technical solution provided in this application involves acquiring a 3D Gaussian model generated based on the original camera viewpoint, selecting an interactive camera viewpoint from the original camera viewpoint in response to a viewpoint selection command, obtaining the 3D world position of the target object based on the 3D Gaussian model under the interactive camera viewpoint, determining the virtual sphere radius based on the 3D world position, and determining the ground normal vector. A virtual sphere coordinate system is constructed with the 3D world position as the sphere center, the virtual sphere radius as the radius, and the ground normal vector as the axis vector. Sampling is performed based on the virtual sphere coordinate system to obtain multiple virtual camera viewpoints. Online segmentation of the target object is then performed based on the virtual camera viewpoints and the interactive camera viewpoints. This application first selects an interactive camera viewpoint to reflect the actual coordinates of the target object in 3D space, thereby reducing position calculation errors caused by viewpoint deviations. Next, a virtual sphere coordinate system is constructed around the target object. The virtual sphere radius matches the actual size of the target object, avoiding the inclusion of irrelevant background areas due to an excessively large radius, or the omission of object edge views due to an excessively small radius. The ground normal vector ensures that the virtual sphere coordinate system maintains consistency with the spatial orientation of the target object, avoiding viewpoint sampling offsets caused by coordinate system tilt. By constructing this coordinate system, the virtual camera viewpoints obtained through subsequent sampling can form a uniform and comprehensive visual distribution around the target object, covering key areas such as the sides and back of the object that the interactive camera viewpoint may not have covered. This solves the problem of incomplete coverage from a single or a small number of original viewpoints. Compared to relying solely on original viewpoints or viewpoints with limited transformations, combining virtual and interactive camera viewpoints can capture the appearance features and spatial relationships of the target object from different directions. This allows the segmentation model to simultaneously acquire multi-dimensional information such as the front, sides, and detailed textures of the object, avoiding inaccurate segmentation boundaries or missegmentation of the object's internal structure due to missing information from a single viewpoint. Ultimately, this improves the accuracy of online 3D Gaussian segmentation results.

[0088] This application also provides an online segmentation device based on three-dimensional Gaussian, which can implement the above-mentioned online segmentation method based on three-dimensional Gaussian, referring to... Figure 8 The device includes: Data acquisition module 810: Used to acquire a 3D Gaussian model generated based on the original camera viewpoint.

[0089] Interactive selection module 820: In response to a viewpoint selection command, select an interactive camera viewpoint from the original camera viewpoint and obtain the 3D world position of the target object based on a 3D Gaussian model under the interactive camera viewpoint.

[0090] Spherical Coordinate System Construction Module 830: Used to determine the radius of the virtual sphere based on the three-dimensional world position and to determine the ground normal vector, and to construct a virtual spherical coordinate system with the three-dimensional world position as the center of the sphere, the radius of the virtual sphere as the radius, and the ground normal vector as the axis vector.

[0091] Virtual sampling module 840: Used to sample based on a virtual spherical coordinate system to obtain multiple virtual camera viewpoints, and to perform online segmentation of the target object based on the virtual camera viewpoints and interactive camera viewpoints.

[0092] The specific implementation of the online segmentation device based on three-dimensional Gaussian in this embodiment is basically the same as the specific implementation of the online segmentation method based on three-dimensional Gaussian described above, and will not be repeated here.

[0093] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in a memory, and the processor executes the at least one program to implement the online segmentation method based on three-dimensional Gaussian as described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0094] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and called and executed by the processor 901 to execute the online segmentation method based on three-dimensional Gaussian in the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0095] This application embodiment also provides a storage medium that stores a computer program. When the computer program is executed by a processor, it implements the above-described online segmentation method based on three-dimensional Gaussian.

[0096] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0097] The online segmentation method, apparatus, device, and storage medium based on 3D Gaussian proposed in this application acquire a 3D Gaussian model generated from the original camera viewpoint. Responding to a viewpoint selection command, an interactive camera viewpoint is selected from the original camera viewpoints. The 3D world position of the target object, obtained from the 3D Gaussian model under the interactive camera viewpoint, is acquired. The radius of a virtual sphere is determined based on the 3D world position, and the ground normal vector is also determined. A virtual sphere coordinate system is constructed with the 3D world position as the sphere center, the virtual sphere radius as the radius, and the ground normal vector as the axis vector. Sampling is performed based on the virtual sphere coordinate system to obtain multiple virtual camera viewpoints. Online segmentation of the target object is then performed based on the virtual camera viewpoints and the interactive camera viewpoint. This application embodiment first reflects the actual coordinates of the target object in 3D space by selecting an interactive camera viewpoint, thereby reducing position calculation errors caused by viewpoint deviations. Next, a virtual spherical coordinate system is constructed around the target object. The radius of the virtual sphere matches the actual size of the target object, avoiding the inclusion of irrelevant background areas due to an excessively large radius, or the omission of object edge views due to an excessively small radius. The ground normal vector ensures that the virtual spherical coordinate system maintains consistency with the spatial orientation of the target object, avoiding viewpoint sampling offsets caused by coordinate system tilt. By constructing this coordinate system, the virtual camera viewpoints obtained through subsequent sampling can form a uniform and comprehensive viewpoint distribution around the target object, covering key areas such as the sides and back of the object that may not be covered by the interactive camera viewpoint, thus solving the problem of incomplete coverage by a single or a few original viewpoints. Compared to relying solely on the original viewpoint or viewpoints with limited transformations, combining the virtual camera viewpoint and the interactive camera viewpoint can capture the appearance features and spatial relationships of the target object from different directions. This allows the segmentation model to simultaneously acquire multi-dimensional information such as the front, sides, and detailed textures of the object, avoiding inaccurate segmentation boundaries or missegmentation of the object's internal structure due to missing information from a single viewpoint, ultimately improving the accuracy of online 3D Gaussian segmentation results.

[0098] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0099] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0100] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0102] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0103] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0105] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0107] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An online segmentation method based on three-dimensional Gaussian, characterized in that, include: Obtain a 3D Gaussian model generated based on the original camera viewpoint; In response to a viewpoint selection command, an interactive camera viewpoint is selected from the original camera viewpoints, and the three-dimensional world position of the target object is obtained based on the three-dimensional Gaussian model under the interactive camera viewpoint. The radius of the virtual sphere is determined based on the three-dimensional world position, and the ground normal vector is determined. A virtual sphere coordinate system is constructed with the three-dimensional world position as the center of the sphere, the radius of the virtual sphere as the radius, and the ground normal vector as the axis vector. Based on the virtual spherical coordinate system, multiple virtual camera viewpoints are obtained, and the target object is segmented online based on the virtual camera viewpoints and the interactive camera viewpoints.

2. The online segmentation method based on three-dimensional Gaussian as described in claim 1, characterized in that, The step of obtaining the 3D world position of the target object based on the 3D Gaussian model from the viewpoint of the interactive camera includes: A two-dimensional mask of the target object is obtained based on the viewpoint of the interactive camera; The target object's three-dimensional world position in the world coordinate system is obtained based on the camera parameters corresponding to the two-dimensional mask and the interactive camera viewpoint.

3. The online segmentation method based on three-dimensional Gaussian as described in claim 2, characterized in that, The step of acquiring a two-dimensional mask of the target object based on the interactive camera viewpoint includes: The interactive view is obtained by rendering the interactive camera viewpoint based on the three-dimensional Gaussian model. Obtain prompt words related to the target object, input the prompt words and the interactive view into the visual basic segmentation model for image processing, and obtain the two-dimensional mask.

4. The online segmentation method based on three-dimensional Gaussian as described in claim 3, characterized in that, The process of obtaining the three-dimensional world position of the target object in the world coordinate system based on the camera parameters corresponding to the two-dimensional mask and the interactive camera viewpoint includes: Calculate the geometric centroid of the foreground region in the two-dimensional mask to obtain the two-dimensional position of the target object in the two-dimensional view; Obtain the interactive depth map rendered from the viewpoint of the interactive camera based on the three-dimensional Gaussian model, and query the interactive depth map to obtain the depth information corresponding to the two-dimensional position; The three-dimensional world position is obtained based at least on the depth information, the two-dimensional position, and the camera parameters.

5. The online segmentation method based on three-dimensional Gaussian as described in claim 4, characterized in that, The camera parameters include an intrinsic parameter matrix and an extrinsic parameter matrix. Obtaining the three-dimensional world position based at least on the depth information, the two-dimensional position, and the camera parameters includes: The three-dimensional coordinates of the target object are obtained based on the two-dimensional position and the depth information; The 3D camera position in the camera coordinate system is obtained based on the intrinsic parameter matrix, the depth information, and the 3D coordinates. The 3D world position is obtained based on the extrinsic parameter matrix and the 3D camera position.

6. The online segmentation method based on three-dimensional Gaussian as described in claim 1, characterized in that, The step of determining the virtual sphere radius based on the three-dimensional world position includes: obtaining the camera world coordinates corresponding to the interactive camera's viewpoint, and obtaining the virtual sphere radius based on the distance between the camera world coordinates and the three-dimensional world position.

7. The online segmentation method based on three-dimensional Gaussian as described in claim 1, characterized in that, The axis vector indicates that the pitch angle is zero. The sampling based on the virtual spherical coordinate system yields multiple virtual camera viewpoints, including: Set the preset pitch angle corresponding to the virtual camera to obtain the sampling plane; On the sampling plane, sampling is performed within a preset sampling range to obtain multiple sampling points. These sampling points are then used as the virtual camera viewpoints to obtain the camera parameters corresponding to each virtual camera viewpoint.

8. An online segmentation device based on three-dimensional Gaussian, characterized in that, include: Data acquisition module: used to acquire a 3D Gaussian model generated based on the original camera viewpoint; Interactive selection module: used to select an interactive camera viewpoint from the original camera viewpoint in response to the viewpoint selection command, and obtain the three-dimensional world position of the target object based on the three-dimensional Gaussian model under the interactive camera viewpoint; Spherical coordinate system construction module: used to determine the radius of the virtual sphere based on the three-dimensional world position, and to determine the ground normal vector, and to construct a virtual sphere coordinate system with the three-dimensional world position as the center of the sphere, the radius of the virtual sphere as the radius, and the ground normal vector as the axis vector; Virtual sampling module: used to sample based on the virtual spherical coordinate system to obtain multiple virtual camera viewpoints, and to perform online segmentation of the target object based on the virtual camera viewpoints and the interactive camera viewpoints.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the online segmentation method based on three-dimensional Gaussian as described in any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the online segmentation method based on three-dimensional Gaussian as described in any one of claims 1 to 7.

Images