3D mask construction method and device, storage medium and electronic equipment

Abstract

The invention provides a 3D mask construction method and device, a storage medium and electronic equipment, which are applied to the technical field of three-dimensional live-action reconstruction, and the method comprises the steps: combining a target single-view-angle image with prompt information input by a user, obtaining a preliminary 2D mask through an image segmentation model in which multi-view-angle features are cached, and generating a preliminary 3D mask through cross-dimension conversion; a 3D mask is coupled with a 3D Gaussian model based on multi-view image training, a more accurate 2D mask is obtained through further optimization, feature information and a prompt output by an intelligent prompt generation network are fused, and the segmentation effect is iteratively improved. According to the method, through multi-round cross-dimensional mask conversion and feature fusion optimization, accurate segmentation and extraction of a single object in the three-dimensional model are realized, so that monomer processing of the object in the three-dimensional model is effectively realized, and operability and application value of the model are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional real scene reconstruction, and in particular to a 3D mask construction method and device, a storage medium and an electronic device. BACKGROUND

[0002] Three-dimensional real scene reconstruction is widely used in the fields of city modeling and cultural relic protection. Although traditional manual modeling is fine, it is time-consuming and costly, and it is difficult to meet the needs of large-scale complex scenes. Oblique photography modeling realizes automatic reconstruction through multi-angle aerial photography, improves efficiency, but has high requirements for data collection, is easily affected by occlusion, and has limited modeling capability for small objects.

[0003] In recent years, three-dimensional scene modeling and real-time rendering technology realizes detailed and realistic three-dimensional models through distributed Gaussian points, supports rendering from any perspective, and has excellent effects. However, the model generated by this method is usually a whole structure, lacking segmentation and operation ability for internal monomer objects, limiting its convenience and practicality.

[0004] Therefore, how to realize the monomerization processing of objects in the three-dimensional model has become a technical problem that technicians in the field need to solve. SUMMARY

[0005] In view of the above problems, the present application provides a 3D mask construction method, device, storage medium and electronic device which overcome the above problems or at least partially solve the above problems, and the technical solutions are as follows:

[0006] A 3D mask construction method comprises:

[0007] obtaining a target monocular image and user input prompt information, wherein the target monocular image is a 2D image of a multi-view image at any specified view angle;

[0008] inputting the target monocular image and the user input prompt information into a preset image segmentation model to obtain a first 2D mask corresponding to the target monocular image output by the preset image segmentation model, wherein the preset image segmentation model caches feature information of the multi-view image;

[0009] performing cross-dimension mask conversion on the first 2D mask to obtain a first 3D mask;

[0010] inputting the first 3D mask into a pre-trained 3D Gaussian model to obtain a second 2D mask output by the 3D Gaussian model, wherein the 3D Gaussian model is trained based on the multi-view image;

[0011] fusing the second 2D mask and the feature information of the multi-view image to obtain comprehensive feature information;

[0012] inputting the comprehensive feature information into a preset intelligent prompt generation network, to obtain two-dimensional mask prompt information output by the preset intelligent prompt generation network;

[0013] inputting the two-dimensional mask prompt information, the second 2D mask, and the comprehensive feature information into the preset image segmentation model, to obtain a third 2D mask corresponding to the target single-view image and output by the preset image segmentation model;

[0014] performing cross-dimension mask conversion on the third 2D mask, to obtain a second 3D mask;

[0015] determining whether an iteration termination condition is reached, and if not, taking the mask obtained by fusing the second 3D mask and the first 3D mask as the first 3D mask, and returning to the step of inputting the first 3D mask into the pre-trained 3D Gaussian model to obtain the second 2D mask output by the 3D Gaussian model.

[0016] Optionally, the 3D mask is a three-dimensional mask composed of a plurality of three-dimensional Gaussian distributions, the three-dimensional Gaussian distribution includes a spatial position, a covariance, a color, an opacity, and a mask confidence score, and the mask confidence score is calculated by updating an intersection weight and a confidence of a light sampling point and a Gaussian distribution.

[0017] Optionally, the process of cross-dimension mask conversion includes:

[0018] initializing three-dimensional Gaussian distributions, wherein each three-dimensional Gaussian distribution stores a current mask confidence score;

[0019] projecting a 2D mask to a three-dimensional space, and mapping a pixel point in the 2D mask to a sampling point in the three-dimensional space by using neural radiance field density distribution information;

[0020] respectively calculating an intersection degree of each sampling point and each three-dimensional Gaussian distribution, and calculating a contribution weight based on a distance and a Gaussian distribution variance;

[0021] updating the current mask confidence score of each three-dimensional Gaussian distribution by a gradient descent algorithm according to the contribution weight and the current mask confidence score;

[0022] calculating a comprehensive mask confidence score of each sampling point by weighted summation using the contribution weight and the updated current mask confidence score, to generate a 3D mask.

[0023] Optionally, the mask confidence score of each three-dimensional Gaussian distribution is updated by a gradient descent algorithm according to the contribution weight and the initial mask confidence score.

[0024] obtaining a rendered 2D mask using the contribution weight and the initial mask confidence score;

[0025] calculating a difference loss between the rendered 2D mask and an original 2D mask;

[0026] adjusting the mask confidence score of each of the three-dimensional Gaussian distribution by a gradient descent algorithm based on the difference loss.

[0027] Optionally, before the obtaining of the target monocular image and the user input prompt information, the method further comprises:

[0028] generating a sparse 3D point cloud of a target scene using the multi-view images;

[0029] initializing a plurality of three-dimensional Gaussian distributions based on the sparse 3D point cloud;

[0030] differentially rendering the three-dimensional Gaussian distribution to generate predicted images of a plurality of training views;

[0031] comparing the predicted images with corresponding real images in the multi-view images to calculate a rendering loss;

[0032] optimizing the three-dimensional Gaussian distribution by backpropagation based on the rendering loss, and applying a 3D smoothing filter and Gaussian parameter fusion to iteratively optimize the three-dimensional Gaussian distribution during the optimization process to obtain a trained 3D Gaussian model.

[0033] Optionally, the inputting of the comprehensive feature information into the preset intelligent prompt generation network to obtain the two-dimensional mask prompt information output by the preset intelligent prompt generation network comprises:

[0034] inputting the comprehensive feature information into the preset intelligent prompt generation network to generate two-dimensional mask prompt information through a fully connected layer and an activation function.

[0035] Optionally, the iteration termination condition comprises that the mask accuracy of the second 3D mask reaches a preset accuracy threshold or the number of iterations reaches a preset iteration threshold.

[0036] A 3D mask construction device comprises a monocular information obtaining unit, a first 2D mask obtaining unit, a first 3D mask obtaining unit, a second 2D mask obtaining unit, a comprehensive feature information obtaining unit, a two-dimensional mask prompt information obtaining unit, a third 2D mask obtaining unit, a second 3D mask obtaining unit, an iteration termination condition judging unit, and an iteration loop control unit,

[0037] The monoscopic information obtaining unit is configured to obtain a target monoscopic image and user input prompt information, wherein the target monoscopic image is a 2D image of the multi-view image at any designated view angle.

[0038] The first 2D mask obtaining unit is configured to input the target monoscopic image and the user input prompt information into a preset image segmentation model, and obtain a first 2D mask corresponding to the target monoscopic image output by the preset image segmentation model, wherein the preset image segmentation model caches feature information of the multi-view image.

[0039] The first 3D mask obtaining unit is configured to perform cross-dimension mask conversion on the first 2D mask to obtain a first 3D mask.

[0040] The second 2D mask obtaining unit is configured to input the first 3D mask into a pre-trained 3D Gaussian model to obtain a second 2D mask output by the 3D Gaussian model, wherein the 3D Gaussian model is trained based on the multi-view image.

[0041] The comprehensive feature information obtaining unit is configured to fuse the second 2D mask and the feature information of the multi-view image to obtain comprehensive feature information.

[0042] The two-dimensional mask prompt information obtaining unit is configured to input the comprehensive feature information into a preset intelligent prompt generation network to obtain two-dimensional mask prompt information output by the preset intelligent prompt generation network.

[0043] The third 2D mask obtaining unit is configured to input the two-dimensional mask prompt information, the second 2D mask and the comprehensive feature information into the preset image segmentation model to obtain a third 2D mask corresponding to the target monoscopic image output by the preset image segmentation model.

[0044] The second 3D mask obtaining unit is configured to perform cross-dimension mask conversion on the third 2D mask to obtain a second 3D mask.

[0045] The iteration termination condition judging unit is configured to judge whether an iteration termination condition is reached, and if not, trigger the iteration loop control unit.

[0046] The iteration loop control unit is configured to take the mask obtained by fusing the second 3D mask and the first 3D mask as a first 3D mask, and re-trigger the second 2D mask obtaining unit.

[0047] A computer readable storage medium having a program stored thereon, wherein the program is executed by a processor to implement the 3D mask construction method.

[0048] An electronic device includes at least one processor, and at least one memory connected with the processor via a bus; wherein the processor, the memory complete mutual communication through the bus; the processor is used to call the program instruction in the memory, to execute the 3D mask construction method.

[0049] By the above technical solution, the 3D mask construction method, device, storage medium and electronic device provided by the application obtain a target monocular image and user input prompt information, wherein the target monocular image is a 2D image of a multi-view image at any specified view angle; the target monocular image and the user input prompt information are input into a preset image segmentation model to obtain a first 2D mask corresponding to the target monocular image output by the preset image segmentation model, wherein the preset image segmentation model caches feature information of the multi-view image; the first 2D mask is subjected to cross-dimension mask conversion to obtain a first 3D mask; the first 3D mask is input into a pre-trained 3D Gaussian model to obtain a second 2D mask output by the 3D Gaussian model, wherein the 3D Gaussian model is trained based on the multi-view image; the second 2D mask and the feature information of the multi-view image are fused to obtain comprehensive feature information; the comprehensive feature information is input into a preset intelligent prompt generation network to obtain two-dimensional mask prompt information output by the preset intelligent prompt generation network; the two-dimensional mask prompt information, the second 2D mask and the comprehensive feature information are input into the preset image segmentation model to obtain a third 2D mask corresponding to the target monocular image output by the preset image segmentation model; the third 2D mask is subjected to cross-dimension mask conversion to obtain a second 3D mask; it is judged whether an iteration termination condition is reached, and if not, the mask obtained by fusing the second 3D mask and the first 3D mask is taken as the first 3D mask, and the step of inputting the first 3D mask into the pre-trained 3D Gaussian model to obtain the second 2D mask output by the 3D Gaussian model is returned. The application realizes accurate segmentation and extraction of a single object inside a three-dimensional model through multiple rounds of cross-dimension mask conversion and feature fusion optimization, thereby effectively realizing single-body processing of objects in the three-dimensional model and improving the operability and application value of the model.

[0050] The above description is only a summary of the technical solutions of the application. In order to more clearly understand the technical means of the application, the specific embodiments of the application can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the application more obvious and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0051] Various other advantages and benefits will become apparent to those of ordinary skill in the art, upon reading the following detailed description of the preferred embodiment. The accompanying drawings are included to provide a description of preferred embodiments, and are not intended to limit the scope of the application. Furthermore, the drawings are included to illustrate embodiments of the application to a person of ordinary skill in the art, and, in the drawings:

[0052] Figure 1 A flow diagram illustrating one embodiment of a 3D mask construction method provided by an embodiment of the application is shown;

[0053] Figure 2 A flow diagram illustrating a process of cross-dimension mask conversion provided by an embodiment of the application is shown;

[0054] Figure 3 A flow diagram illustrating another embodiment of a 3D mask construction method provided by an embodiment of the application is shown;

[0055] Figure 4 A logic block diagram illustrating one embodiment of a 3D mask construction method provided by an embodiment of the application is shown;

[0056] Figure 5 A structural diagram illustrating a 3D mask construction apparatus provided by an embodiment of the application is shown;

[0057] Figure 6 A structural diagram illustrating an electronic device provided by an embodiment of the application is shown. DETAILED DESCRIPTION

[0058] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0059] With the development of computer vision and three-dimensional reconstruction technology, three-dimensional real scene reconstruction has been widely applied in urban modeling, cultural relic protection, virtual reality, etc. Existing three-dimensional modeling methods mainly include manual modeling and automatic modeling based on oblique photography.

[0060] The manual modeling method usually relies on professional three-dimensional designers to manually build scene structures using modeling software. Although this method has certain advantages in model fineness and customization, the modeling process is extremely tedious, time-consuming, and labor-intensive, making it difficult to apply to large-scale or complex scene modeling requirements, and only suitable for small-scale and simple scene application scenarios.

[0061] Oblique photogrammetry modeling involves capturing multi-view 2D images of a target scene from multiple angles using aerial photography, and then generating a scene model using 3D reconstruction algorithms. While this method improves modeling efficiency and automation to some extent, it requires a high level of expertise in data acquisition and is susceptible to occlusion, leading to incomplete reconstructed models. Furthermore, oblique photogrammetry has limited ability to model small, complex objects within a scene, making it difficult to meet the demands of refined modeling.

[0062] In recent years, the 3D Gaussian Splatting method has achieved efficient representation of scene details by distributing a large number of Gaussian points in 3D space. It can generate realistically lit and richly detailed 3D models and supports rendering from any viewpoint. Compared with traditional methods, 3D Gaussian Splatting has significant advantages in visual effects and rendering efficiency. However, the 3D models generated by this method are often presented as a whole, lacking the ability to individually identify and segment objects within the scene. Users find it difficult to directly manipulate individual objects within the model when editing, analyzing, or managing it. This limitation significantly affects the practicality and operability of 3D Gaussian Splatting technology, becoming one of the main obstacles to its further promotion and application.

[0063] Based on this, this invention provides a 3D mask construction method. First, by combining the target single-view image with user input prompts, a preliminary 2D mask is obtained using a cached image segmentation model with multi-view features. This preliminary 2D mask is then generated through cross-dimensional transformation. Next, the 3D mask is coupled with a 3D Gaussian model trained on multi-view images for further optimization to obtain a more accurate 2D mask. Finally, feature information and prompts from an intelligent prompt generation network are fused to iteratively improve the segmentation effect. Thus, this invention achieves accurate segmentation and extraction of individual objects within a 3D model through multiple rounds of cross-dimensional mask transformation and feature fusion optimization, effectively realizing the individualization of objects in a 3D model and enhancing the model's operability and application value.

[0064] like Figure 1 The diagram shows a flowchart of one embodiment of the 3D mask construction method provided by this invention. The method may include:

[0065] S100. Obtain the target single-view image and user input prompt information, wherein the target single-view image is a 2D image of the multi-view image under any specified view.

[0066] Among them, the target single-view image refers to a two-dimensional image obtained from multi-view image data at any specified viewpoint, which is used for subsequent segmentation and mask generation operations for that specific viewpoint.

[0067] The user input prompt information refers to auxiliary information provided by the user through interaction, usually a coordinate point, a framed area or a text instruction on a two-dimensional image, to guide the image segmentation model to identify the target object.

[0068] The multi-view image refers to a set of two-dimensional images of the same scene or object taken from different angles and positions.

[0069] The 2D (Two-Dimensional) image refers to an image on a two-dimensional plane, usually a photo or a rendering image under a single view, used for image analysis and mask generation.

[0070] Specifically, the embodiment of the present application can select a user-specified view from a pre-collected multi-view image set, extract the two-dimensional image corresponding to the view as the target single-view image, and receive the user input prompt information through the user interaction interface, which can include the target object position coordinates, the framed area or the labeled points clicked by the user, to assist subsequent image segmentation and target identification.

[0071] S110, input the target single-view image and the user input prompt information into the preset image segmentation model to obtain the first 2D mask corresponding to the target single-view image output by the preset image segmentation model, wherein the preset image segmentation model caches the feature information of the multi-view image.

[0072] The preset image segmentation model refers to a pre-trained deep learning model (such as FastSAM (Fast Segment Anything)), which can automatically generate a corresponding target region mask based on the input image and prompt information, and cache the feature information of the multi-view image to improve the segmentation effect.

[0073] The first 2D mask refers to the first round two-dimensional mask generated by the preset image segmentation model in combination with the target single-view image and the user prompt information, which identifies the pixel region of the target object in the image.

[0074] The feature information of the multi-view image refers to the deep learning feature vector extracted from the multi-view image by a feature extraction encoder (such as a FastSAM encoder), which contains information such as texture, edge and color of the image, and is used to assist subsequent segmentation and prompt generation.

[0075] Specifically, the embodiment of the present application can input the target monocular image and the two-dimensional prompt information (such as the coordinate point of the target object) input by the user into the preset image segmentation model FastSAM; the model uses the image feature information extracted from the multi-view image training data in advance and combined with the input prompt information to accurately analyze and segment the target monocular image, thereby generating a first 2D mask corresponding to the view angle, effectively identifying the pixel region of the target object and providing a basis for subsequent three-dimensional mask construction.

[0076] S120, cross-dimension mask conversion is performed on the first 2D mask to obtain a first 3D mask.

[0077] The cross-dimension mask conversion refers to a process of mapping two-dimensional mask information to three-dimensional space to form a three-dimensional mask by using Gaussian distribution of three-dimensional space and NeRF technology.

[0078] The first 3D (Three-Dimensional) mask refers to a three-dimensional mask generated by cross-dimension mask conversion of the first 2D mask, representing the voxels or regions of the target object in the three-dimensional space, usually composed of multiple 3D Gaussian distribution bodies, with a confidence score.

[0079] Optionally, the 3D mask can be a three-dimensional mask composed of multiple three-dimensional Gaussian distributions, which can include spatial position, covariance, color, opacity and mask confidence score. The mask confidence score can be calculated by updating the intersection weight and confidence of the ray sampling point and the Gaussian distribution.

[0080] Specifically, the embodiment of the present application can project the two-dimensional mask to the three-dimensional space and map the sampling points by using the neural radiation field density information, combine the pre-initialized three-dimensional Gaussian distribution and its mask confidence score, update the confidence score based on the intersection degree and contribution weight of the sampling points and the Gaussian distribution by using gradient descent optimization, and finally generate a three-dimensional mask by weighted fusion.

[0081] S130, input the first 3D mask into the pre-trained 3D Gaussian model to obtain a second 2D mask output by the 3D Gaussian model, wherein the 3D Gaussian model is trained based on multi-view images.

[0082] The 3D Gaussian model refers to a three-dimensional Gaussian distribution model trained based on multi-view images, which uses Gaussian points to represent the light, color and spatial information in the scene, supports rendering and mask generation, such as 3D Gaussian Splatting.

[0083] The second 2D mask refers to a two-dimensional mask in a new view generated by inputting the first 3D mask into a 3D Gaussian model in combination with the rendering capability of the model, and is a projection representation of the target object in another view.

[0084] Specifically, the embodiment of the present application can input the first 3D mask into a 3D Gaussian model that has been pre-trained based on multi-view image data. The 3D Gaussian model uses the multi-view spatial structure and object distribution features learned in the training stage to perform probability modeling and projection calculation of the input 3D mask in three-dimensional space. The 3D Gaussian model projects the three-dimensional mask onto the corresponding two-dimensional plane according to the set target view parameter, thereby outputting the second 2D mask corresponding to the target view, and realizing more accurate two-dimensional mask generation.

[0085] To realize high-precision three-dimensional mask construction, the 3D Gaussian model uses its neural radiance field capability to predict the attributes such as illumination and color of each pixel point in the target new view, and then combines the input three-dimensional mask to extract the pixel points belonging to the mask region and remove or set the background region pixel points to transparent, finally generates a two-dimensional mask consistent with the three-dimensional mask in the new view, realizing high-quality mask conversion from three-dimensional to two-dimensional. This process can be iterated to automatically generate two-dimensional masks in different views.

[0086] S140, fuse the second 2D mask and the feature information of the multi-view image to obtain comprehensive feature information.

[0087] The comprehensive feature information refers to the feature representation obtained by fusing the second 2D mask and the feature information of the multi-view image, which contains image texture, spatial structure and target region information, and is used by the intelligent prompt generation network.

[0088] Specifically, the embodiment of the present application can use the second 2D mask as spatial constraint information, and align it with the feature information pre-extracted and cached in the multi-view image in the corresponding view. The spatial position information of the mask region and the deep features of the image are fused by using convolution operation or attention mechanism, and the feature expression of the target region is strengthened. In the fusion process, methods such as weighted superposition or feature splicing can be used to effectively combine the spatial structure information of the mask and the rich semantic features of the multi-view image, and finally generate comprehensive feature information containing the target object more complete and detailed, which is used for subsequent analysis and processing.

[0089] As some examples, the embodiment of the present application can align and splice the spatial region features corresponding to the second 2D mask with the multi-view feature vectors pre-extracted and cached from the multi-view images, then input the spliced feature sequence into the constructed feature fusion network, which is based on the Transformer architecture, effectively captures the spatial correlation and context information between different view features through its self-attention mechanism, uniformly models the complete form and details of the target object, and finally outputs the fused comprehensive feature information. This comprehensive feature not only contains the local segmentation details of the second 2D mask, but also integrates the global spatial information provided by the multi-view, providing rich and consistent semantic representation for subsequent intelligent prompt generation and mask optimization.

[0090] S150, input the comprehensive feature information into the preset intelligent prompt generation network to obtain the two-dimensional mask prompt information output by the preset intelligent prompt generation network.

[0091] Among them, the preset intelligent prompt generation network refers to a network model based on deep learning, which inputs the fused comprehensive feature information to automatically generate two-dimensional mask prompt information (such as prompt point coordinates) to assist the image segmentation model to enhance the segmentation accuracy.

[0092] Among them, the two-dimensional mask prompt information refers to the auxiliary information output by the intelligent prompt generation network, which is usually the key points or coordinates of the target object in the two-dimensional image, guiding the image segmentation model to more accurately identify the target region.

[0093] Specifically, the embodiment of the present application can input the comprehensive feature information into the preset intelligent prompt generation network, reduce the dimension through the full connection layer and the activation function, and generate the two-dimensional mask prompt information.

[0094] Further, in the specific implementation process of inputting the comprehensive feature information into the preset intelligent prompt generation network, the embodiment of the present application first inputs the comprehensive feature with a dimension of D into the input layer, and processes it through the full connection transformation of the first hidden layer and the ReLU activation function, reducing the feature dimension from D to H, in order to extract more key features and reduce the calculation complexity, the formula is H1 = ReLU(W1x + b1), wherein The activation function is a nonlinear function; x is the input feature, with a dimension of D; W1 is the weight matrix of the first hidden layer, with a dimension of [H, D]; b1 is the bias vector of the first hidden layer, with a dimension of [H]. (The hidden layer can be designed in multiple layers according to the complexity of the scene)); Then, the output H1 of the first hidden layer is converted into an output vector O with a dimension of 2 through the full connection mapping of the output layer, O = W2H1 + b2, which represents the prompt point coordinate information of the two-dimensional mask, wherein H1 is the output of the first hidden layer, with a dimension of H; is the weight matrix of the output layer, with dimensions [2, H]; is the bias vector of the output layer, with dimensions [2]. Through this network structure, the fused comprehensive features can be effectively compressed and converted into accurate two-dimensional hint coordinates, assisting the generation and optimization of subsequent masks.

[0095] S160, input the two-dimensional mask hint information, the second 2D mask and the comprehensive feature information into the preset image segmentation model, and obtain the third 2D mask corresponding to the target single-view image output by the preset image segmentation model.

[0096] Among them, the third 2D mask refers to the updated two-dimensional mask generated by combining the two-dimensional mask hint information, the second 2D mask and the comprehensive feature information through the preset image segmentation model, which has higher accuracy and robustness.

[0097] Specifically, the embodiment of the application can input the two-dimensional mask hint information, the newly generated second 2D mask and the cached comprehensive feature information as input into the preset image segmentation model FastSAM. The model uses its powerful segmentation capability to combine the input spatial mask hint and rich image features, performs multi-scale feature fusion and context analysis, accurately captures the boundary and morphological features of the target object, and then generates the third 2D mask corresponding to the target single-view image, improves the accuracy and completeness of segmentation, and provides more detailed mask results for subsequent tasks.

[0098] S170, cross-dimension mask conversion is performed on the third 2D mask to obtain a second 3D mask.

[0099] Among them, the second 3D mask refers to the latest three-dimensional mask generated by cross-dimension mask conversion from the third 2D mask, which is fused with the previous 3D mask to continuously optimize the segmentation result of the three-dimensional target object.

[0100] Step S170 can refer to the description at step S120, which will not be repeated here.

[0101] S180, if not, step S190 is executed.

[0102] Among them, the iteration termination condition refers to the standard for judging whether the iteration process is ended, and the iteration is stopped when the condition is met, and the final 3D mask is output. Optionally, the iteration termination condition includes that the mask accuracy of the second 3D mask reaches a preset accuracy threshold or the number of iterations reaches a preset iteration threshold.

[0103] S190, the mask obtained by fusing the second 3D mask and the first 3D mask is taken as the first 3D mask, and step S130 is returned.

[0104] The application provides a 3D mask construction method, which comprises the following steps: obtaining a target single-view image and user input prompt information, wherein the target single-view image is a 2D image of a multi-view image at any specified view angle; inputting the target single-view image and the user input prompt information into a preset image segmentation model to obtain a first 2D mask corresponding to the target single-view image output by the preset image segmentation model, wherein the preset image segmentation model caches feature information of the multi-view image; performing cross-dimension mask conversion on the first 2D mask to obtain a first 3D mask; inputting the first 3D mask into a pre-trained 3D Gaussian model to obtain a second 2D mask output by the 3D Gaussian model, wherein the 3D Gaussian model is trained based on the multi-view image; fusing the second 2D mask and the feature information of the multi-view image to obtain comprehensive feature information; inputting the comprehensive feature information into a preset intelligent prompt generation network to obtain two-dimensional mask prompt information output by the preset intelligent prompt generation network; inputting the two-dimensional mask prompt information, the second 2D mask and the comprehensive feature information into the preset image segmentation model to obtain a third 2D mask corresponding to the target single-view image output by the preset image segmentation model; performing cross-dimension mask conversion on the third 2D mask to obtain a second 3D mask; determining whether an iteration termination condition is reached, and if not, taking the mask obtained by fusing the second 3D mask and the first 3D mask as the first 3D mask and returning to the step of inputting the first 3D mask into the pre-trained 3D Gaussian model to obtain the second 2D mask output by the 3D Gaussian model. Through multiple rounds of cross-dimension mask conversion and feature fusion optimization, the application realizes accurate segmentation and extraction of a single object inside a three-dimensional model, thereby effectively realizing single-body processing of objects in the three-dimensional model and improving the operability and application value of the model.

[0105] Optionally, based on Figure 1 As shown in the method, Figure 2 As shown in the process flow diagram of the cross-dimension mask conversion process provided by the embodiment of the application, the cross-dimension mask conversion process can comprise:

[0106] S200, initializing a three-dimensional Gaussian distribution, wherein each three-dimensional Gaussian distribution stores a current mask confidence score.

[0107] The three-dimensional Gaussian distribution refers to a probability distribution defined in the form of a multivariate Gaussian function in a three-dimensional space, which is used to describe the attribute characteristics of a region in the space. Each three-dimensional Gaussian distribution represents the spatial characteristics and visual attributes of the region through a mean vector (representing the center position), a covariance matrix (representing the shape and diffusion direction), color, opacity and other information. In the present scheme, the three-dimensional Gaussian distribution is used as a basic unit for constructing a three-dimensional mask, representing a local region of a target object in the space.

[0108] The current mask confidence score refers to the probability estimation value of each three-dimensional Gaussian distribution at the current iteration or training state that the three-dimensional space region represented by the three-dimensional Gaussian distribution belongs to the target mask. The score is initialized and continuously adjusted by an optimization algorithm to more accurately reflect the confidence of the region belonging to the target object.

[0109] Specifically, the embodiment of the present application can define a plurality of three-dimensional Gaussian distributions in three-dimensional space in advance. Each Gaussian distribution determines the center position by a mean vector, determines the spatial shape and diffusion range by a covariance matrix, and is accompanied by attribute information such as color and opacity. For each Gaussian distribution, initialize its current mask confidence score, which represents the initial probability of the space region belonging to the target mask. Uniform assignment or setting according to prior information can be used. All Gaussian distributions constitute the initial three-dimensional mask representation basis.

[0110] S210, project the 2D mask into three-dimensional space, and use the NeRF density distribution information to map the pixel points in the 2D mask to the sampling points in the three-dimensional space.

[0111] The sampling point refers to a specific spatial position in the three-dimensional space that is uniformly or according to a certain strategy selected along the ray from the camera center passing through the pixel point in the two-dimensional mask during the projection of the two-dimensional mask into the three-dimensional space. Each sampling point is used to evaluate the intersection degree with the three-dimensional Gaussian distribution and the mask confidence contribution.

[0112] Specifically, the embodiment of the present application can emit a ray from the camera center of the corresponding view angle for each pixel point in the two-dimensional mask, and determine the direction of the ray through the camera projection model. Along this light direction, combined with the three-dimensional density distribution information learned by NeRF, a plurality of three-dimensional points are uniformly or according to the density change sampled on the light. In this way, the pixel points in the two-dimensional mask are mapped to a plurality of sampling points in the three-dimensional space, reflecting the possible position of the pixel in space, forming a set of three-dimensional sampling points.

[0113] S220, respectively calculate the intersection degree of each sampling point with each three-dimensional Gaussian distribution, and calculate the contribution weight based on the distance and the variance of the Gaussian distribution.

[0114] The contribution weight represents the influence degree of a certain three-dimensional Gaussian distribution on the mask confidence score of a specific sampling point. The contribution weight is usually calculated based on the distance between the sampling point and the center of the Gaussian distribution, the variance of the Gaussian distribution, etc. reflects the confidence contribution proportion of the Gaussian distribution in the space covered by the sampling point.

[0115] Specifically, the embodiment of the present application can calculate the spatial distance of each sampling point from all three-dimensional Gaussian distributions, evaluate the probability density value of the sampling point falling into each Gaussian distribution as a quantitative indicator of the degree of intersection by using the kernel function formula of the Gaussian distribution. Based on the probability density value and the variance parameter of each Gaussian distribution, the contribution weight is calculated, which is used to reflect the contribution proportion of each Gaussian distribution to the mask confidence score of the sampling point.

[0116] S230, updating the current mask confidence score of each three-dimensional Gaussian distribution by a gradient descent algorithm according to the contribution weight and the current mask confidence score.

[0117] Specifically, the embodiment of the present application can obtain a rendered 2D mask by using the contribution weight and the initial mask confidence score; calculate the difference loss between the rendered 2D mask and the original 2D mask; and adjust the mask confidence score of each three-dimensional Gaussian distribution by a gradient descent algorithm based on the difference loss.

[0118] Further, the embodiment of the present application can perform weighted summation on the confidence scores of the sampling points by using the current mask confidence score and the contribution weight, and render a two-dimensional mask prediction under the corresponding view angle. The rendered mask and the original two-dimensional mask are compared for difference, and a loss function (such as cross-entropy or IoU loss) is defined to measure the gap between them. The gradient of the loss with respect to the confidence score of each Gaussian distribution is calculated by using back propagation, and the gradient descent algorithm is applied to iteratively update the current mask confidence score, so that it gradually approaches the confidence distribution of the real mask, thereby improving the accuracy of the three-dimensional mask.

[0119] S240, weighted summation is performed on the comprehensive mask confidence score of each sampling point by using the contribution weight and the updated current mask confidence score, and a 3D mask is generated.

[0120] The comprehensive mask confidence score refers to the final mask confidence probability value obtained by weighted summation of the contribution weight and the respective mask confidence score of all three-dimensional Gaussian distributions intersecting with the sampling point. The score comprehensively reflects the overall confidence of the sampling point belonging to the target mask in the three-dimensional space, and is used to construct a complete three-dimensional mask representation.

[0121] Specifically, the embodiment of the present application can combine the updated current mask confidence score and the corresponding contribution weight, and perform weighted summation on each sampling point to obtain the comprehensive mask confidence score of the sampling point, which represents the probability of the point belonging to the target mask. The comprehensive confidence scores of all sampling points are summarized in the three-dimensional space to form a three-dimensional mask representation reflecting the spatial distribution and morphological structure of the target object.

[0122] The embodiment of the present application can effectively map and optimize the two-dimensional mask and the three-dimensional space, enhance the three-dimensional space consistency and accuracy of the mask, improve the utilization efficiency of multi-view image information, effectively promote the collaborative optimization from two-dimensional segmentation to three-dimensional reconstruction, significantly improve the fineness and robustness of target segmentation, and meet the needs of high-quality three-dimensional mask construction in complex scenes.

[0123] Optionally, based on Figure 1 As shown in the method, Figure 3 As shown in the flowchart of another embodiment of the 3D mask construction method provided by the embodiment of the present application, before step S100, the method can further include:

[0124] S300, generating a sparse 3D point cloud of the target scene by using multi-view images.

[0125] Specifically, the embodiment of the present application can collect the target scene by using multi-view images, use the Structure from Motion (SfM) technology, first extract and match features from the multi-view images, gradually recover the camera pose and the spatial coordinates of the scene points through the incremental reconstruction process, and perform global optimization through bundle adjustment, and finally output the sparse 3D point cloud of the scene and the camera parameters of each view. The point cloud contains the three-dimensional position information of the key feature points in the scene, and provides a spatial reference for subsequent three-dimensional Gaussian distribution initialization.

[0126] S310, initializing a plurality of three-dimensional Gaussian distributions based on the sparse 3D point cloud.

[0127] Specifically, the embodiment of the present application can initialize a corresponding three-dimensional Gaussian distribution with each point in the sparse 3D point cloud as the center. Each Gaussian distribution is defined by parameters such as mean, covariance matrix, and initial weight, covers the key area of the target scene, and ensures that the Gaussian distribution can effectively express the dense area of the point cloud in the three-dimensional space and its spatial relationship.

[0128] S320, differentiable rendering is performed on the three-dimensional Gaussian distribution to generate predicted images of a plurality of training views.

[0129] Specifically, the embodiment of the present application can use the differentiable rendering technology to project the current three-dimensional Gaussian distribution model into a plurality of views used during training. The camera parameters corresponding to the view are calculated, the Gaussian distribution is integrated and sampled along the sight line, the color and density information of each pixel under the view are obtained, and finally the predicted image is rendered, so as to maintain the differentiable characteristic, and facilitate subsequent adjustment of the Gaussian distribution parameters through gradient transmission.

[0130] S330, comparing the predicted image with the corresponding real image in the multi-view image to calculate the rendering loss.

[0131] Specifically, the embodiment of the present application can compare the predicted image of each generated training view with the real multi-view image of the corresponding view pixel by pixel. The difference between the predicted image and the real image is quantified by using mean square error, structural similarity index or other perceptual loss functions to obtain a rendering loss value, which reflects the fitting degree of the current three-dimensional Gaussian distribution model to the scene lighting and texture.

[0132] S340, based on the rendering loss, the three-dimensional Gaussian distribution is optimized by back propagation, and the three-dimensional Gaussian distribution is iteratively optimized by applying a 3D smoothing filter and Gaussian parameter fusion in the optimization process to obtain a trained 3D Gaussian model.

[0133] Specifically, the embodiment of the present application can use the calculated rendering loss to calculate the gradient of the loss to the three-dimensional Gaussian distribution parameters (mean, covariance, weight, etc.) by using a back propagation algorithm, and then use an optimizer (such as Adam) to update these parameters. After each parameter update, the Gaussian distribution parameters are smoothed by applying a 3D smoothing filter to reduce noise and discontinuity, and a Gaussian parameter fusion technique is used to reasonably merge or adjust adjacent Gaussian distributions to improve the continuity and rendering quality of the model. After multiple iterations, the model parameters gradually converge, and finally a trained 3D Gaussian model with compact structure, no jaggedness and strong realism is obtained.

[0134] Before obtaining the target monocular image and the user input prompt information, the embodiment of the present application first generates a sparse 3D point cloud of the target scene using multi-view images, and initializes multiple three-dimensional Gaussian distributions based on the point cloud. Then, the predicted image of the training view is generated by differentiable rendering, which is compared with the real multi-view image to calculate the rendering loss, and the three-dimensional Gaussian distribution is optimized based on the loss by back propagation, and iteratively optimized by combining the 3D smoothing filter and the Gaussian parameter fusion technology, which can effectively train a high-quality 3D Gaussian model without jaggedness and artifacts.

[0135] In order to facilitate understanding of the overall technical process of the embodiment of the present application, the following Figure 4 are used as examples for illustration: Figure 4The diagram illustrates a logical block diagram of a 3D mask construction method provided in this embodiment of the invention. During the training phase, a 3D Gaussian Splatting model is first trained using multi-view image data to construct a high-quality 3D scene radiation field. A 3D smoothing filter and Gaussian parameter fusion technique are then used to generate an anti-aliasing 3D Gaussian Splatting model, providing data support for subsequent 3D mask generation of new-view 2D masks and ensuring the model is free of artifacts and jagged edges during later mask generation. Subsequently, a FastSAM (Fast Segment Anything) encoder is used to extract features from the training images and establish a feature cache. Based on user-input 2D image prompts (such as coordinates), an initial 2D mask for the selected viewpoint is generated using FastSAM, and the 2D mask is converted to an initial 3D mask using a cross-dimensional mask transformation method. After entering the alternating iteration process, a new-view 2D mask is generated using an inaccurate 3D mask, and AI is used to automatically obtain prompts for the new-view 2D mask by fusing multi-view information. The new perspective 2D mask hints, the new perspective 2D mask, and cached feature information are input into FastSAM to generate an updated 2D mask. This updated 2D mask is then converted into a new 3D mask using a cross-dimensional mask transformation method and merged with the cached 3D mask. This process is repeated until the accuracy of the 3D mask reaches a preset threshold or the number of iterations reaches its maximum, ultimately yielding a relatively accurate 3D mask.

[0136] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0137] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0138] Corresponding to the above method embodiments, this invention also provides a 3D mask construction apparatus, the structure of which is as follows: Figure 5 As shown, it may include: a single-view information acquisition unit 10, a first 2D mask acquisition unit 11, a first 3D mask acquisition unit 12, a second 2D mask acquisition unit 13, a comprehensive feature information acquisition unit 14, a two-dimensional mask prompt information acquisition unit 15, a third 2D mask acquisition unit 16, a second 3D mask acquisition unit 17, an iteration termination condition judgment unit 18, and an iteration loop control unit 19.

[0139] The single-view information obtaining unit 10 is configured to obtain a target single-view image and user input prompt information, wherein the target single-view image is a 2D image of the multi-view image at any specified view angle.

[0140] The first 2D mask obtaining unit 11 is configured to input the target single-view image and the user input prompt information into a preset image segmentation model, and obtain a first 2D mask corresponding to the target single-view image output by the preset image segmentation model, wherein the preset image segmentation model is cached with feature information of the multi-view image.

[0141] The first 3D mask obtaining unit 12 is configured to perform cross-dimension mask conversion on the first 2D mask to obtain a first 3D mask.

[0142] The second 2D mask obtaining unit 13 is configured to input the first 3D mask into a pre-trained 3D Gaussian model to obtain a second 2D mask output by the 3D Gaussian model, wherein the 3D Gaussian model is trained based on the multi-view image.

[0143] The comprehensive feature information obtaining unit 14 is configured to fuse the second 2D mask and the feature information of the multi-view image to obtain comprehensive feature information.

[0144] The two-dimensional mask prompt information obtaining unit 15 is configured to input the comprehensive feature information into a preset intelligent prompt generation network to obtain two-dimensional mask prompt information output by the preset intelligent prompt generation network.

[0145] The third 2D mask obtaining unit 16 is configured to input the two-dimensional mask prompt information, the second 2D mask and the comprehensive feature information into the preset image segmentation model to obtain a third 2D mask corresponding to the target single-view image output by the preset image segmentation model.

[0146] The second 3D mask obtaining unit 17 is configured to perform cross-dimension mask conversion on the third 2D mask to obtain a second 3D mask.

[0147] The iteration termination condition judging unit 18 is configured to judge whether an iteration termination condition is reached, and if not, trigger the iteration loop control unit 19.

[0148] The iteration loop control unit 19 is configured to take the mask obtained by fusing the second 3D mask and the first 3D mask as the first 3D mask, and re-trigger the second 2D mask obtaining unit 13.

[0149] Optionally, the 3D mask is a three-dimensional mask composed of a plurality of three-dimensional Gaussian distributions, the three-dimensional Gaussian distribution includes a spatial position, a covariance, a color, an opacity and a mask confidence score, and the mask confidence score is calculated by updating the intersection weight and the confidence of the light sampling point and the Gaussian distribution.

[0150] Optionally, the 3D mask is a three-dimensional mask composed of a plurality of three-dimensional Gaussian distributions, the three-dimensional Gaussian distribution includes a spatial position, a covariance, a color, an opacity, and a mask confidence score, and the mask confidence score is calculated by updating an intersection weight and a confidence of a light sampling point and the Gaussian distribution.

[0151] Optionally, the first 3D mask obtaining unit 12 and / or the second 3D mask obtaining unit 17 can be specifically used in a cross-dimension mask conversion process: initializing a three-dimensional Gaussian distribution, wherein each three-dimensional Gaussian distribution stores a current mask confidence score; projecting a 2D mask to a three-dimensional space, mapping a pixel point in the 2D mask to a sampling point in the three-dimensional space by using a neural radiance field density distribution information; calculating an intersection degree of each sampling point and each three-dimensional Gaussian distribution respectively, and calculating a contribution weight based on a distance and a Gaussian distribution variance; updating the current mask confidence score of each three-dimensional Gaussian distribution by a gradient descent algorithm according to the contribution weight and the current mask confidence score; and calculating a comprehensive mask confidence score of each sampling point by weighted summation using the contribution weight and the updated current mask confidence score, to generate a 3D mask.

[0152] Optionally, the first 3D mask obtaining unit 12 and / or the second 3D mask obtaining unit 17 can also be specifically used to obtain a rendered 2D mask by using the contribution weight and the initial mask confidence score; calculate a difference loss between the rendered 2D mask and the original 2D mask; and adjust the mask confidence score of each three-dimensional Gaussian distribution by a gradient descent algorithm based on the difference loss.

[0153] Optionally, the 3D mask construction device can further include a 3D Gaussian model training unit.

[0154] The 3D Gaussian model training unit is configured to, before the single-view information obtaining unit 10 obtains the target single-view image and the user input prompt information, generate a sparse 3D point cloud of a target scene by using multi-view images; initialize a plurality of three-dimensional Gaussian distributions based on the sparse 3D point cloud; perform differentiable rendering on the three-dimensional Gaussian distributions to generate predicted images of a plurality of training views; compare the predicted images with corresponding real images in the multi-view images to calculate a rendering loss; and optimize the three-dimensional Gaussian distributions by back propagation based on the rendering loss, and iteratively optimize the three-dimensional Gaussian distributions by applying a 3D smoothing filter and Gaussian parameters fusion in the optimization process to obtain a trained 3D Gaussian model.

[0155] Optionally, the two-dimensional mask prompt information obtaining unit 15 can be specifically used to input the comprehensive feature information into a preset intelligent prompt generation network, and generate two-dimensional mask prompt information by dimension reduction through a full connection layer and an activation function.

[0156] Optionally, the iteration termination condition comprises that a mask precision of the second 3D mask reaches a preset precision threshold or an iteration number reaches a preset iteration threshold.

[0157] The 3D mask construction device provided by the application comprises the following steps: obtaining a target monoscopic image and user input prompt information, wherein the target monoscopic image is a 2D image of a multi-view image at any specified view angle; inputting the target monoscopic image and the user input prompt information into a preset image segmentation model to obtain a first 2D mask corresponding to the target monoscopic image output by the preset image segmentation model, wherein the preset image segmentation model caches feature information of the multi-view image; performing cross-dimension mask conversion on the first 2D mask to obtain a first 3D mask; inputting the first 3D mask into a pre-trained 3D Gaussian model to obtain a second 2D mask output by the 3D Gaussian model, wherein the 3D Gaussian model is trained based on the multi-view image; fusing the second 2D mask and the feature information of the multi-view image to obtain comprehensive feature information; inputting the comprehensive feature information into a preset intelligent prompt generation network to obtain two-dimensional mask prompt information output by the preset intelligent prompt generation network; inputting the two-dimensional mask prompt information, the second 2D mask and the comprehensive feature information into the preset image segmentation model to obtain a third 2D mask corresponding to the target monoscopic image output by the preset image segmentation model; performing cross-dimension mask conversion on the third 2D mask to obtain a second 3D mask; and judging whether an iteration termination condition is reached, if not, taking a mask obtained by fusing the second 3D mask and the first 3D mask as the first 3D mask, and returning to the step of inputting the first 3D mask into the pre-trained 3D Gaussian model to obtain the second 2D mask output by the 3D Gaussian model. Through multiple rounds of cross-dimension mask conversion and feature fusion optimization, the application realizes accurate segmentation and extraction of a single object inside a three-dimensional model, thereby effectively realizing single-body processing of objects in the three-dimensional model and improving the operability and application value of the model.

[0158] As to the device in the above-mentioned embodiments, the specific manners in which various units perform operations have been described in detail in the embodiments relating to the method, and thus will not be described in detail here.

[0159] The 3D mask construction device comprises a processor and a memory, and the above-mentioned monoscopic information obtaining unit 10, first 2D mask obtaining unit 11, first 3D mask obtaining unit 12, second 2D mask obtaining unit 13, comprehensive feature information obtaining unit 14, two-dimensional mask prompt information obtaining unit 15, third 2D mask obtaining unit 16, second 3D mask obtaining unit 17, iteration termination condition judging unit 18 and iteration loop control unit 19 are all stored in the memory as program units, and the corresponding functions are realized by the processor executing the above-mentioned program units stored in the memory.

[0160] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, multiple rounds of cross-dimensional mask transformation and feature fusion optimization can be performed to achieve accurate segmentation and extraction of individual objects within a 3D model. This effectively realizes the individualization of objects in the 3D model, improving the model's operability and application value.

[0161] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the 3D mask construction method.

[0162] This invention provides a processor for running a program, wherein the program executes the 3D mask construction method during runtime.

[0163] like Figure 6 As shown, this embodiment of the invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned 3D mask construction method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0164] The present invention also provides a computer program product that, when executed on an electronic device, is adapted to execute a program that initializes a 3D mask construction method step.

[0165] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0166] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0167] The memory can include non-persistent memory in the computer readable medium, random access memory (RAM), and / or non-volatile memory such as read only memory (ROM) or flash memory, among others, including at least one memory chip. The memory is an example of computer readable media.

[0168] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carrier waves.

[0169] In the description of the present application, it needs to be understood that if the orientation or positional relationship indicated by the terms such as "upper", "lower", "front", "back", "left" and "right" is based on the orientation or positional relationship shown in the drawings, it is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the position or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0170] It should be noted that in this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. It should be noted that the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, product or equipment. Without more limitation, the element defined by the sentence "including a…" does not exclude the presence of other identical elements in the process, method, product or equipment including the element.

[0171] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code thereon for use by or in connection with an instruction execution system. For the purposes of this description, a computer usable or computer readable storage medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) including a computer readable storage medium. Examples of a computer readable storage medium include an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) including a computer readable storage medium.

[0172] The foregoing is merely illustrative of the principles of the application, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the application. Any such modifications are intended to fall within the scope of the application.

Claims

1. A 3D mask construction method, characterized in that, include: Obtain a target single-view image and user input prompt information, wherein the target single-view image is a 2D image of a multi-view image at any specified viewpoint; The target single-view image and the user input prompt information are input into a preset image segmentation model to obtain a first 2D mask output by the preset image segmentation model corresponding to the target single-view image. The preset image segmentation model caches the feature information of the multi-view image. Perform cross-dimensional mask transformation on the first 2D mask to obtain the first 3D mask; The first 3D mask is input into a pre-trained 3D Gaussian model to obtain a second 2D mask output by the 3D Gaussian model, wherein the 3D Gaussian model is trained based on the multi-view image; The feature information of the second 2D mask and the multi-view image are fused to obtain comprehensive feature information; The comprehensive feature information is input into a preset intelligent prompt generation network to obtain the two-dimensional mask prompt information output by the preset intelligent prompt generation network; The two-dimensional mask prompt information, the second 2D mask, and the comprehensive feature information are input into the preset image segmentation model to obtain the third 2D mask output by the preset image segmentation model that corresponds to the target single-view image; Perform cross-dimensional mask transformation on the third 2D mask to obtain the second 3D mask; Determine whether the iteration termination condition has been met. If not, use the mask obtained by fusing the second 3D mask and the first 3D mask as the first 3D mask, and return to the step of inputting the first 3D mask into the pre-trained 3D Gaussian model to obtain the second 2D mask output by the 3D Gaussian model.

2. The method according to claim 1, characterized in that, The 3D mask is a three-dimensional mask composed of multiple three-dimensional Gaussian distributions. The three-dimensional Gaussian distributions include spatial location, covariance, color, opacity, and mask confidence score. The mask confidence score is calculated by updating the intersection weights and confidence scores of the light sampling points and the Gaussian distribution.

3. The method according to claim 2, characterized in that, The cross-dimensional masking process includes: Initialize a three-dimensional Gaussian distribution, wherein each of the three-dimensional Gaussian distributions stores the current mask confidence score; A 2D mask is projected onto a three-dimensional space, and the pixels in the 2D mask are mapped to sampling points in the three-dimensional space using the neural radiation field density distribution information. The degree of intersection between each sampling point and each of the three-dimensional Gaussian distributions is calculated, and the contribution weight is calculated based on the distance and the variance of the Gaussian distribution. Based on the contribution weights and the current mask confidence scores, the current mask confidence scores of each of the three-dimensional Gaussian distributions are updated using a gradient descent algorithm. Using the contribution weights and the updated current mask confidence score, a weighted summation is performed to calculate the comprehensive mask confidence score for each sampling point, thereby generating a 3D mask.

4. The method according to claim 3, characterized in that, The mask confidence scores of each of the three-dimensional Gaussian distributions are updated using a gradient descent algorithm based on the contribution weights and the initial mask confidence scores. The rendering 2D mask is obtained using the contribution weights and the initial mask confidence score. Calculate the difference loss between the rendered 2D mask and the original 2D mask; Based on the difference loss, the mask confidence scores of each of the three-dimensional Gaussian distributions are adjusted using the gradient descent algorithm.

5. The method according to claim 1, characterized in that, Before obtaining the target single-view image and user input prompts, the method further includes: The multi-view images are used to generate a sparse 3D point cloud of the target scene; Multiple three-dimensional Gaussian distributions were initialized based on the sparse 3D point cloud; Differentiable rendering is performed on the three-dimensional Gaussian distribution to generate prediction images from multiple training perspectives; The predicted image is compared with the corresponding real image in the multi-view image to calculate the rendering loss; Based on the rendering loss, the three-dimensional Gaussian distribution is optimized through backpropagation. During the optimization process, a 3D smoothing filter and Gaussian parameters are fused to iteratively optimize the three-dimensional Gaussian distribution, thereby obtaining the trained 3D Gaussian model.

6. The method according to claim 1, characterized in that, The step of inputting the comprehensive feature information into a preset intelligent prompt generation network to obtain the two-dimensional mask prompt information output by the preset intelligent prompt generation network includes: The comprehensive feature information is input into a preset intelligent prompt generation network, and dimensionality reduction is achieved through a fully connected layer and activation function to generate two-dimensional mask prompt information.

7. The method according to any one of claims 1 to 6, characterized in that, The iteration termination conditions include the mask accuracy of the second 3D mask reaching a preset accuracy threshold or the number of iterations reaching a preset iteration threshold.

8. A 3D mask construction apparatus, characterized in that, include: The system includes a single-view information acquisition unit, a first 2D mask acquisition unit, a first 3D mask acquisition unit, a second 2D mask acquisition unit, a comprehensive feature information acquisition unit, a two-dimensional mask prompt information acquisition unit, a third 2D mask acquisition unit, a second 3D mask acquisition unit, an iteration termination condition judgment unit, and an iteration loop control unit. The single-view information acquisition unit is used to acquire a target single-view image and user input prompt information, wherein the target single-view image is a 2D image of a multi-view image under any specified view. The first 2D mask acquisition unit is used to input the target single-view image and the user input prompt information into a preset image segmentation model to obtain the first 2D mask output by the preset image segmentation model corresponding to the target single-view image, wherein the preset image segmentation model caches the feature information of the multi-view image; The first 3D mask acquisition unit is used to perform cross-dimensional mask transformation on the first 2D mask to obtain the first 3D mask; The second 2D mask acquisition unit is used to input the first 3D mask into a pre-trained 3D Gaussian model to obtain the second 2D mask output by the 3D Gaussian model, wherein the 3D Gaussian model is trained based on the multi-view image; The comprehensive feature information acquisition unit is used to fuse the feature information of the second 2D mask and the multi-view image to obtain comprehensive feature information; The two-dimensional mask prompt information acquisition unit is used to input the comprehensive feature information into a preset intelligent prompt generation network to obtain the two-dimensional mask prompt information output by the preset intelligent prompt generation network; The third 2D mask acquisition unit is used to input the two-dimensional mask prompt information, the second 2D mask and the comprehensive feature information into the preset image segmentation model to obtain the third 2D mask output by the preset image segmentation model corresponding to the target single-view image; The second 3D mask obtaining unit is used to perform cross-dimensional mask transformation on the third 2D mask to obtain the second 3D mask; The iteration termination condition judgment unit is used to determine whether the iteration termination condition has been met. If not, the iteration loop control unit is triggered. The iterative loop control unit is used to take the mask obtained by fusing the second 3D mask and the first 3D mask as the first 3D mask and re-trigger the second 2D mask acquisition unit.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the 3D mask construction method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the 3D mask construction method as described in any one of claims 1 to 7.

Images