A face identity protection method and system for three-dimensional Gaussian point cloud scene editing

Abstract

The application relates to a face identity protection method and system for three-dimensional Gaussian point cloud scene editing. The method comprises the following steps: acquiring three-dimensional Gaussian point cloud scene data and camera internal and external parameters, generating a scene image through multi-view rendering; detecting a face to obtain a region mask, and projecting two-dimensional face pixels to a three-dimensional space to screen out target Gaussian points related to the identity of a person; performing text-driven image editing based on a diffusion editing model, extracting face features before and after editing to calculate identity consistency loss and visual feature difference loss; generating weights according to the identity leakage degree of each view angle, and obtaining a multi-view identity protection loss through weighted fusion, and the multi-view identity protection loss is combined with a three-dimensional scene editing loss to be optimized, and finally, scene parameters are updated. The method positions target Gaussian points through multi-view face back projection, and combines multi-view weighted identity protection loss to be optimized, so that the face identity protection capability and scene editing security in the Gaussian point cloud scene editing process are significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D content editing technology, and in particular relates to a method and system for protecting facial identity in 3D Gaussian point cloud scene editing. Background Technology

[0002] In recent years, with the development of computer vision and 3D reconstruction technologies, 3D Gaussian point clouds have become a core technology for 3D scene representation due to their efficient 3D scene representation capabilities and high-quality new perspective rendering effects. They have enormous potential for application in cultural tourism and digital museum scenarios such as the digital exhibition hall of the Inner Mongolia Museum, smart scenic spots, grassland VR tours, and virtual human representations of intangible cultural heritage. Based on this, researchers have proposed text-driven 3D scene editing technology, enabling users to directly modify the appearance of characters, object attributes, and environmental elements in 3D scenes through natural language descriptions, significantly improving the flexibility and convenience of 3D content editing.

[0003] In practical applications, user selfies / scans to generate 3DGS avatars have gradually become an important form of interaction in the aforementioned scenarios. However, user facial information is extremely vulnerable to theft, fabrication, and forgery, posing a serious threat to identity privacy and data security. Furthermore, when editing involves people, existing text-driven 3D Gaussian point cloud editing methods can easily and irregularly alter key identity features such as facial texture and facial structure, leading to abnormal tampering of facial identity information. Moreover, 3D scenes possess multi-view observation attributes, and facial editing results from different perspectives are prone to inconsistencies in features, further compromising the authenticity and stability of the person's identity.

[0004] Currently, mainstream 3D scene editing technologies only focus on optimizing the visual presentation of scene editing, without building a dedicated protection mechanism for facial identity information. They lack technical solutions for multi-view identity feature constraints, identity leakage assessment, and weighted optimization, making it impossible to simultaneously achieve effective protection of facial identity while ensuring the effect of 3D scene editing. This greatly reduces the security and reliability of 3D content editing systems and makes them unsuitable for practical application scenarios with strict requirements for identity privacy.

[0005] Therefore, how to protect the identity information of characters while maintaining the editing effect of 3D scenes has become an important technical problem that urgently needs to be solved in the field of 3D content editing. Summary of the Invention

[0006] Therefore, it is necessary to provide a method and system for facial identity protection in 3D Gaussian point cloud scene editing to address the aforementioned technical problems. This aims to solve the issue that existing 3D scene editing technologies are prone to causing changes in facial features during text-driven editing, leading to abnormal alterations in facial identity information. By introducing facial identity feature constraints and a multi-view identity weighting mechanism during 3D scene editing, the consistency of facial identity features can be effectively maintained during scene editing operations, thereby ensuring effective protection of facial identity information while maintaining editing quality.

[0007] Firstly, this application provides a method for face identity protection in 3D Gaussian point cloud scene editing, including:

[0008] Obtain the 3D Gaussian point cloud scene data to be edited, and load the camera intrinsic and extrinsic parameter matrices corresponding to the 3D Gaussian point cloud scene data; perform multi-view rendering based on the 3D Gaussian point cloud scene data, camera intrinsic and extrinsic parameter matrices to obtain a set of multi-view scene images;

[0009] Face region detection is performed on a collection of multi-view scene images to obtain face region masks for each view. Based on the face region masks for each view, the camera intrinsic and extrinsic matrices, the two-dimensional face pixels are back-projected into three-dimensional space to obtain the projection results. The projection results are then filtered to obtain a set of target three-dimensional Gaussian points associated with the person's identity.

[0010] Based on the text editing instructions input by the user, the multi-view scene image set is edited using a diffusion editing model to obtain the edited multi-view image set. Based on the face region corresponding to the target 3D Gaussian point set, features are extracted from the face regions of the multi-view scene image set and the edited multi-view image set to obtain the first face identity feature and the second face identity feature. Based on the first face identity feature and the second face identity feature, the identity consistency loss and the visual feature difference loss are calculated.

[0011] The degree of identity leakage from each perspective is calculated based on the degree of change between the first and second facial identity features. The degree of identity leakage from each perspective is then normalized to generate weights for multiple perspectives. The weights of multiple perspectives are then used to weight and fuse the identity consistency loss and visual feature difference loss to obtain the multi-perspective identity protection loss.

[0012] The 3D scene editing loss is calculated based on the multi-view scene image set and the edited multi-view image set. The 3D scene editing loss and the multi-view identity protection loss are jointly optimized to obtain the optimization result. Based on the optimization result, the scene parameters corresponding to the 3D Gaussian point cloud scene data are updated, and the final 3D Gaussian point cloud scene parameters are output.

[0013] In one embodiment, the projection result is a set of three-dimensional spatial points. Based on the face region mask from each viewpoint, the camera intrinsic and extrinsic parameter matrices, the two-dimensional face pixels are back-projected into three-dimensional space to obtain the projection result. The projection result is then filtered to obtain a set of target three-dimensional Gaussian points associated with the person's identity, including:

[0014] By using the camera intrinsic and extrinsic parameter matrices, the three-dimensional rays corresponding to the two-dimensional face pixels corresponding to the face region mask at each viewpoint are calculated to obtain the set of three-dimensional rays.

[0015] By combining the geometric structure of the 3D Gaussian point cloud scene data, the 3D spatial position corresponding to the 3D ray is determined, and a set of 3D spatial points is obtained;

[0016] Calculate the Euclidean distance between the set of three-dimensional spatial points and the three-dimensional Gaussian points in the three-dimensional Gaussian point cloud scene data, and obtain the distance calculation result;

[0017] Based on the covariance parameter of the three-dimensional Gaussian points, an adaptive distance threshold is determined. Based on the adaptive distance threshold and the distance calculation results, the three-dimensional Gaussian points in the three-dimensional Gaussian point cloud scene data are initially screened to obtain the set of face-related Gaussian points corresponding to each viewpoint.

[0018] The sets of Gaussian points associated with faces from each viewpoint are merged to obtain a fused set of Gaussian points associated with identities.

[0019] Calculate the number of times each Gaussian point in the identity-associated Gaussian point set is identified as a face region in multiple views, and assign a weight to each Gaussian point in the identity-associated Gaussian point set based on the number of times;

[0020] The set of Gaussian points associated with identity is weighted and filtered according to the weights to obtain the target three-dimensional Gaussian point set.

[0021] In one embodiment, based on the first facial identity features and the second facial identity features, the identity consistency loss and visual feature difference loss are calculated, including:

[0022] Calculate the cosine similarity between the first and second facial identity features, and then calculate the identity consistency loss based on the cosine similarity.

[0023] The first visual features of the multi-view scene image set and the second visual features of the edited multi-view image set were extracted using a visual feature extraction network.

[0024] Calculate the L2 distance between the first visual feature and the second visual feature to obtain the visual feature difference loss.

[0025] In one embodiment, the degree of identity leakage for each viewpoint is calculated based on the degree of change between the first and second facial identity features from each perspective; the degree of identity leakage for each viewpoint is then normalized to generate weights for multiple views, including:

[0026] Calculate the change in similarity between the first and second facial identity features from each viewpoint, and determine the change in similarity as the degree of identity leakage for the corresponding viewpoint;

[0027] The sum of the degree of identity leakage from each perspective is calculated to obtain the total degree of identity leakage.

[0028] The weight of each perspective is calculated by comparing the degree of identity leakage from a single perspective with the sum of the degree of identity leakage.

[0029] In one embodiment, a 3D scene editing loss is calculated based on a set of multi-view scene images and a set of edited multi-view images; the 3D scene editing loss and the multi-view identity protection loss are jointly optimized to obtain an optimization result, including:

[0030] The image feature extraction network extracts the original image features of the multi-view scene image set and the edited image features of the edited multi-view image set respectively;

[0031] Calculate the feature distance between the original image features and the edited image features to obtain the 3D scene editing loss;

[0032] The 3D scene editing loss and the multi-view identity protection loss are combined to construct a joint optimization objective function;

[0033] The gradient descent method is used to iteratively calculate the joint optimization objective function to obtain the optimization result.

[0034] In one embodiment, based on text editing instructions input by the user, a multi-view scene image set is edited using a diffusion editing model to obtain an edited multi-view image set, including:

[0035] The text editing instructions input by the user are encoded to obtain a text semantic feature vector;

[0036] The text semantic feature vector is input as a conditional constraint into the diffusion editing model. The diffusion editing model performs multi-view consistent denoising editing on the multi-view scene image set, generating an edited multi-view image set that matches the text editing instructions.

[0037] In one embodiment, the multi-view identity protection loss is calculated using the following formula:

[0038]

[0039] in, Losses due to multi-perspective identity protection For the first Weight of each perspective For the first The identity consistency loss from each perspective, where λ is a preset weighting coefficient. For the first Loss of visual feature differences from individual perspectives Here, N represents the viewpoint number, and N represents the total number of viewpoints.

[0040] In one embodiment, the visual feature difference loss is calculated using the following formula:

[0041]

[0042] in, For visual feature difference loss, For visual feature extraction function, A collection of multi-view scene images. This is a collection of edited multi-view images. This refers to the 2-norm operation.

[0043] In one embodiment, the identity consistency loss is calculated using the following formula:

[0044]

[0045] in, For loss of identity consistency, This is a function for extracting facial features. A collection of multi-view scene images. This is a collection of edited multi-view images.

[0046] Secondly, this application also provides a face identity protection system for editing 3D Gaussian point cloud scenes, including:

[0047] The 3D scene acquisition module is used to acquire the 3D Gaussian point cloud scene data to be edited, and load the camera intrinsic and extrinsic parameter matrices corresponding to the 3D Gaussian point cloud scene data; based on the 3D Gaussian point cloud scene data, camera intrinsic and extrinsic parameter matrices, multi-view rendering is performed to obtain a set of multi-view scene images;

[0048] The face detection module is used to detect face regions in a collection of multi-view scene images and obtain face region masks from each view.

[0049] The identity region mapping module is used to back-project two-dimensional face pixels onto three-dimensional space based on face region masks from various viewpoints, camera intrinsic and extrinsic matrix, obtain projection results, filter according to projection results, and obtain a set of target three-dimensional Gaussian points associated with the person's identity.

[0050] The scene editing module is used to edit a collection of multi-view scene images based on the text editing instructions input by the user, and obtain an edited collection of multi-view images.

[0051] The identity constraint module is used to: extract features from the face regions of the multi-view scene image set and the edited multi-view image set based on the face regions corresponding to the target three-dimensional Gaussian point set, respectively, to obtain the first face identity feature and the second face identity feature, and calculate the identity consistency loss and visual feature difference loss based on the first face identity feature and the second face identity feature.

[0052] The degree of identity leakage from each perspective is calculated based on the degree of change between the first and second facial identity features. The degree of identity leakage from each perspective is then normalized to generate weights for multiple perspectives. The weights of multiple perspectives are then used to weight and fuse the identity consistency loss and visual feature difference loss to obtain the multi-perspective identity protection loss.

[0053] The optimization and update module is used to calculate the 3D scene editing loss based on the multi-view scene image set and the edited multi-view image set; the 3D scene editing loss and the multi-view identity protection loss are jointly optimized to obtain the optimization result; based on the optimization result, the scene parameters corresponding to the 3D Gaussian point cloud scene data are updated, and the final 3D Gaussian point cloud scene parameters are output.

[0054] The aforementioned face identity protection method and system for editing 3D Gaussian point cloud scenes has at least the following advantages compared with existing technologies:

[0055] (1) By using multi-view face region detection and two-dimensional to three-dimensional spatial back projection, it is possible to accurately locate the set of target three-dimensional Gaussian points associated with the identity of the person in the three-dimensional Gaussian point cloud scene, realize the targeted locking of the face identity protection area, and effectively solve the problem that traditional three-dimensional scene editing cannot accurately distinguish between the identity area and the ordinary scene area;

[0056] (2) By extracting facial identity features before and after editing and constructing identity consistency loss and visual feature difference loss, it is possible to strictly constrain abnormal changes in facial features during text-driven editing, and ensure that facial identity information is not tampered with from the feature level, thus ensuring the visual authenticity of the facial area.

[0057] (3) Based on the degree of change of facial identity features from multiple perspectives, the degree of identity leakage is calculated and corresponding weights are generated. Key perspectives that are prone to identity leakage can be automatically identified and constraints are strengthened, effectively solving the technical defects of inconsistent identity in face editing in three-dimensional multi-view scenarios.

[0058] (4) By jointly optimizing the 3D scene editing loss and the multi-view identity protection loss, it is possible to meet the user's text editing needs and ensure the 3D scene editing effect while achieving stable protection of facial identity information, further improving the security and reliability of the 3D content editing system. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A flowchart of a face identity protection method for editing 3D Gaussian point cloud scenes, provided as an exemplary embodiment of the present invention;

[0061] Figure 2 A flowchart of a method for obtaining optimized results is provided as an exemplary embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of a face identity protection system for editing 3D Gaussian point cloud scenes, provided as an exemplary embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] In one embodiment, such as Figure 1 As shown, a face identity protection method for editing 3D Gaussian point cloud scenes is provided. This embodiment illustrates the method using a terminal as an example. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0065] S101: Obtain the 3D Gaussian point cloud scene data to be edited, and load the camera intrinsic and extrinsic parameter matrices corresponding to the 3D Gaussian point cloud scene data; perform multi-view rendering based on the 3D Gaussian point cloud scene data, camera intrinsic and extrinsic parameter matrices to obtain a multi-view scene image set.

[0066] Specifically, the 3D scene is represented using a 3D Gaussian point cloud. Compared to traditional 3D mesh models or voxel representation methods, 3D Gaussian point clouds can model the scene using a large number of Gaussian functions with continuous distribution characteristics, thereby achieving efficient expression of complex scene structures and high computational efficiency while ensuring rendering quality. In the 3D Gaussian point cloud representation method, the entire 3D scene can be composed of a large number of Gaussian points with spatial distribution characteristics, and each Gaussian point can be used to describe a local region in the scene. By combining these Gaussian points, a continuous approximate representation of the complete 3D scene can be formed. This representation method can efficiently express the geometric structure and appearance information of complex 3D scenes. The camera intrinsic parameter matrix can be used to characterize the internal projection mapping relationship of camera imaging, while the extrinsic parameter matrix can be used to characterize the rotation and translation pose relationship of the camera in the 3D world coordinate system. Both provide the geometric calculation basis for the subsequent forward projection of 3D spatial points to the 2D image plane and the backward projection of 2D image pixels to 3D space. Subsequently, multi-view rendering can be performed based on the above data. That is, the three-dimensional Gaussian points can be projected onto the two-dimensional image plane according to the camera projection model, and the projection transformation from the three-dimensional covariance matrix to the two-dimensional space can be completed simultaneously. The color contribution of a single pixel position to multiple Gaussian points is weighted and fused to calculate, and finally a set of multi-view scene images covering different observation angles is generated. This avoids the one-sidedness of single-view observation and provides basic data for subsequent face region detection, three-dimensional mapping of identity regions and text-driven scene editing, ensuring the multi-view collaboration and structural stability of three-dimensional scene editing.

[0067] S102: Perform face region detection on the multi-view scene image set to obtain face region masks for each view; based on the face region masks for each view, the camera intrinsic and extrinsic matrix, back-project the two-dimensional face pixels to three-dimensional space to obtain the projection results; filter according to the projection results to obtain a set of target three-dimensional Gaussian points associated with the person's identity.

[0068] Specifically, a deep neural network-based face detection model can be used to perform face detection and localization on a view-by-view basis in multi-view scene images. After obtaining the face bounding box, a refined face region mask can be generated using face semantic segmentation methods. Within this mask, a pixel value of 1 represents the face region, and a pixel value of 0 represents the background region, thus achieving pixel-level precise segmentation of the face region and eliminating background interference. Subsequently, based on the camera imaging geometry model, the two-dimensional face pixels can be back-projected into three-dimensional space. That is, for each pixel coordinate within the face mask, the inverse operation of the camera intrinsic parameter matrix can be used to convert it into a three-dimensional ray direction. Then, combined with the camera extrinsic parameter matrix, the ray is transformed to the world coordinate system, obtaining the set of three-dimensional ray points corresponding to the pixel. When filtering target three-dimensional Gaussian points, the distance between the three-dimensional ray and the center of each three-dimensional Gaussian point can be calculated to filter out the target three-dimensional Gaussian point set. This set locks the Gaussian point regions in the three-dimensional scene that are directly related to the identity of the person's face, ensuring that subsequent identity constraints only apply to identity-sensitive areas, thus guaranteeing the effectiveness of identity protection without affecting the normal editing of non-identity areas in the scene.

[0069] S103: Based on the text editing instructions input by the user, the multi-view scene image set is edited using a diffusion editing model to obtain the edited multi-view image set; based on the face region corresponding to the target three-dimensional Gaussian point set, features are extracted from the face regions of the multi-view scene image set and the edited multi-view image set respectively to obtain the first face identity feature and the second face identity feature; based on the first face identity feature and the second face identity feature, the identity consistency loss and visual feature difference loss are calculated.

[0070] Specifically, the diffusion editing model, as the core network for text-driven 3D scene editing, first converts user-input text editing commands into standardized semantic feature vectors via a text encoding network. Then, using these semantic features as constraints, it performs joint denoising editing on multi-view scene images, generating a set of edited multi-view images. This avoids geometric conflicts between editing results from different perspectives and ensures the consistency of the overall structure of the 3D scene. Further, a deep neural network-based face recognition model can be used to map face region images into high-dimensional face identity feature vectors. For example, the first face identity feature can be extracted from the face region of the multi-view scene image before editing, and the second face identity feature can be extracted from the face region of the multi-view image after editing. Subsequently, the similarity between the first and second face identity features, such as cosine similarity, can be calculated to obtain the identity consistency loss. This loss quantifies the degree of difference in face identity before and after editing; a smaller loss value indicates higher identity consistency. The visual feature difference loss can be obtained by using a visual feature extraction network to obtain the visual feature representations of the images before and after editing, calculating the difference between the two, and is used to constrain visual structural distortion in the face region, avoiding excessive changes to facial texture and contours. Through the synergistic effect of the two types of loss, facial identity features can be stabilized from both semantic and visual morphological dimensions, preventing abnormal changes in identity information during the editing process.

[0071] S104: Calculate the degree of identity leakage from each perspective based on the degree of change between the first and second facial identity features from each perspective; normalize the degree of identity leakage from each perspective to generate weights for multiple perspectives; and use the weights of multiple perspectives to perform weighted fusion of identity consistency loss and visual feature difference loss to obtain the multi-perspective identity protection loss.

[0072] Specifically, the degree of identity leakage is determined by the change in the similarity of facial identity features before and after editing from each perspective. The more significant the change in facial features, the higher the degree of identity leakage, indicating a higher priority for identity protection from that perspective. The weights of multiple perspectives are obtained by normalizing the degree of identity leakage for each perspective. After normalization, the sum of the weights of all perspectives is 1, and the higher the degree of identity leakage, the greater the corresponding weight. This allows for the automatic identification and strengthening of identity constraints from key perspectives such as frontal views and views with complete facial features. This weighting allows for the weighted summation of the identity consistency loss and visual feature difference loss from each perspective to obtain the multi-perspective identity protection loss. This loss, through a multi-perspective weighting mechanism, not only effectively addresses the problem of uneven importance of facial expression from different perspectives but also imposes stronger constraints on perspectives with high risk of identity leakage, improving the facial identity protection effect in multi-perspective scenarios and avoiding identity tampering caused by imbalanced constraints from a single perspective.

[0073] S105: Calculate the 3D scene editing loss based on the multi-view scene image set and the edited multi-view image set; jointly optimize the 3D scene editing loss and the multi-view identity protection loss to obtain the optimization result; based on the optimization result, update the scene parameters corresponding to the 3D Gaussian point cloud scene data and output the final 3D Gaussian point cloud scene parameters.

[0074] Specifically, the 3D scene editing loss measures the degree of matching between the 3D Gaussian point cloud rendering result and the text editing target. It is obtained by calculating the pixel and feature differences between the multi-view scene rendering image and the edited multi-view image, ensuring that the editing result strictly conforms to the user's text instructions. The 3D scene editing loss, identity consistency loss, and visual feature difference loss are incorporated into a unified optimization framework. The gradient of the total loss function is calculated using a gradient descent optimization algorithm, and scene parameters such as the spatial position, color, opacity, and covariance matrix of the 3D Gaussian points are iteratively updated based on the gradient. During optimization, the identity region corresponding to the target 3D Gaussian point set is subject to stronger constraints to prevent identity tampering caused by abnormal parameter adjustments, while parameters in non-identity regions can be optimized normally to meet editing requirements. After optimization, the updated 3D Gaussian point cloud scene parameters can be output. The 3D scene corresponding to these parameters achieves both text-driven editing effects and stably maintains the consistency of facial identity features, effectively reducing the risk of identity leakage and tampering in 3D scene editing and significantly improving the security and reliability of the 3D content editing system.

[0075] The aforementioned method first addresses the issues of inefficient 3D scene representation and missing basic data for multiple perspectives by acquiring 3D Gaussian point cloud scene data and camera parameters and performing multi-view rendering. Secondly, it solves the problems of inaccurate 3D positioning of facial identity regions and ambiguous identity constraints by filtering the target 3D Gaussian point set based on face detection and 2D pixel back-projection. Furthermore, it addresses the issues of easy tampering with facial identities and facial visual structure distortion during 3D scene editing by using text-driven diffusion editing and calculating identity consistency loss and visual feature difference loss. Even further, it generates weights based on the degree of identity leakage from multiple perspectives and fuses them to obtain multi-view identity protection loss, solving the problems of unbalanced multi-view constraints and insufficient protection effects for key perspectives. Finally, by jointly optimizing and updating the 3D Gaussian point cloud scene parameters, it achieves a balance between scene editing effects and facial identity protection, improving the security, reliability, and identity consistency of 3D Gaussian point cloud scene editing.

[0076] In one alternative implementation, the initial structural information of the 3D scene can be obtained through a multi-view image reconstruction algorithm, and a corresponding set of Gaussian points can be generated in 3D space. For example, let the set of Gaussian points in the 3D scene be: , where G represents the set of three-dimensional Gaussian points; Indicates the first A Gaussian point, This represents the total number of Gaussian points. Each Gaussian point can contain multiple parameters, such as 3D spatial location parameters, color attribute parameters, opacity parameters, and covariance matrix parameters, used to describe its geometric location, shape, and color information in 3D space. Specifically, the... A Gaussian point can be represented as: ,in This indicates the position of the Gaussian point in three-dimensional space. This represents the color attribute of the Gaussian point. The opacity parameter represents the Gaussian point. The covariance matrix of a Gaussian point describes the shape and orientation of the Gaussian distribution in three-dimensional space. Furthermore, the extent of the Gaussian point's influence in space can usually be described by a three-dimensional Gaussian distribution function; for example, let any point in three-dimensional space be... Its Gaussian distribution function It can be represented as:

[0077]

[0078] in, Represents any point in three-dimensional space. Indicates the center position of the Gaussian point. Denotes the inverse of the covariance matrix. This indicates the transpose operation. This function represents the distance relationship between a point in space and the center of the Gaussian point. When a point in space is close to the center of the Gaussian point, the value of the Gaussian function is larger; as the distance increases, the value of the Gaussian function gradually decreases, thus forming a continuous spatial distribution.

[0079] The above method can be used to establish a set of Gaussian points to represent a 3D scene, providing a foundation for subsequent multi-view rendering and scene editing.

[0080] Furthermore, in the process of multi-view rendering based on 3D Gaussian point cloud scene data, camera intrinsic and extrinsic parameter matrices, it is necessary to project the 3D Gaussian points onto a 2D image plane. For example, let the camera's intrinsic parameter matrix be... The extrinsic parameter matrix is Then three-dimensional points The process of projecting onto the image plane can be represented as:

[0081]

[0082] in Represents the two-dimensional pixel coordinates after projection. Represents the camera rotation matrix. Represents the camera translation vector. This represents the camera intrinsic parameter matrix.

[0083] Through the projection process described above, three-dimensional Gaussian points can be mapped onto a two-dimensional image plane. To preserve the spatial shape of the Gaussian distribution, the three-dimensional covariance matrix also needs to be projected onto the two-dimensional space. Furthermore, after projection, the color contribution of each Gaussian point at its pixel location needs to be calculated. Since a pixel location is typically influenced by multiple Gaussian points during actual rendering, the color contributions of multiple Gaussian points can be fused. For example, let the pixel location in the image plane be... The color value at this position can be represented as:

[0084]

[0085] in, Indicates pixel color, Indicates the color of the Gaussian point. This indicates the weight. The weight... It can be calculated based on the Gaussian distribution function value and the opacity parameter of the Gaussian point, thus reflecting the degree of influence of different Gaussian points on pixel color.

[0086] This fusion calculation method enables the color fusion of multiple Gaussian points to generate the final image. For 3D scene editing, multiple viewpoint images can also be generated. Assuming there are N viewpoints, the generated image set can be: ,in Indicates the first A rendered image from N perspectives, where N represents the number of perspectives.

[0087] These multi-view images can serve as input for subsequent face region detection, identity region localization, and 3D scene editing. Furthermore, during 3D scene editing, Gaussian points are not only used to generate images but also serve as the primary parameter carrier for subsequent scene optimization. By adjusting the position, color, opacity, and spatial distribution parameters of the Gaussian points, modifications to the structure and appearance of the 3D scene can be achieved. Therefore, the aforementioned 3D Gaussian point cloud representation method not only enables high-quality 3D scene rendering but also provides a direct target for the subsequent identity protection optimization module, thus laying the foundation for constructing a 3D scene editing method with identity protection capabilities.

[0088] In one alternative implementation, after constructing the 3D Gaussian point cloud scene and rendering multi-view images, it is necessary to further locate the human figures within the scene. Indicatively, the identity of a person is primarily determined by facial structure and texture features; therefore, accurate identification of face regions in multi-view images is crucial. Furthermore, in real-world 3D scenes, since people may be observed from different perspectives, relying solely on a single perspective for face detection often fails to yield complete face region information. Therefore, this embodiment employs a multi-view face detection method, jointly analyzing face regions in images from multiple perspectives to obtain more stable and accurate face region localization results.

[0089] Specifically, in this embodiment, the multi-view image set can be represented as: ,in Indicates the first These are rendered images from multiple perspectives, where N represents the number of perspectives. These multi-view images are generated from the previous stage's 3D Gaussian point cloud rendering operation and can reflect the 3D scene structure from different angles.

[0090] After obtaining multi-view images, the first step is to perform face detection on each view image. Specifically, a deep learning-based face detection model, such as a convolutional neural network-based face detection model, can be used to perform region-by-region detection on the input image to identify potential face regions. When analyzing a particular view image... During detection, the face detection model can output the location of the corresponding face region. This region can typically be represented by a rectangular bounding box.

[0091]

[0092] in, This represents the pixel coordinates of the top-left corner of the face region. This represents the width and height of the face region. Through the above detection process, candidate face regions can be obtained in each viewpoint image.

[0093] However, in practical applications, relying solely on bounding boxes is often insufficient to accurately describe the precise shape of a face. For example, the bounding box may include parts of the background region, thus affecting the accuracy of subsequent identity region mapping. Therefore, after obtaining the face bounding box, this implementation further employs a face semantic segmentation method to refine the face region.

[0094] Specifically, semantic segmentation algorithms can be performed within the detected face regions to obtain a more accurate face region mask. This mask is used to identify the pixel positions belonging to the face region in the image. For example, the first... The face region mask from each perspective is... Its definition is:

[0095]

[0096] Among them, when =1 indicates that the pixel belongs to the face region, when =0 indicates that the pixel belongs to the background area.

[0097] Through the multi-view face detection and region segmentation process described above, the spatial extent of the facial region can be obtained more accurately, thus providing a more reliable input for subsequent identity region mapping. After obtaining the face region mask for each viewpoint, the corresponding set of face pixels can be further extracted. Specifically, all pixel positions in the mask image can be traversed, and those that satisfy the following conditions can be extracted. =1 pixel coordinate. In this way, the set of face pixels from the v-th viewpoint can be obtained.

[0098] Since facial regions may differ from different viewpoints—for example, some viewpoints may only show the profile or part of the face—this embodiment utilizes a multi-view face detection method to comprehensively utilize facial information from different viewpoints, thereby obtaining a more complete description of the facial region.

[0099] Furthermore, consistency screening can be performed on face detection results from different viewpoints. For example, if a face region detected from a certain viewpoint deviates significantly from the results from other viewpoints, the result can be filtered out using a confidence assessment method, thereby preventing abnormal detection results from affecting subsequent processing.

[0100] Through the above multi-view face detection and region segmentation process, a set of face regions from all views can be obtained. This set can be represented as: , This represents the face region mask from the v-th viewpoint. These face region masks not only describe the spatial location of faces in the image, but also provide basic data for subsequent 2D-to-3D identity region mapping.

[0101] In the above process, the facial region detection and segmentation can locate the person's identity region without directly participating in the 3D scene editing process. Furthermore, by determining the facial region in advance, additional constraints can be applied to that region during the subsequent 3D scene editing process, thereby preventing significant changes in the person's identity features.

[0102] In one optional implementation, the projection result can be a set of three-dimensional spatial points. Based on the face region mask from each viewpoint, the camera intrinsic and extrinsic parameter matrices, the two-dimensional face pixels are back-projected into three-dimensional space to obtain the projection result. The projection result is then filtered to obtain a set of target three-dimensional Gaussian points associated with the person's identity, which may include:

[0103] By using the camera intrinsic and extrinsic parameter matrices, the three-dimensional rays corresponding to the two-dimensional face pixels corresponding to the face region mask at each viewpoint are calculated to obtain the set of three-dimensional rays.

[0104] By combining the geometric structure of the 3D Gaussian point cloud scene data, the 3D spatial position corresponding to the 3D ray is determined, and a set of 3D spatial points is obtained;

[0105] Calculate the Euclidean distance between the set of three-dimensional spatial points and the three-dimensional Gaussian points in the three-dimensional Gaussian point cloud scene data, and obtain the distance calculation result;

[0106] Based on the covariance parameter of the three-dimensional Gaussian points, an adaptive distance threshold is determined. Based on the adaptive distance threshold and the distance calculation results, the three-dimensional Gaussian points in the three-dimensional Gaussian point cloud scene data are initially screened to obtain the set of face-related Gaussian points corresponding to each viewpoint.

[0107] The sets of Gaussian points associated with faces from each viewpoint are merged to obtain a fused set of Gaussian points associated with identities.

[0108] Calculate the number of times each Gaussian point in the identity-associated Gaussian point set is identified as a face region in multiple views, assign a weight to each Gaussian point in the identity-associated Gaussian point set based on the number of times, and perform weighted filtering on the identity-associated Gaussian point set based on the weight to obtain the target 3D Gaussian point set.

[0109] Specifically, after completing multi-view face detection and region segmentation, the face region mask and face pixel set corresponding to each view image can be obtained. However, the aforementioned face regions are still located in two-dimensional image space, while the three-dimensional scene editing process mainly operates on a three-dimensional Gaussian point cloud scene. Therefore, it is necessary to further map the face regions in the two-dimensional image to three-dimensional space to determine the three-dimensional Gaussian point regions associated with the person's identity. Therefore, after obtaining the face region masks, camera intrinsic and extrinsic parameter matrices for each view, pixel-level processing can be performed on the face regions of each view first. For example, the face region masks of each view can be traversed, and the pixel coordinates belonging to the face regions can be extracted. Let the face region mask of the v-th view image be... Then, the corresponding set of face pixels can be obtained by traversing the mask.

[0110] Furthermore, in computer vision, camera imaging models can be used to map the aforementioned two-dimensional pixel coordinates into three-dimensional space. For example, for any pixel coordinates in an image... This can be converted into a ray direction in 3D space using the camera intrinsic parameter matrix. Specifically, the direction vector of a pixel in the camera coordinate system can be calculated first as follows:

[0111]

[0112] in, Represents pixel coordinates in a two-dimensional image. This represents the inverse of the camera intrinsic parameter matrix. This indicates the direction of the three-dimensional ray corresponding to this pixel.

[0113] Through the above calculations, the set of rays representing the pixels of the face region in three-dimensional space can be obtained. Furthermore, the spatial position of this ray in the world coordinate system can be determined by combining it with the camera extrinsic parameter matrix, thereby realizing the mapping of the two-dimensional face region to three-dimensional space.

[0114] It's important to note that in a 3D Gaussian point cloud scene, each Gaussian point has a corresponding 3D center position. Therefore, after obtaining the set of 3D rays, the spatial relationships between these rays and the 3D Gaussian points can be further calculated to determine which Gaussian points belong to the face region. For example, for each 3D ray, the spatial distance between the ray and the center of the 3D Gaussian point can be calculated; this distance can be Euclidean distance. When the distance between the center of the Gaussian point and the ray is less than an adaptive distance threshold, the Gaussian point can be considered to belong to the face region corresponding to the current viewpoint. This threshold can be determined based on the covariance parameter of the 3D Gaussian points, which can be used to characterize the spatial distribution shape, influence range, and geometric characteristics of the Gaussian points. By traversing and calculating all rays, the set of face-related Gaussian points under the current viewpoint can be obtained.

[0115] In a 3D scene, a person is usually observed from multiple perspectives simultaneously. Therefore, the facial region under different perspectives may be mapped to different sets of Gaussian points. To obtain a more stable identity region, this embodiment can further perform unified modeling of the identity Gaussian points through multi-view fusion.

[0116] Specifically, the set of face-related Gaussian points obtained from each viewpoint can be merged, and duplicate Gaussian points can be removed to obtain a unified set of identity-related Gaussian points. For example, let the first... The set of Gaussian points obtained from each perspective is Then the final set of identity-associated Gaussian points can be represented as:

[0117]

[0118] in Represents the set of Gaussian points associated with identity. Indicates the first The set of Gaussian points associated with the face corresponding to each viewpoint This represents a set union operation. Through the above multi-view fusion process, a more stable representation of the identity region can be obtained.

[0119] Furthermore, the number of times each Gaussian point is identified as a face region across multiple viewpoints can be calculated, and a corresponding weight can be assigned to the Gaussian point based on this number. If a Gaussian point is identified as a face region in multiple viewpoints, it indicates that the Gaussian point has a high identity relevance, and therefore can be assigned a larger weight. The set of identity-related Gaussian points is then weighted and filtered based on these weights to obtain the target 3D Gaussian point set. This method allows Gaussian points that are detected as facial regions in multiple viewpoints to have higher weight priority in subsequent optimizations, thereby improving the identity protection effect.

[0120] Through the above process, the target 3D Gaussian point set can be obtained in a 3D Gaussian point cloud scene. This set is used to identify areas in a 3D scene that are related to a person's identity, and will serve as the object of identity protection constraints during subsequent 3D scene editing.

[0121] In one optional implementation, the multi-view scene image set is edited according to the user-input text editing instructions using a diffusion editing model to obtain an edited multi-view image set, which may include:

[0122] The text editing instructions input by the user are encoded to obtain a text semantic feature vector;

[0123] The text semantic feature vector is input as a conditional constraint into the diffusion editing model. The diffusion editing model performs multi-view consistent denoising editing on the multi-view scene image set, generating an edited multi-view image set that matches the text editing instructions.

[0124] In this embodiment, 3D scene editing is primarily achieved through a text-driven approach. Specifically, users can modify the appearance of characters, object attributes, or scene structure in a 3D scene by inputting natural language descriptions. For example, users can modify character clothing colors, scene lighting conditions, or environmental elements using text prompts.

[0125] To achieve the aforementioned text-driven editing functionality, this embodiment employs an image editing method based on a diffusion model. The diffusion model is a generative model widely used in image generation and editing tasks in recent years. Its core idea is to generate images that meet conditional constraints through a progressive denoising process. After obtaining a set of multi-view scene images, user-inputted text editing instructions (including text prompts), 3D Gaussian point cloud scene parameters, and a target 3D Gaussian point set, the user-inputted text prompts can first be encoded. Specifically, a text encoding network can be used to convert natural language descriptions into corresponding semantic feature vectors, thereby providing conditional information for subsequent image editing processes. For example, if the user-inputted text prompt is T, then the corresponding semantic representation can be obtained through a text encoder:

[0126]

[0127] Where T represents the text prompt entered by the user. Represents a text encoding network. This represents the semantic features of the text. Through the text encoding process described above, natural language descriptions can be converted into feature representations that the model can process.

[0128] After obtaining the semantic features of the text, the scene image can be edited based on this semantic information. In this embodiment, 3D scene editing is mainly achieved through conditional generation of multi-view images. Specifically, multi-view scene images can be input into a diffusion editing network, using the spatial location and spatial distribution of the target 3D Gaussian point set as spatial anchoring constraints, and utilizing the text semantic features as conditional constraints, thereby generating a new image that conforms to the text description.

[0129] The basic process of a diffusion model typically includes a forward diffusion process and a backward denoising process. In the forward diffusion process, noise is gradually added to the image, transforming the original image into a noise distribution. In the backward process, noise is gradually removed using a denoising network, thereby generating a new image.

[0130] Schematably, the diffusion process can be represented as: , Represents the original image. Indicates the first Noisy images at each time step, Indicates Gaussian noise. This represents the noise scheduling parameters.

[0131] In the reverse denoising process, the model predicts noise and gradually restores the image structure to generate an edited result that conforms to the text conditions.

[0132] It should be noted that in 3D scene editing tasks, editing only a single viewpoint image may lead to inconsistent editing results between different viewpoints, thereby disrupting the 3D scene structure. Therefore, in this embodiment, a multi-viewpoint consistent editing method can be used to jointly process images from all viewpoints.

[0133] Specifically, multiple viewpoint images can be simultaneously input into a diffusion editing network, using the same textual semantic features as conditional constraints to generate edited images from multiple perspectives. For example, suppose the set of edited multi-viewpoint images is: ,in This represents the edited image from the v-th viewpoint. Indicates the number of viewpoints.

[0134] By using the multi-view editing methods described above, consistent editing effects can be achieved across different perspectives, thereby avoiding inconsistencies in the three-dimensional structure.

[0135] In this embodiment, 3D scene editing not only involves the 2D image level but also requires updating the 3D Gaussian point cloud parameters. Specifically, the color attributes, opacity, and spatial distribution parameters of the 3D Gaussian points in the target 3D Gaussian point set can be adjusted based on the edited image results, thereby achieving a comprehensive modification of the 3D scene. For example, optimization methods such as gradient descent-based iterative optimization can be used to gradually bring the 3D Gaussian point cloud rendering results closer to the edited image, thus achieving consistency between the 3D scene and the 2D editing results.

[0136] Through the above text-driven 3D scene editing process, the 3D scene can be flexibly modified according to the text prompts entered by the user, and the editing results can be provided for the subsequent identity protection optimization module.

[0137] In one optional implementation, calculating the identity consistency loss and visual feature difference loss based on the first facial identity features and the second facial identity features may include:

[0138] Calculate the cosine similarity between the first and second facial identity features, and then calculate the identity consistency loss based on the cosine similarity.

[0139] The first visual features of the multi-view scene image set and the second visual features of the edited multi-view image set were extracted using a visual feature extraction network.

[0140] Calculate the L2 distance between the first visual feature and the second visual feature to obtain the visual feature difference loss.

[0141] Specifically, after completing text-driven 3D scene editing, edited images from multiple perspectives can be obtained. However, in actual editing, relying solely on text prompts to modify the scene often leads to changes in the facial structure of characters, thereby compromising their identity information. For example, when modifying a character's clothing or environmental attributes, the generated model may simultaneously alter the character's facial texture or facial features, resulting in inconsistencies between the edited character's identity and the original character's identity.

[0142] Therefore, in this embodiment, identity consistency constraints can be introduced during the 3D scene editing process to maintain the stability of character identities while meeting editing requirements. To this end, this embodiment introduces an identity consistency constraint loss function, which can constrain changes in character identities by comparing facial features of images before and after editing.

[0143] Schematic illustration: After obtaining the set of multi-view scene images before editing, the set of multi-view images after editing, the face region masks for each viewpoint, and the target 3D Gaussian point set, feature extraction can be performed on the face regions in the images before and after editing based on the face regions corresponding to the target 3D Gaussian point set. Specifically, a face recognition network can be used to encode the face regions to obtain the corresponding face feature vectors. For example, a face recognition model based on deep neural networks, such as a convolutional neural network structure, can be used to extract features from the input face image. This network can map the input face image into a high-dimensional feature vector, which can then be used to describe the person's identity information. Assume the face feature extraction function is... Therefore, for input image III, its facial features can be represented as: Where I represents the input face image, This represents a facial feature extraction network. This represents the corresponding facial feature vector. Through the above feature extraction process, facial feature representations of the image before and after editing can be obtained, namely, the first facial identity feature and the second facial identity feature.

[0144] Furthermore, the degree of identity leakage can be measured by comparing the similarity of facial features between the images before and after editing. For example, cosine similarity can be used to calculate the similarity between two feature vectors. Let the facial features of the image before editing be... The facial features of the edited image are The similarity between the two can be expressed as:

[0145]

[0146] Where S represents the similarity between two feature vectors. This represents the facial features of the image before editing. This represents the facial features of the edited image. The more similar the features are, the closer the similarity value is to 1. However, when the identity of the person changes significantly, the similarity value decreases significantly. Schematic, this similarity S can also be represented as... .

[0147] Furthermore, to constrain identity changes during the optimization process, this embodiment also constructs an identity consistency loss function. This loss function can be used to measure the difference between the character's identity before and after editing, and to minimize this difference during the optimization process. For example, the identity consistency loss can be expressed as:

[0148]

[0149] in, This indicates a loss of identity consistency. This indicates the similarity between facial features before and after editing. When the person's identity remains the same before and after editing, the similarity score is... When the similarity is close to 1, the identity loss approaches 0. However, when a person's identity changes, the similarity decreases, leading to a greater identity loss. By minimizing this loss function during the optimization process, changes in facial structure can be effectively constrained, maintaining the stability of the person's identity.

[0150] Furthermore, relying solely on identity feature similarity may still be insufficient to fully constrain changes in facial details. Therefore, this embodiment also introduces a visual feature difference loss to constrain the differences in visual features between the images before and after editing. Specifically, a visual feature extraction network can be used to encode the multi-view scene image set before editing and the multi-view image set after editing, and calculate the differences in image features before and after editing. For example, let the visual feature extraction function be... Then the visual feature difference loss can be expressed as: ,in This refers to the image before editing. This indicates the edited image. This represents a visual feature extraction network. This represents the L2 norm. This loss function can constrain the differences in the feature space between the edited and unedited images, thereby further improving identity preservation.

[0151] In this embodiment, identity consistency loss and visual feature difference loss can work together to form a comprehensive identity constraint mechanism. This approach allows for the simultaneous maintenance of consistency in character identity features and visual structure during 3D scene editing. Specifically, the identity constraint loss primarily affects the identity regions determined by the identity Gaussian mapping, thereby imposing stronger constraints on these regions during subsequent optimization processes, thus preventing excessive modification of the character's facial areas.

[0152] In one optional implementation, the degree of identity leakage for each viewpoint is calculated based on the degree of change between the first and second facial identity features from each perspective; normalization is then performed based on the degree of identity leakage for each viewpoint to generate weights for multiple views, which may include:

[0153] Calculate the change in similarity between the first and second facial identity features from each viewpoint, and determine the change in similarity as the degree of identity leakage for the corresponding viewpoint;

[0154] The sum of the degree of identity leakage from each perspective is calculated to obtain the total degree of identity leakage.

[0155] The weight of each perspective is calculated by comparing the degree of identity leakage from a single perspective with the sum of the degree of identity leakage.

[0156] Specifically, the aforementioned identity consistency constraint loss function can, to some extent, limit changes in facial features during the editing process. However, in real-world 3D scenes, since people are typically observed simultaneously from multiple perspectives, facial regions may exhibit different forms from different viewpoints. For example, in some perspectives, the face may be directly facing the camera, resulting in a clearer facial structure, while in other perspectives, the face may appear in profile or partially occluded. If the same constraint strength is applied to all perspectives during optimization, changes in facial features in some key perspectives may still not be effectively controlled. To address this issue, this embodiment further proposes a multi-view identity weighted optimization method. By analyzing the degree of facial feature changes from different perspectives, different weights are assigned to each perspective, thereby strengthening the identity constraints of key perspectives during the optimization process.

[0157] Indicatively, the degree of change in facial features at each viewpoint can be assessed first. This can be achieved by using the aforementioned facial feature extraction network to calculate the first and second facial identity features of the images before and after editing, respectively, and then using feature similarity to measure the degree of identity leakage. For example, let the image before editing at the v-th viewpoint be... The edited image is Then, their facial features can be extracted separately:

[0158]

[0159]

[0160] in This represents a facial feature extraction network. This represents the facial features of the image before editing from the v-th viewpoint. This represents the facial features of the image after editing from the v-th viewpoint.

[0161] Furthermore, cosine similarity can be used to measure the degree of identity leakage by calculating the similarity between the two feature vectors mentioned above. This calculation yields an identity change score for each viewpoint, and the score increases accordingly when facial features change significantly from a particular viewpoint. This identity change score can then be defined as the degree of identity leakage. Let the... The degree of identity leakage from each perspective is This value can be obtained through feature similarity calculation. The greater the difference in facial features between the images before and after editing, the more obvious the change in facial structure from that perspective. Therefore, stronger identity constraints need to be applied to that perspective during the optimization process.

[0162] After obtaining the degree of identity leakage from all perspectives, the weight of each perspective in the optimization process can be further calculated. Specifically, each perspective can be normalized according to its degree of identity leakage to obtain the corresponding perspective weight. Illustratively, the weight of the v-th perspective can be expressed as:

[0163]

[0164] in, This represents the optimization weight for the i-th viewpoint. This indicates the degree to which the identity of the person in question is revealed. This represents the total number of viewpoints. Through the normalization calculation described above, the sum of the weights of all viewpoints can be made equal to 1. When the facial features in a particular viewpoint change significantly, the corresponding level of identity leakage will be greater, thus earning it a higher weight in the weight calculation.

[0165] This method automatically strengthens identity constraints at key perspectives during the optimization process. Furthermore, additional enhancement mechanisms can be applied to perspectives with a high degree of identity leakage. For example, when the identity leakage level of a certain perspective exceeds a preset threshold, its weight in the optimization process can be further increased, thereby improving the identity protection effect. Through the above multi-perspective calculation process, the weight of each perspective in the optimization process can be obtained.

[0166] Finally, the identity losses from each perspective can be weighted and fused based on these weights to obtain the final multi-perspective identity protection loss. Let the identity loss from the v-th perspective be... Visual feature difference loss is The loss of multi-perspective identity protection can then be expressed as:

[0167]

[0168] in This indicates the loss of multi-perspective identity protection. This represents the weight of the v-th viewpoint. This indicates a loss of identity consistency. Indicates loss of visual feature differences. represents the weighting coefficient, and N represents the total number of viewpoints. This multi-view weighted loss function allows for focused constraints on viewpoints with significant identity changes during the optimization process, thereby further improving the preservation of character identities.

[0169] In 3D scene editing tasks, facial regions from different viewpoints may have varying degrees of importance. For example, in a frontal view, the facial structure is usually more complete, making this viewpoint more important for identity recognition. The multi-view identity weighting method proposed in this embodiment can automatically identify these key viewpoints and assign them higher weights during the optimization process, thereby maximizing the consistency of character identity while ensuring editing quality.

[0170] In one alternative implementation, such as Figure 2 As shown, the 3D scene editing loss is calculated based on a set of multi-view scene images and a set of edited multi-view images. The 3D scene editing loss and the multi-view identity protection loss are then jointly optimized to obtain the optimization result, which may include:

[0171] S201: Extract the original image features of the multi-view scene image set and the edited image features of the edited multi-view image set respectively through an image feature extraction network;

[0172] S202: Calculate the feature distance between the original image features and the edited image features to obtain the 3D scene editing loss;

[0173] S203: Combine the 3D scene editing loss with the multi-view identity protection loss to construct a joint optimization objective function;

[0174] S204: The gradient descent method is used to iteratively calculate the joint optimization objective function to obtain the optimization result.

[0175] Specifically, the foregoing embodiments introduce methods for 3D Gaussian point cloud scene representation, multi-view face detection and region segmentation, identity Gaussian mapping, text-driven 3D scene editing, and the design of an identity consistency constraint loss function and a multi-view identity weighted optimization method. These methods can achieve 3D scene representation, identity region localization, and identity protection constraints, respectively.

[0176] However, in actual 3D scene editing, the aforementioned operations need to work collaboratively through a unified optimization process to ensure both editing quality and character identity protection. Therefore, this embodiment proposes a joint optimization method for 3D Gaussian point cloud scenes. By constructing a unified optimization objective function, the parameters of the 3D Gaussian point cloud scene are updated, thereby achieving text-driven identity-protected 3D scene editing.

[0177] Schematic illustration: First, an image feature extraction network can be used to extract the original image features from a multi-view scene image set and the edited image features from a multi-view image set after editing. This network can be constructed based on a deep learning neural network. Then, the feature distance between the original and edited image features can be calculated. This feature distance quantifies the difference between the current rendering result of the 3D Gaussian point cloud and the target edited result. This difference is used as the 3D scene editing loss, which can constrain the parameter update direction of the 3D Gaussian point cloud, making the rendering result gradually approach the edited target image. This feature distance can be Euclidean distance.

[0178] Furthermore, after obtaining the original 3D Gaussian point cloud scene parameters, the edited multi-view image set, the multi-view scene image set, the 3D scene editing loss, and the multi-view identity protection loss, a unified optimization objective function can be constructed. Illustratively, in a 3D scene editing task, the system needs to ensure that the editing result conforms to the user's text description while maintaining the consistency of the character's identity. Therefore, in this invention, the 3D scene editing loss and the multi-view identity protection loss can be jointly modeled to form a unified optimization objective.

[0179] Specifically, the constructed joint optimization objective function can be expressed as:

[0180]

[0181] in This represents the total optimization loss. Indicates the editing loss of a 3D scene. This indicates a loss of identity consistency. Indicates loss of visual feature differences. , This represents the weighting coefficient.

[0182] In the above loss functions, the editing loss is mainly used to ensure that the generated image conforms to the text description entered by the user; the identity consistency loss is used to constrain the consistency of the identity features of the person before and after editing; and the visual feature difference loss is used to constrain changes in image structure, thereby further maintaining the stability of the person's facial structure.

[0183] After constructing the objective function, the parameters of the 3D Gaussian point cloud scene can be updated based on the loss function. In the 3D Gaussian point cloud representation, each Gaussian point typically contains the following parameters: Gaussian point spatial location, Gaussian point color attribute, Gaussian point opacity parameter, and Gaussian point covariance matrix.

[0184] During 3D scene editing, the scene structure and appearance can be modified by adjusting the aforementioned parameters. For example, adjusting the Gaussian point color parameter can change the appearance of an object, while adjusting the Gaussian point position parameter can change its spatial structure. In each optimization iteration, a corresponding multi-view rendering image can be generated based on the current 3D Gaussian point cloud scene parameters. Subsequently, the rendered image is compared with the edited image generated by text-driven editing to calculate the corresponding 3D scene editing loss. Simultaneously, the identity consistency constraint module can be used to calculate the facial feature differences between the images before and after editing, and the corresponding identity constraint loss can be calculated based on the multi-view identity weighting method.

[0185] After obtaining the aforementioned losses, the parameters of the 3D Gaussian point cloud can be optimized using gradient descent based on the joint optimization objective function. For example, gradient descent-based optimization methods can be used to update the Gaussian point parameters. Let the current optimization iteration number be... The parameters of the 3D Gaussian point cloud are: Then the parameters for the next iteration can be expressed as:

[0186]

[0187] in, Indicates the current Gaussian point parameters. This represents the updated Gaussian point parameters. Indicates the learning rate. This represents the gradient of the loss function.

[0188] Through the above optimization process, the corresponding optimization results can be output, which are the parameter update gradients and optimal adjustment directions after the joint optimization objective function has converged iteratively. Based on these optimization results, the parameters of the 3D Gaussian point cloud scene can be gradually adjusted so that the scene rendering results meet both the requirements of text editing and maintain consistency in the character's identity.

[0189] In one optional implementation, after completing the joint optimization of the 3D Gaussian point cloud scene, based on the optimization results output by the above embodiments, scene parameters such as spatial location, color attributes, opacity, and covariance matrix corresponding to the 3D Gaussian point cloud scene data can be updated to obtain updated 3D Gaussian point cloud scene parameters. Finally, the edited 3D Gaussian point cloud scene parameters that maintain consistency with the character's identity are output. Subsequently, these parameters can be used to perform multi-view rendering of the scene, generating a 3D scene editing result that matches the user's text prompts. Furthermore, the 3D Gaussian point regions related to the character's identity remain stable, preventing erroneous modification of the character's facial structure during editing. This satisfies scene editing requirements while maintaining stable facial identity features. In addition, in the final generated 3D scene, other non-identity areas can still be edited based on the user's input text prompts, thus achieving a 3D scene editing effect that satisfies scene editing requirements while maintaining consistency with the character's identity.

[0190] Based on the same inventive concept, this application also provides a face identity protection system for 3D Gaussian point cloud scene editing, which implements the face identity protection method for 3D Gaussian point cloud scene editing described above. The solution provided by this system is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the face identity protection system for 3D Gaussian point cloud scene editing provided below can be found in the limitations of the face identity protection method for 3D Gaussian point cloud scene editing described above, and will not be repeated here.

[0191] In one exemplary embodiment, such as Figure 3 As shown, a face identity protection system 300 for editing 3D Gaussian point cloud scenes is provided, including:

[0192] The 3D scene acquisition module 301 is used to acquire the 3D Gaussian point cloud scene data to be edited, and load the camera intrinsic and extrinsic parameter matrices corresponding to the 3D Gaussian point cloud scene data; and perform multi-view rendering based on the 3D Gaussian point cloud scene data, camera intrinsic and extrinsic parameter matrices to obtain a set of multi-view scene images.

[0193] The face detection module 302 is used to detect face regions in a collection of multi-view scene images and obtain face region masks from each view.

[0194] The identity region mapping module 303 is used to back-project two-dimensional face pixels onto three-dimensional space based on face region masks from various viewpoints, camera intrinsic and extrinsic matrix, to obtain projection results. Based on the projection results, the module filters to obtain a set of target three-dimensional Gaussian points associated with the person's identity.

[0195] The scene editing module 304 is used to edit the multi-view scene image set according to the text editing instructions input by the user, and obtain the edited multi-view image set by using a diffusion editing model;

[0196] The identity constraint module 305 is used to: extract features from the face regions of the multi-view scene image set and the edited multi-view image set based on the face regions corresponding to the target three-dimensional Gaussian point set, respectively, to obtain the first face identity feature and the second face identity feature, and calculate the identity consistency loss and visual feature difference loss based on the first face identity feature and the second face identity feature.

[0197] The degree of identity leakage from each perspective is calculated based on the degree of change between the first and second facial identity features. The degree of identity leakage from each perspective is then normalized to generate weights for multiple perspectives. The weights of multiple perspectives are then used to weight and fuse the identity consistency loss and visual feature difference loss to obtain the multi-perspective identity protection loss.

[0198] The optimization and update module 306 is used to calculate the 3D scene editing loss based on the multi-view scene image set and the edited multi-view image set; jointly optimize the 3D scene editing loss and the multi-view identity protection loss to obtain the optimization result; based on the optimization result, update the scene parameters corresponding to the 3D Gaussian point cloud scene data and output the final 3D Gaussian point cloud scene parameters.

[0199] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for protecting facial identity in 3D Gaussian point cloud scene editing, characterized in that, The method includes: Obtain the 3D Gaussian point cloud scene data to be edited, and load the camera intrinsic and extrinsic parameter matrices corresponding to the 3D Gaussian point cloud scene data; perform multi-view rendering based on the 3D Gaussian point cloud scene data, the camera intrinsic and extrinsic parameter matrices to obtain a multi-view scene image set; Face region detection is performed on the multi-view scene image set to obtain face region masks for each view; based on the face region masks for each view, the camera intrinsic parameter matrix and the extrinsic parameter matrix, the two-dimensional face pixels are back-projected into three-dimensional space to obtain the projection result; the projection result is then filtered to obtain a set of target three-dimensional Gaussian points associated with the person's identity. Based on the text editing instructions input by the user, the multi-view scene image set is edited using a diffusion editing model to obtain an edited multi-view image set; based on the face region corresponding to the target 3D Gaussian point set, features are extracted from the face regions of the multi-view scene image set and the edited multi-view image set to obtain a first face identity feature and a second face identity feature; based on the first face identity feature and the second face identity feature, identity consistency loss and visual feature difference loss are calculated. Based on the degree of change between the first and second facial identity features from each perspective, the degree of identity leakage from each perspective is calculated; the degree of identity leakage from each perspective is normalized to generate weights for multiple perspectives; the identity consistency loss and the visual feature difference loss are weighted and fused using the weights of the multiple perspectives to obtain the multi-perspective identity protection loss. The 3D scene editing loss is calculated based on the multi-view scene image set and the edited multi-view image set; the 3D scene editing loss and the multi-view identity protection loss are jointly optimized to obtain the optimization result; based on the optimization result, the scene parameters corresponding to the 3D Gaussian point cloud scene data are updated, and the final 3D Gaussian point cloud scene parameters are output.

2. The method according to claim 1, characterized in that, The projection result is a set of points in three-dimensional space; The method involves back-projecting two-dimensional face pixels into three-dimensional space based on the face region masks from each viewpoint, the camera intrinsic matrix, and the extrinsic matrix, to obtain projection results. These projection results are then filtered to obtain a set of target three-dimensional Gaussian points associated with the person's identity, including: Using the camera intrinsic parameter matrix and the extrinsic parameter matrix, the three-dimensional rays corresponding to the two-dimensional face pixels corresponding to the face region mask of each viewpoint are calculated to obtain a set of three-dimensional rays. The three-dimensional spatial position corresponding to the three-dimensional ray is determined by combining the geometric structure of the three-dimensional Gaussian point cloud scene data, and the set of three-dimensional spatial points is obtained; Calculate the Euclidean distance between the set of three-dimensional spatial points and the three-dimensional Gaussian points in the three-dimensional Gaussian point cloud scene data to obtain the distance calculation result; Based on the covariance parameter of the three-dimensional Gaussian points, an adaptive distance threshold is determined. Based on the adaptive distance threshold and the distance calculation result, the three-dimensional Gaussian points in the three-dimensional Gaussian point cloud scene data are initially screened to obtain a set of face-related Gaussian points corresponding to each viewpoint. The set of face-related Gaussian points corresponding to each viewpoint is merged to obtain a fused set of identity-related Gaussian points. Calculate the number of times each Gaussian point in the identity-associated Gaussian point set is identified as a face region in multiple views, and assign a weight to each Gaussian point in the identity-associated Gaussian point set based on the number of times; The identity-associated Gaussian point set is weighted and filtered according to the weights to obtain the target three-dimensional Gaussian point set.

3. The method according to claim 1, characterized in that, The step of calculating identity consistency loss and visual feature difference loss based on the first facial identity features and the second facial identity features includes: Calculate the cosine similarity between the first facial identity feature and the second facial identity feature, and calculate the identity consistency loss based on the cosine similarity. The first visual features of the multi-view scene image set and the second visual features of the edited multi-view image set are extracted by a visual feature extraction network. Calculate the L2 distance between the first visual feature and the second visual feature to obtain the visual feature difference loss.

4. The method according to claim 1, characterized in that, The degree of identity leakage from each perspective is calculated based on the degree of change between the first facial identity features and the second facial identity features from each perspective. Based on the degree of identity leakage from each perspective, normalization is performed to generate weights for multiple perspectives, including: Calculate the change in similarity between the first facial identity feature and the second facial identity feature from each viewpoint, and determine the change in similarity as the degree of identity leakage corresponding to that viewpoint; The summation of the degree of identity leakage from each of the aforementioned perspectives yields the total degree of identity leakage. The weight corresponding to the single-viewpoint's degree of identity leakage is calculated by ratioing it to the sum of the degree of identity leakage.

5. The method according to claim 1, characterized in that, The 3D scene editing loss is calculated based on the set of multi-view scene images and the set of edited multi-view images. The 3D scene editing loss and the multi-view identity protection loss are jointly optimized to obtain the optimization result, including: The original image features of the multi-view scene image set and the edited image features of the edited multi-view image set are extracted by an image feature extraction network. Calculate the feature distance between the original image features and the edited image features to obtain the 3D scene editing loss; The 3D scene editing loss and the multi-view identity protection loss are combined to construct a joint optimization objective function; The joint optimization objective function is iteratively calculated using the gradient descent method to obtain the optimization result.

6. The method according to claim 1, characterized in that, The step of editing the multi-view scene image set according to the user's input text editing instructions, using a diffusion editing model, to obtain an edited multi-view image set includes: The text editing instructions input by the user are encoded to obtain a text semantic feature vector; The text semantic feature vector is input as a conditional constraint into the diffusion editing model. The diffusion editing model performs multi-view consistent denoising editing on the multi-view scene image set to generate an edited multi-view image set that matches the text editing instructions.

7. The method according to claim 1, characterized in that, The multi-view identity protection loss is calculated using the following formula: in, For the loss of multi-view identity protection, For the first The weights mentioned from each perspective, For the first The identity consistency loss from each perspective, where λ is a preset weighting coefficient. For the first The visual feature difference loss from each viewpoint, Here, N represents the viewpoint number, and N represents the total number of viewpoints.

8. The method according to claim 1, characterized in that, The visual feature difference loss is calculated using the following formula: in, For the visual feature difference loss, For visual feature extraction function, The set of multi-view scene images, The edited multi-view image set, This refers to the 2-norm operation.

9. The method according to claim 3, characterized in that, The identity consistency loss is calculated using the following formula: in, For the aforementioned identity consistency loss, This is a function for extracting facial features. The set of multi-view scene images, The edited multi-view image set.

10. A face identity protection system for editing 3D Gaussian point cloud scenes, characterized in that, The system includes: The 3D scene acquisition module is used to acquire the 3D Gaussian point cloud scene data to be edited, and load the camera intrinsic and extrinsic parameter matrices corresponding to the 3D Gaussian point cloud scene data; and perform multi-view rendering based on the 3D Gaussian point cloud scene data, the camera intrinsic and extrinsic parameter matrices to obtain a multi-view scene image set. The face detection module is used to detect face regions in the multi-view scene image set and obtain face region masks for each view. The identity region mapping module is used to back-project two-dimensional face pixels onto three-dimensional space based on the face region mask of each viewpoint, the camera intrinsic parameter matrix and the extrinsic parameter matrix, to obtain the projection result, and to filter according to the projection result to obtain a set of target three-dimensional Gaussian points associated with the person's identity. The scene editing module is used to edit the multi-view scene image set according to the text editing instructions input by the user, and obtain the edited multi-view image set by using a diffusion editing model; The identity constraint module is used to: extract features from the face regions of the multi-view scene image set and the edited multi-view image set based on the face regions corresponding to the target three-dimensional Gaussian point set, respectively, to obtain the first face identity feature and the second face identity feature, and calculate the identity consistency loss and visual feature difference loss based on the first face identity feature and the second face identity feature. Based on the degree of change between the first and second facial identity features from each perspective, the degree of identity leakage from each perspective is calculated; the degree of identity leakage from each perspective is normalized to generate weights for multiple perspectives; the identity consistency loss and the visual feature difference loss are weighted and fused using the weights of the multiple perspectives to obtain the multi-perspective identity protection loss. The optimization and update module is used to calculate the 3D scene editing loss based on the multi-view scene image set and the edited multi-view image set; jointly optimize the 3D scene editing loss and the multi-view identity protection loss to obtain the optimization result; and update the scene parameters corresponding to the 3D Gaussian point cloud scene data based on the optimization result to output the final 3D Gaussian point cloud scene parameters.

Images