Static human body reconstruction method, apparatus, and electronic device

Abstract

The application provides a static human body reconstruction method and device and electronic equipment, and relates to the technical field of computer vision, and the method comprises the following steps: acquiring a multi-view image set of a target human body collected by a multi-camera array, wherein the multi-view image set comprises images collected by a long-focus lens and images collected by a short-focus lens; preprocessing the multi-view image set to obtain a pure foreground mask corresponding to each image in the multi-view image set, an accurate camera pose, and a human body sparse point cloud; based on the multi-view image set, the pure foreground mask corresponding to each image, the camera pose corresponding to each image, and the human body sparse point cloud corresponding to each image, a multi-view joint optimization strategy is used to optimize a three-dimensional Gaussian splash model to obtain a three-dimensional Gaussian representation model of the target human body. The application obtains a more detailed and realistic static human body reconstruction result, and improves the effect of static human body reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a static human body reconstruction method, apparatus, and electronic device. Background Technology

[0002] High-quality static human body reconstruction has long been a hot research topic in computer vision and computer graphics, holding significant academic importance and facing broad industrial applications. For example, in virtual reality (VR), augmented reality (AR), digital humans, film and television production, the gaming industry, and remote interaction scenarios, high-fidelity static human body models can significantly enhance immersion and interactive experiences. Therefore, how to efficiently and accurately reconstruct detailed and structurally complete static human body models remains a core problem that urgently needs to be solved in this field.

[0003] Reconstructing a high-quality static human body relies on efficient and high-quality data capture and high-fidelity, real-time reconstruction rendering algorithms. In terms of data acquisition, most existing human body datasets utilize multi-camera arrays for full-body, short-focus, fixed-point photography. While this method covers the entire field of view and yields relatively complete reconstruction results, it still falls short in detail for the upper body and head regions, which are of utmost importance in digital human applications, requiring improvement in realism and accuracy. Furthermore, in terms of data preprocessing, existing methods typically use foreground human images obtained through segmentation algorithms. However, due to the inherent inaccuracies in segmentation, background areas are often mistakenly included in the foreground. Directly inputting these "incorrectly segmented" results into the reconstruction algorithm can easily lead to background interference in the model, thus affecting the reconstruction quality.

[0004] In terms of reconstruction algorithms, with the development of 3D representation methods, Three-Dimensional Gaussian Splatting (3DGS) has gradually become the mainstream solution for static scene and human body modeling. Compared with mesh-based methods, 3DGS-based methods offer faster reconstruction speeds, achieve photorealistic results without complex material modeling, and can reconstruct areas where coarse meshes struggle to express geometric details, significantly shortening the production cycle. Compared with Neural Radiance Fields (NeRF) based on implicit representation, 3DGS also has significant advantages in real-time rendering efficiency and further improves reconstruction quality. However, when the original 3DGS algorithm and its subsequent improvements are directly applied to static human body reconstruction, the results are unsatisfactory: due to segmentation errors and multi-view consistency issues, existing methods often produce black artifact edges and incorrect background noise in the results, leading to a decrease in the quality of the static human body reconstruction model and thus affecting the viewing experience in VR.

[0005] Therefore, an effective technical solution is urgently needed to solve the above-mentioned technical problems. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a static human body reconstruction method, apparatus, and electronic device, which achieves more refined and realistic static human body reconstruction results and improves the effectiveness of static human body reconstruction.

[0007] In a first aspect, the present invention provides a static human body reconstruction method, the method comprising the following steps: Acquire a multi-view image set of a target human body captured by a multi-camera array; the multi-view image set includes images captured by a telephoto lens and images captured by a short-focus lens; The multi-view image set is preprocessed to obtain the clean foreground mask, accurate camera pose, and sparse human point cloud corresponding to each image in the multi-view image set. Based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is used to optimize the three-dimensional Gaussian splash model and reconstruct the three-dimensional Gaussian representation model of the target human body.

[0008] According to a static human body reconstruction method provided by the present invention, the method involves optimizing a 3D Gaussian splash model using a multi-view joint optimization strategy based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, to reconstruct a 3D Gaussian representation model of the target human body. The method includes: In any iteration of optimization, the following optimization steps are performed: A primary view camera is selected in the current iteration; based on the geometric observation relationship and viewpoint complementarity between the primary view camera and the other cameras in the multi-view human image set (excluding the primary view camera), at least one associated camera is dynamically determined for the primary view camera from the multi-view human image set; image data corresponding to the primary view camera and image data corresponding to each associated camera are determined from the multi-view image set, the clean foreground mask corresponding to each image, and the camera pose corresponding to each image; based on the image data corresponding to the primary view camera and the image data corresponding to each associated camera, the model parameters of the 3D Gaussian splash model are jointly updated using gradients to obtain the updated model parameters under the current iteration. The updated model parameters under the current iteration optimization are used as the model parameters in the next iteration optimization, and the optimization steps are continued until the number of iterations is reached to obtain the three-dimensional Gaussian representation model of the target human body.

[0009] According to a static human body reconstruction method provided by the present invention, the step of dynamically determining at least one associated camera for the main-view camera from the multiple cameras corresponding to the multi-view human body image set, based on the geometric observation relationship and viewpoint complementarity between the main-view camera and the other cameras in the multi-view human body image set (excluding the main-view camera), includes: Based on the number of 3D points jointly observed by the main-view camera and each of the other cameras, the correlation strength between the main-view camera and each of the other cameras is calculated. Based on the aforementioned correlation strengths, a subset of candidate correlated cameras is selected for the main-view camera; Determine the angle between the normal direction of the main view camera and the normal directions of each camera in the candidate associated camera subset; Each associated camera is determined from the subset of candidate associated cameras based on the size of each included angle.

[0010] According to a static human body reconstruction method provided by the present invention, the method involves optimizing a 3D Gaussian splash model using a multi-view joint optimization strategy based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, to reconstruct a 3D Gaussian representation model of the target human body. The method includes: Based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, the multi-view joint optimization strategy and the joint supervision mechanism of transparency mask supervision and random background perturbation are used to optimize the three-dimensional Gaussian splash model and reconstruct the three-dimensional Gaussian representation model of the target human body.

[0011] According to a static human body reconstruction method provided by the present invention, the joint supervision mechanism of transparency mask supervision and random background perturbation includes: Using the clean foreground mask corresponding to each of the images as a supervision signal, the first loss is calculated for the opacity image generated by the model rendering. The random background color of each of the images is dynamically generated. The random background color of each of the images is combined with the image to form a new target image using the clean foreground mask corresponding to each of the images. Based on the new target image, a second loss is calculated for the color image generated by the model rendering. Based on the first loss and the second loss, determine the total loss function; Based on the total loss function, the model parameters of the three-dimensional Gaussian splash model are adjusted.

[0012] According to a static human body reconstruction method provided by the present invention, the step of using the clean foreground mask corresponding to each of the images as a supervision signal to calculate a first loss on the opacity image generated by the model rendering includes: Using the clean foreground mask corresponding to each image as a supervision signal, the first loss of the opacity image is calculated using the binary cross-entropy loss function.

[0013] According to a static human body reconstruction method provided by the present invention, the preprocessing of the multi-view image set to obtain a clean foreground mask, accurate camera pose, and sparse human body point cloud corresponding to each image in the multi-view image set further includes: Perform foreground segmentation based on background subtraction on each of the images in the multi-view image set to obtain each initial mask; The maximum connected component is extracted from each of the initial masks to obtain the clean foreground mask corresponding to each of the images; By combining the prior pose information of the camera with the clean foreground mask corresponding to each image, the multi-view image set is bundled and adjusted to obtain the pose of each camera and the sparse point cloud of each human body.

[0014] According to a static human body reconstruction method provided by the present invention, the method further includes: The target camera pose is input into the three-dimensional Gaussian representation model of the target human body; the three-dimensional Gaussian representation model of the target human body is composed of multiple three-dimensional Gaussian ellipsoids, each of which has the attributes of position, covariance, opacity and spherical harmonic coefficients; Based on the target camera pose, the three-dimensional Gaussian ellipsoids in the three-dimensional Gaussian representation model of the target human body are filtered and sorted. Based on the sorted three-dimensional Gaussian ellipsoids, a two-dimensional rendered image of the target human body under the current rendering viewpoint corresponding to the pose of the target camera is generated through parallel rasterization and alpha hybrid synthesis.

[0015] Secondly, the present invention also provides a static human body reconstruction device, which includes the following modules: The acquisition module is used to acquire a multi-view image set of the target human body captured by a multi-camera array; the multi-view image set includes images captured by a telephoto lens and images captured by a short-focus lens. The data preprocessing module is used to preprocess the multi-view image set to obtain the clean foreground mask, accurate camera pose, and human sparse point cloud corresponding to each image in the multi-view image set. The reconstruction module is used to optimize the three-dimensional Gaussian splash model based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the human body sparse point cloud corresponding to each image, and to reconstruct the three-dimensional Gaussian representation model of the target human body.

[0016] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the static human body reconstruction method as described above.

[0017] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the static human body reconstruction method as described above.

[0018] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the static human body reconstruction method as described above.

[0019] This invention provides a static human body reconstruction method, apparatus, and electronic device. First, a multi-view image set of the target human body is acquired using a multi-camera array, including images acquired through a telephoto lens and images acquired through a short-focus lens. Then, the multi-view image set is preprocessed to obtain a clean foreground mask, precise camera pose, and sparse point cloud of the human body for each image. Next, based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is used to optimize the three-dimensional Gaussian splash model, thereby reconstructing a three-dimensional Gaussian representation model of the target human body.

[0020] The multi-view image set acquired in this invention includes images captured by telephoto lenses and images captured by short-focus lenses, which can cover the full range of perspectives and details of the target human body. In the data preprocessing stage, mask extraction and precise camera pose calibration are introduced, which significantly reduces the impact of segmentation errors and calibration deviations on the reconstruction model. The clean foreground mask and precise camera pose are input together with the original image into the 3D Gaussian splash model for model optimization, which improves the robustness and reliability of the input data from the source. In terms of reconstruction algorithm, based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is adopted to optimize the 3D Gaussian splash model, and a 3D Gaussian representation model of the target human body is reconstructed. This improves the overall consistency of the human body model between different perspectives, enhances the ability to restore local details, obtains more refined and realistic static human body reconstruction results, and improves the effect of static human body reconstruction. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the method flow for static human body reconstruction provided by existing technology.

[0023] Figure 2 This is one of the flowcharts of the static human body reconstruction method provided by the present invention.

[0024] Figure 3 This is the second flowchart of the static human body reconstruction method provided by the present invention.

[0025] Figure 4 This is a flowchart illustrating the data preprocessing method provided by the present invention.

[0026] Figure 5 This is a schematic diagram of the rendering result of the three-dimensional Gaussian representation model of the target human body provided by the present invention.

[0027] Figure 6 This is a schematic diagram of the static human body reconstruction device provided by the present invention.

[0028] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first node can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0031] To more clearly understand the various embodiments provided by the present invention, the technical content involved in the present invention will first be described as follows: With the rapid development of virtual reality (VR), augmented reality (AR), and the digital human industry, high-quality static human reconstruction plays a crucial role in various application areas such as immersive interaction, film and television production, and game modeling. Commonly used solutions in academia and industry include... Figure 1 As shown, Figure 1 This is a schematic diagram of the method flow for static human body reconstruction provided by existing technology, such as... Figure 1 As shown, the method includes: Data acquisition: synchronous, 360-degree, high-resolution acquisition.

[0032] Data processing: including calibration data.

[0033] After initialization, the algorithm is used for reconstruction, that is, to reconstruct a high-quality static human body.

[0034] VR applications: such as rendering and viewing.

[0035] However, existing technologies have shortcomings in data acquisition, segmentation accuracy, and the robustness of reconstruction algorithms, and are prone to problems such as blurred human body edges, background noise interference, and loss of detail.

[0036] To address the aforementioned technical issues, researchers have used mask transparency constraints in their 3DGS-Avatar work to suppress background noise, which has improved the background problem to some extent, but has also resulted in blurring of the human body outline and visible black artifacts at the edges of the reconstructed results.

[0037] To address the shortcomings of existing technologies, this invention provides a static human body reconstruction method based on Gaussian characterization.

[0038] The following is combined with Figures 2 to 7 The present invention describes a static human body reconstruction method, apparatus, and electronic device.

[0039] Figure 2 This is one of the flowcharts illustrating the static human body reconstruction method provided by the present invention, such as... Figure 2 As shown, the method includes the following: Step 201: Obtain a multi-view image set of the target human body acquired by a multi-camera array; the multi-view image set includes images acquired by a telephoto lens and images acquired by a short-focus lens.

[0040] First, a multi-view image set is acquired, that is, the original multi-view image set of the target human body. The original multi-view image set includes images acquired through a telephoto lens and images acquired through a short-focus lens.

[0041] In practical applications, by using telephoto and telephoto lenses in a multi-camera array to take pictures separately, one can obtain images with both global information and detailed images with clear textures.

[0042] Step 202: Preprocess the multi-view image set to obtain the clean foreground mask, accurate camera pose, and sparse human point cloud corresponding to each image in the multi-view image set.

[0043] Specifically, it should be noted that existing methods directly use segmented foreground images for training, which often affects the reconstruction quality due to segmentation errors and inaccurate camera pose estimation.

[0044] This invention proposes a more robust data preprocessing workflow, including background removal based on the principle of background subtraction, extraction of the maximum connected component from the segmented foreground mask to eliminate parts that the algorithm mistakenly identifies as foreground human figures, and camera pose calibration using camera priors.

[0045] Camera pose calibration is performed on each input multi-view image of the human body, such as the original red, green, and blue (RGB) image, and three key parameters are calculated: 1. External references: Rotation: The orientation of the camera in three-dimensional space (usually represented by a rotation matrix or quaternion).

[0046] Translation: The specific position of the camera in three-dimensional space (represented by a three-dimensional vector).

[0047] Together, these two elements define the camera pose, answering the question, "From what position and from what angle was this photo taken?"

[0048] 2. Internal Reference: The inherent properties of a camera, such as focal length, optical center, and lens distortion parameters.

[0049] 3. Sparse point cloud of the human body: The three-dimensional feature points of the human body represent the preliminary three-dimensional structure of the human body.

[0050] After data preprocessing, the clean foreground mask, accurate camera pose, and sparse human point cloud corresponding to each image in the multi-view image set are obtained. Then, the original RGB images in the multi-view image set, the clean foreground mask, accurate camera pose, and sparse human point cloud corresponding to each image are used as input data.

[0051] The key improvement in this invention lies in the fact that instead of simply feeding the raw image to the calibration tool, such as Constructing Large-scale Maps with Point Clouds (COLMAP), it inputs a refined, clean foreground mask and the known prior pose of the camera system. This provides strong clues and constraints during the complex calculation process, guiding the algorithm to avoid pitfalls such as repeated textures in clothing, thus obtaining more accurate and reliable calibration results than the standard procedure.

[0052] Step 203: Based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the human body sparse point cloud corresponding to each image, the three-dimensional Gaussian splash model is optimized by adopting a multi-view joint optimization strategy to reconstruct the three-dimensional Gaussian representation model of the target human body.

[0053] Specifically, after obtaining high-quality data through preprocessing steps as the initial input for the reconstruction algorithm, this invention proposes a multi-view joint optimization mechanism to enhance details, especially training data that is compatible with both long and short focal lengths, while also enhancing the geometric consistency between viewpoints, making the reconstructed static human body more detailed and realistic.

[0054] Specifically, the problem that the multi-perspective joint optimization strategy aims to solve is: In standard 3D reconstruction optimization, including the original 3DGS, each iteration typically randomly samples one or a batch of independent viewpoints for loss calculation and parameter updates. This method has two key problems: ① Optimization ambiguity: The projection information (color, position) of a 3D point under a single viewpoint may correspond to multiple different 3D states. Constrained by only one viewpoint, the model is prone to getting trapped in local optima, leading to blurred details, geometric drift, or "floating object" artifacts. ② Viewpoint inconsistency: Because each optimization only focuses on the fit of the current viewpoint, the model may try to "please" each independent viewpoint, resulting in contradictory 3D geometry or appearances under different viewpoints, leading to multi-viewpoint inconsistency in the final model.

[0055] The core idea of ​​the multi-view joint optimization strategy is to actively and selectively allow observation signals from multiple different, geometrically complementary viewpoints to simultaneously constrain the model parameters of the same local 3D region when optimizing a 3D Gaussian representation model. Essentially, this strategy introduces additional, cross-view geometric consistency constraints during the optimization process. By constructing a dynamic "view collaboration network," the model is forced to learn a self-consistent 3D representation across all associated viewpoints, thereby reducing ambiguity. The cross-observation from multiple viewpoints, like triangulation, can more accurately locate the position and shape of 3D points. Furthermore, complementary viewpoints can provide lighting and texture information from different angles on the same surface region, helping to restore finer geometric wrinkles and texture details. It is also more robust to areas with local occlusion or repetitive textures (such as the front and back of clothing).

[0056] In practical applications, model optimization includes the following steps 1-4: Step 1: Establish the viewpoint association network (offline / initialization phase) Input: All precisely calibrated camera poses.

[0057] The correlation strength of each camera is calculated, which is equivalent to calculating the number of 3D Gaussian points jointly observed by every two cameras. This constructs a network with cameras as nodes and common observation points as connecting edges. The more common observation points, the larger the shared area seen by the two cameras, and the stronger the potential correlation of their optimization constraints.

[0058] Objective: To identify which camera pairs have sufficient "common topics" (overlapping observation areas) to serve as candidate pairs for subsequent joint optimization.

[0059] Step 2: Dynamically select the "best partner" perspective for each main viewpoint (in training iterations) After selecting a main-view camera to be optimized in each iteration, a smart selection is performed: Preliminary screening: From all cameras, select the top K (e.g., 10) cameras with the highest correlation strength with the main viewpoint to form a "candidate correlation set". This ensures that the selected cameras and the main viewpoint have a sufficient common observation area.

[0060] Geometric complementarity selection: From the "candidate association set", select the camera with the largest angle with the normal direction of the main view camera as the "association camera" for this iteration.

[0061] Step 3: Perform joint optimization In a single forward propagation, predicted images (RGB and opacity) of the main viewpoint and a selected associated viewpoint are rendered simultaneously. Losses for both views are calculated concurrently, including color loss, Structural Similarity Index Measure (SSIM) loss, and opacity masking loss. The losses from both views are combined, and then a unified backpropagation is performed to update all parameters of the 3D Gaussian model.

[0062] Effect: This parameter update simultaneously satisfied constraints from two geometrically complementary viewpoints. The model was forced to adjust the position, shape, and color of the Gaussian points to present the correct projections from both significantly different viewpoints. This directly enhanced the accuracy and consistency of the 3D structure.

[0063] Step 4, Dynamics and Cycles In the next training iteration, another main viewpoint will be randomly selected, and steps 2 and 3 will be repeated for it to dynamically select a new best partner.

[0064] As training progresses, the entire 3D Gaussian model is repeatedly refined and corrected by a dynamically paired "constraint net" composed of all cameras, eventually converging to a highly consistent and detailed state.

[0065] The method provided in this embodiment of the invention first acquires a multi-view image set of the target human body acquired by a multi-camera array, the multi-view image set including images acquired by a telephoto lens and images acquired by a short-focus lens; then, the multi-view image set is preprocessed to obtain a clean foreground mask, accurate camera pose, and sparse point cloud of the human body corresponding to each image in the multi-view image set; further, based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is used to optimize the three-dimensional Gaussian splash model to reconstruct a three-dimensional Gaussian representation model of the target human body.

[0066] The multi-view image set acquired in this invention includes images captured by telephoto lenses and images captured by short-focus lenses, which can cover the full range of perspectives and details of the target human body. In the data preprocessing stage, mask extraction and precise camera pose calibration are introduced, which significantly reduces the impact of segmentation errors and calibration deviations on the reconstruction model. The clean foreground mask and precise camera pose are input together with the original image into the 3D Gaussian splash model for model optimization, which improves the robustness and reliability of the input data from the source. In terms of reconstruction algorithm, based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is adopted to optimize the 3D Gaussian splash model, and a 3D Gaussian representation model of the target human body is reconstructed. This improves the overall consistency of the human body model between different perspectives, enhances the ability to restore local details, obtains more refined and realistic static human body reconstruction results, and improves the effect of static human body reconstruction.

[0067] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0068] According to the static human body reconstruction method provided by the present invention, based on a multi-view image set, a clean foreground mask corresponding to each image, a camera pose corresponding to each image, and a sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is used to optimize the three-dimensional Gaussian splash model to reconstruct a three-dimensional Gaussian representation model of the target human body, including: In any iteration of optimization, the following optimization steps are performed: Select a main-view camera in the current iteration; Based on the geometric observation relationship and view complementarity between the main-view camera and the other cameras in the multi-camera set corresponding to the multi-view human image set (excluding the main-view camera), dynamically determine at least one associated camera from the multi-camera set corresponding to the main-view camera; Determine the image data corresponding to the main-view camera and the image data corresponding to each associated camera from the multi-view image set, the clean foreground mask corresponding to each image, and the camera pose corresponding to each image; Based on the image data corresponding to the main-view camera and the image data corresponding to each associated camera, perform joint gradient update on the model parameters of the 3D Gaussian splash model to obtain the updated model parameters under the current iteration of optimization; The updated model parameters in the current iteration are used as the model parameters in the next iteration. The optimization steps continue until the number of iterations is reached, and a three-dimensional Gaussian representation model of the target human body is obtained.

[0069] Specifically, in some embodiments, step 103, model optimization to reconstruct the three-dimensional Gaussian representation model of the target human body, includes the following steps: In any iteration of optimization, perform the following optimization steps 1-4: Step 1: Select the main view camera in the current iteration optimization. .

[0070] The main view camera corresponds to the viewpoint that is currently being optimized during training. In other words, during the training process, a main view camera is randomly selected in each iteration. Accordingly, the algorithm will use the main view camera. and related cameras Joint optimization was carried out.

[0071] Step 2: Identify the associated camera.

[0072] Based on the geometric observation relationships and perspective complementarity among the cameras other than the main-view camera in the multi-camera set corresponding to the main-view camera, at least one associated camera is dynamically determined from the multi-camera set corresponding to the main-view camera. The geometric observation relationships are determined based on the number of three-dimensional (3D) points jointly observed by each camera, and the perspective complementarity is determined based on the angle between the normal directions of each camera.

[0073] Step 3: Multi-perspective joint optimization.

[0074] Obtain the image data corresponding to the main view camera and the associated cameras. That is, determine the image data corresponding to the main view camera and the image data corresponding to each associated camera from the multi-view image set, the clean foreground mask corresponding to each image, and the camera pose corresponding to each image.

[0075] Subsequently, based on the image data from the main-view camera and the associated camera, the model parameters of the 3D Gaussian splash model are jointly updated using gradients to obtain the updated model parameters under the current iteration of optimization. The parameters jointly optimized include the position p of the 3D Gaussian point, the spherical harmonic function f representing the color of the Gaussian sphere, and the opacity of the Gaussian point. The scale s of the three axes of the Gaussian ellipsoid, and the rotation r of the Gaussian ellipsoid.

[0076] Furthermore, the updated model parameters in the current iteration are used as the model parameters in the next iteration, and the optimization steps continue to be executed until the number of iterations is reached (e.g., 30,000). The model parameters of the last iteration are then fixed to obtain the three-dimensional Gaussian representation model of the target human body.

[0077] It should be noted that if training and testing sets are used for training, the maximum associated camera of a certain camera may not be in the training set. In this case, the search is performed downwards according to the normal angle from large to small until a usable associated camera is found to ensure the continuity of training.

[0078] For example, Figure 3 This is the second flowchart of the static human body reconstruction method provided by the present invention, as shown below. Figure 3 As shown, the method includes: First, data loading and preprocessing are performed to obtain a multi-view human body image set, a clean foreground mask for each image, a camera pose for each image, and a sparse point cloud of the human body.

[0079] Then initialization is performed, that is, parameter initialization of the 3D Gaussian splash model. Further, loop training begins.

[0080] Check if the number of iterations is less than 30000: If the number of iterations is less than 30,000, then execute the following steps in sequence: multi-view selection strategy, (determine) batch size Batchsize=N, generate random background, Gaussian volume rendering, loss calculation, and cumulative loss. Then determine whether the inner loop of the batch has ended. If yes, then execute the following steps in sequence: backpropagation and parameter update, clone splitting, progress update, and iteration count +1. If no, then re-execute the following steps: "(determine) batch size Batchsize=N, generate random background, Gaussian volume rendering, loss calculation, and cumulative loss".

[0081] If the number of iterations is greater than or equal to 30,000, the model is saved and reconstruction is completed.

[0082] The method provided in this invention, through a multi-view joint optimization mechanism, will have richer details in the reconstruction result. At the same time, the model can learn the geometric consistency of the same Gaussian under different views, which can significantly reduce the ambiguity of model optimization.

[0083] According to a static human body reconstruction method provided by the present invention, based on the geometric observation relationship and viewpoint complementarity among the cameras other than the main view camera in the multi-camera set corresponding to the main view camera, at least one associated camera is dynamically determined from the multi-camera set corresponding to the main view camera, including: Based on the number of 3D points jointly observed by the main-view camera and all other cameras, calculate the correlation strength between the main-view camera and each of the other cameras. Based on the correlation strength, a subset of candidate correlated cameras is selected from the main viewpoint camera. Determine the angle between the normal direction of the main view camera and the normal directions of each camera in the candidate associated camera subset; Based on the size of each included angle, each associated camera is determined from the subset of candidate associated cameras.

[0084] Specifically, in some embodiments, the process of dynamically determining at least one associated camera in any iteration is achieved through the following steps: First, based on the number of 3D points jointly observed by the main-view camera and all other cameras, the correlation strength between the main-view camera and each of the other cameras is calculated. To establish the geometric correlation between viewpoints, this invention constructs a correlation matrix based on the number of jointly observed 3D points. (Camera) and correlation strength Defined as: in Represents the cardinality of a set. For camera The observed set of 3D points For camera The set of 3D points observed.

[0085] Furthermore, based on the correlation strength, a subset of candidate correlated cameras is selected from the main viewpoint camera. For example, for each camera... Select the top with the strongest association. One camera (in the implementation of this invention) ) constitute candidate association set .

[0086] Furthermore, in the candidate association set The viewpoint with the strongest geometric complementarity is selected, and its corresponding camera is used as the associated camera. This includes the following steps (1)-(3): For example, (1) first calculate the normal direction of each camera (that is, the normal direction of the camera's shooting direction): Let the training camera set be... Each of the cameras Extrinsic rotation quaternion With displacement vector First, the rotation matrix is ​​obtained through quaternion transformation. Calculate the normal direction for each camera: in, For camera The direction of the normal, For camera The rotation matrix.

[0087] (2) Then determine the angle between the normal direction of the main view camera and the normal direction of each camera in the candidate associated camera subset. The angle between the normal directions of two cameras is calculated as follows: in, Indicates camera Normal direction and camera The angle between the directions of the normals, For camera The direction of the normal, For camera The direction of the normal.

[0088] (3) Finally, based on the size of each included angle, a subset of candidate associated cameras is selected. The associated cameras are determined in the process. For example, the associated camera with the largest included normal angle is selected: in, For camera Associated camera, Indicates camera , This represents a subset of candidate associated cameras.

[0089] The largest included angle signifies the greatest difference in shooting directions between the two cameras. For example, a frontal view and a side view. Such a combination of perspectives provides the most complementary geometric information about the human body surface: the frontal view provides details of the chest, while the side view provides contour and thickness information. Optimizing by combining them most effectively eliminates the ambiguity of a single view and strengthens the constraints of the 3D structure.

[0090] Through the multi-view joint optimization mechanism proposed in this invention, the reconstruction results will have richer details, and the model can learn the geometric consistency of the same Gaussian under different views, which significantly reduces the ambiguity of algorithm optimization.

[0091] According to the static human body reconstruction method provided by the present invention, based on a multi-view image set, a clean foreground mask corresponding to each image, a camera pose corresponding to each image, and a sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is used to optimize the three-dimensional Gaussian splash model to reconstruct a three-dimensional Gaussian representation model of the target human body, including: Based on a multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy and a joint supervision mechanism of transparency mask supervision and random background perturbation are adopted to optimize the 3D Gaussian splash model and reconstruct the 3D Gaussian representation model of the target human body.

[0092] In some embodiments, based on the introduction of a multi-view joint optimization mechanism, the present invention further proposes a Gaussian optimization strategy that combines random background and transparency mask supervision, thereby obtaining a high-quality static human body model with noise-free background and clear contours.

[0093] The joint supervision mechanism of transparency mask supervision and random background perturbation is also known as the transparency mask supervision optimization mechanism combined with the random background optimization mechanism.

[0094] The method provided in this invention, by introducing random background perturbation during training and using transparency masks for supervision, effectively suppresses black artifacts and residual noise that easily appear at the boundary between the foreground and background, thereby significantly improving the contour clarity and rendering quality of the human body model.

[0095] According to the static human body reconstruction method provided by the present invention, a joint supervision mechanism of transparency mask supervision and random background perturbation includes: The clean foreground mask corresponding to each image is used as a supervision signal to calculate the first loss for the opacity image generated by the model rendering. The random background color of each image is dynamically generated. The random background color of each image is combined with the image to form a new target image using the clean foreground mask corresponding to each image. Based on each new target image, the second loss is calculated for the color image generated by the model rendering. Determine the total loss function based on the first loss and the second loss; The model parameters of the 3D Gaussian splash model are adjusted based on the total loss function.

[0096] Specifically, in some embodiments, the joint supervision mechanism of transparency mask supervision and random background perturbation is as follows: 1) Transparency mask supervised optimization mechanism: In order to effectively eliminate background noise and enhance surface details, this invention introduces a transparency mask supervised optimization mechanism.

[0097] Specifically, the imported foreground mask is strictly binarized and denoted as: ,in These are pixel coordinates.

[0098] Each iteration can also generate an opacity image based on rasterization. The value at each pixel represents the result of alpha blending of all Gaussian opacities at the current pixel position.

[0099] Alpha blending is a core technique used in rendering to handle transparency or semi-transparency effects. The alpha channel, usually denoted by α, is the fourth color channel in an image, in addition to the red, green, and blue channels. It stores the opacity information of each pixel (typically ranging from 0.0 to 1.0, where 0 represents complete transparency and 1 represents complete opacity). When rendering a semi-transparent object, its color needs to be blended with the background color. Alpha blending defines the mathematical formula for this blending. For example, the most common "Over" blending operation is: Final Color = (Foreground Color × Foreground Opacity) + (Background Color × (1 - Foreground Opacity)).

[0100] After obtaining the rendered opacity image, this invention designs a method based on... The binary cross-entropy is used as the transparency loss (first loss) to obtain a numerically stable optimization objective, as follows: Let the image size be H×W, then: in, The loss is the transparency mask supervision loss, i.e., the first loss. This is the binarized result of the foreground mask, i.e., the ground truth. These are the predicted values ​​from the model.

[0101] The loss can be stably backpropagated to the opacity parameters of each Gaussian. Through the transparency mask supervised optimization mechanism, the model can effectively distinguish between the foreground and background, significantly reduce background noise, and enhance the representation of surface details.

[0102] 2) Random background optimization mechanism: Relying solely on transparency supervision still results in unclear edges, especially at the boundary between the foreground and background, where semi-transparent artifacts and blurred boundaries are prone to occur.

[0103] To address this issue and further improve contour quality, this invention also introduces a random background enhancement mechanism. During training, a random background color is dynamically generated in each iteration. The ground truth image used for training loss calculation is synthesized by masking. : in, The ground truth image (a ground truth image with a random background overlay) is used for training loss calculation. The original RGB image obtained after data preprocessing steps. The background color is randomized. When the model's output transparency is inaccurate in the boundary region, the randomized background amplifies the reconstruction error, increasing the RGB reconstruction loss and forcing the model to correct the transparency prediction, thus achieving a clearer contour boundary.

[0104] Let I be the rendered RGB image. In summary, the loss function for the j-th viewpoint after the algorithm's improvement is: For the loss at the j-th perspective, The ground truth image (a ground truth image with a random background overlay) is used for training loss calculation. For the image predicted by the model, To balance the weights of L1 loss and SSIM loss, The weights for the transparency mask supervision loss, The transparency mask supervision loss is also known as the first loss.

[0105] in, Calculated using the following formula: in, It is the total number of pixels. It predicts the value of the i-th pixel in the image. It is the value of the i-th pixel in the truth map.

[0106] Calculated using the following formula: in, It is SSIM loss. It is the average pixel value of the real image. It predicts the pixel mean of the image. It is the pixel variance of the real image. It is the pixel variance of the predicted image. It is the pixel covariance between the real image and the predicted image. , It is the stability constant.

[0107] Furthermore, the improved total loss function of this algorithm This is the sum of the losses under the primary perspective and related perspectives chosen in each iteration. It is expressed as follows: in, Total loss function , The loss is the loss from the j-th perspective.

[0108] Using the aforementioned loss function, the improved algorithm achieves results surpassing the current leading methods in the reconstruction and rendering of static human bodies, providing strong support for research in this field.

[0109] The method provided in this invention effectively solves the problems of background noise and unclear edges that are common in static Gaussian modeling of human body reconstruction by organically combining the transparency mask supervised optimization mechanism and the random background optimization mechanism, while enhancing the surface details of the model.

[0110] According to a static human body reconstruction method provided by the present invention, the clean foreground mask corresponding to each image is used as a supervision signal to calculate a first loss on the opacity image generated by the model rendering, including: Using the clean foreground mask corresponding to each image as a supervision signal, the first loss of the opacity image is calculated using the binary cross-entropy loss function.

[0111] Specifically, the present invention designs a system based on The binary cross-entropy is used as the transparency loss (first loss) to obtain a numerically stable optimization objective, as follows: Let the image size be H×W, then: in, The loss is the transparency mask supervision loss, i.e., the first loss. This is the binarized result of the foreground mask, i.e., the ground truth. These are the predicted values ​​from the model.

[0112] The method provided in this invention, through a transparency mask supervised optimization mechanism, enables the model to effectively distinguish between foreground and background, significantly reduce background noise, and enhance the representation of surface details.

[0113] According to a static human body reconstruction method provided by the present invention, a multi-view image set is preprocessed to obtain a clean foreground mask, accurate camera pose, and sparse human body point cloud corresponding to each image in the multi-view image set, and the method further includes: Foreground segmentation based on background subtraction is performed on each image in the multi-view image set to obtain each initial mask; Maximum connected component extraction is performed on each initial mask to obtain the clean foreground mask corresponding to each image; By combining the prior pose information of the camera with the clean foreground mask corresponding to each image, the multi-view image set is bundled and adjusted to obtain the pose of each camera and the sparse point cloud of each human body.

[0114] Specifically, in some embodiments, step 102, data preprocessing, includes the following steps: Step 1: Background removal: Foreground segmentation based on background subtraction is performed on each image in the multi-view image set to obtain initial masks. For example, the Background MattingV2 algorithm, based on the background subtraction principle, is used to separate the foreground and background, initially obtaining the foreground mask corresponding to each image. The mask value ranges from 0 to 255, where 0 represents the background region and 255 represents the human foreground region.

[0115] Step 2: Elimination of Missegmentation The maximum connected component is extracted from the segmented foreground mask to eliminate parts that the algorithm mistakenly identifies as foreground human figures. Specifically, the maximum connected component is extracted from each initial mask to obtain a clean foreground mask for each image. The clean foreground mask can then be superimposed on the original RGB image to obtain a clean foreground human figure image, providing a data foundation for subsequent calibration processes.

[0116] For example, extracting the clean foreground mask corresponding to each image includes the following steps: The method for extracting a clean foreground mask from the maximum connected component first employs the connected component analysis function with statistical information, such as the `connectedComponentsWithStats` function, from the Open Source Computer Vision Library (OpenCV). This function performs 8-connectivity analysis on the input mask image (the segmented initial foreground mask). The algorithm uses a depth-first search or breadth-first search traversal strategy to label interconnected pixels as the same connected component, returning the number of connected components, a pixel label matrix, a statistical information matrix, and the centroid coordinates of each connected component. Then, it iterates through all connected components, obtaining the pixel area of ​​each connected component by accessing the area field in the statistical information matrix. The area values ​​of each connected component are compared to identify the label of the connected component with the largest area. Finally, a zero matrix of the same size as the original mask (initial foreground mask) is created based on the identified maximum connected component label. Pixel values ​​belonging to the maximum connected component in the original mask (initial foreground mask) are copied to their corresponding positions, resulting in a mask image containing only the maximum foreground region, i.e., the clean foreground mask. The clean foreground mask is then superimposed on the original RGB image to obtain a clean foreground human image.

[0117] Step 3: Camera pose calibration Unlike traditional methods that directly input images into COLMAP for calibration, this invention incorporates preliminary poses obtained based on the camera model and the design of the acquisition system as prior information during the COLMAP calibration process. Specifically, by combining the prior camera pose information with the clean foreground mask corresponding to each image, the multi-view image set is bundled and adjusted to obtain the poses of each camera and the sparse point cloud of each human figure. This avoids camera pose estimation errors caused by very similar textures such as clothing, further improving the accuracy of camera pose calibration.

[0118] For example, Figure 4 This is a flowchart illustrating the data preprocessing method provided by the present invention, as shown below. Figure 4 As shown, the method includes: Input the raw data, including both long and short focal lengths; The BackgroundMattingV2 algorithm is applied to the original data to remove the background and obtain the foreground mask (initial foreground mask). The foreground mask is extracted using the maximum connected component to obtain the processed mask (i.e., the clean foreground mask). The processed mask is overlaid on the original RGB image to obtain the data used for calibration (i.e., a clean foreground human image). Import pose priors, perform colmap calibration, and output calibrated data. The calibrated data includes each original RGB image, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the human sparse point cloud corresponding to each image.

[0119] The method provided in this embodiment of the invention, through the above-described data preprocessing steps, can minimize background interference and pose errors at the source data level, thereby providing high-quality and clean input for subsequent modeling and improving the reconstruction effect.

[0120] According to a static human body reconstruction method provided by the present invention, the method further includes: The target camera pose is input into the three-dimensional Gaussian representation model of the target human body; the three-dimensional Gaussian representation model of the target human body is composed of multiple three-dimensional Gaussian ellipsoids, each of which has the attributes of position, covariance, opacity and spherical harmonic coefficients. Based on the target camera pose, the three-dimensional Gaussian ellipsoids in the three-dimensional Gaussian representation model of the target human body are screened and sorted. Based on the sorted three-dimensional Gaussian ellipsoids, a two-dimensional rendered image of the target human body in the current rendering viewpoint corresponding to the target camera pose is generated through parallel rasterization and alpha hybrid synthesis.

[0121] Specifically, in some embodiments, the method further includes a step of real-time rendering based on a reconstructed 3D Gaussian representation model of the target human body. The rendering utilizes a forward decoding process of a trained model. Real-time rendering can be applied to virtual reality, augmented reality, digital human display, real-time video communication, or interactive entertainment applications. Specifically, it includes the following: First, the target camera pose is input into the three-dimensional Gaussian representation model of the target human body; the three-dimensional Gaussian representation model of the target human body is composed of multiple three-dimensional Gaussian ellipsoids, each of which has attributes such as position, covariance, opacity and spherical harmonic coefficients.

[0122] Furthermore, based on the target camera pose, the 3D Gaussian ellipsoids in the 3D Gaussian representation model of the target human body are filtered and sorted. First, based on the view frustum of the current rendering viewpoint, view frustum culling is performed on the 3D Gaussian ellipsoids to remove those located outside the field of view, thus greatly reducing the computational load. Then, the 3D Gaussian ellipsoids remaining after view frustum culling are sorted from back to front according to their depth (distance) to the current rendering camera's imaging plane. This is the foundation for achieving correct alpha blending.

[0123] Finally, based on the sorted three-dimensional Gaussian ellipsoids, a two-dimensional rendered image of the target human body in the current rendering viewpoint corresponding to the target camera pose is generated through parallel rasterization and alpha hybrid synthesis.

[0124] Parallel rasterization, also known as tile-based rasterization, is crucial for real-time rendering in 3DGS. It divides the screen into multiple small tiles (e.g., 16x16 pixels), each handled by a GPU thread block. The system identifies which Gaussian ellipsoids might project and affect the pixels within that tile. Each Gaussian ellipsoid performs parallel calculations on the pixels within its influence range, contributing its color and opacity values. This process is entirely parallel on the GPU, resulting in extremely high efficiency. Pixel-wise alpha blending refers to the processor iterating through all the sorted Gaussian ellipsoids that contribute to each pixel in reverse order. Applying the standard "Over" alpha blending formula, the color (calculated using spherical harmonics, view-dependent) of each Gaussian ellipsoid and its opacity are accumulated and blended. Ultimately, each pixel receives a composite RGB color value, forming a complete 2D rendered image.

[0125] For example, the two-dimensional image plane to be rendered is divided into multiple tiles. For each sorted three-dimensional Gaussian ellipsoid, one or more tiles affected by its projection range are determined. For each tile, all three-dimensional Gaussian ellipsoids affecting the tile are processed in parallel, and the contribution value of each three-dimensional Gaussian ellipsoid to the color and opacity of each pixel in the tile is calculated. For each pixel, the color and opacity contribution values ​​of all three-dimensional Gaussian ellipsoids contributing to the pixel are calculated by Alpha blending in reverse order to synthesize the final color of the pixel. Figure 5 This is a schematic diagram of the rendering result of the three-dimensional Gaussian representation model of the target human body provided by the present invention, including two-dimensional rendering images from multiple perspectives.

[0126] The method provided in this invention trains a 3D Gaussian representation model of the target human body through multi-view joint optimization and joint supervision mechanisms. This model achieves optimal Gaussian point distribution, color, and transparency. Therefore, the input to this rendering process is a "clean, accurate, and detailed" model, capable of rendering high-quality images. Furthermore, the strong transparency supervision during training optimizes the opacity of Gaussian points outside the human body contour to near zero. During rendering, these points contribute almost nothing to the final image, fundamentally avoiding background noise and black artifacts. The efficiency of this rendering process ensures that the reconstructed digital human can achieve high-quality real-time driving and display in VR / AR, games, real-time interactive applications.

[0127] The static human body reconstruction device provided by the present invention is described below. The static human body reconstruction device described below and the static human body reconstruction method described above can be referred to in correspondence.

[0128] Figure 6This is a structural schematic diagram of the static human body reconstruction device provided by the present invention, as shown below. Figure 6 As shown, the static human body reconstruction device 600 includes the following modules: The acquisition module 610 is used to acquire a multi-view image set of the target human body acquired by a multi-camera array; the multi-view image set includes images acquired by a telephoto lens and images acquired by a short-focus lens. The data preprocessing module 620 is used to preprocess the multi-view image set to obtain the clean foreground mask, accurate camera pose, and human sparse point cloud corresponding to each image in the multi-view image set. The reconstruction module 630 is used to optimize the three-dimensional Gaussian splash model based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the human body sparse point cloud corresponding to each image, and to reconstruct the three-dimensional Gaussian representation model of the target human body.

[0129] The multi-view image set acquired in this invention includes images captured by telephoto lenses and images captured by short-focus lenses, which can cover the full range of perspectives and details of the target human body. In the data preprocessing stage, mask extraction and precise camera pose calibration are introduced, which significantly reduces the impact of segmentation errors and calibration deviations on the reconstruction model. The clean foreground mask and precise camera pose are input together with the original image into the 3D Gaussian splash model for model optimization, which improves the robustness and reliability of the input data from the source. In terms of reconstruction algorithm, based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is adopted to optimize the 3D Gaussian splash model, and a 3D Gaussian representation model of the target human body is reconstructed. This improves the overall consistency of the human body model between different perspectives, enhances the ability to restore local details, obtains more refined and realistic static human body reconstruction results, and improves the effect of static human body reconstruction.

[0130] According to the present invention, a static human body reconstruction device 600 is provided, wherein the reconstruction module 630 is specifically used for: In any iteration of optimization, the following optimization steps are performed: A primary view camera is selected in the current iteration; based on the geometric observation relationship and viewpoint complementarity between the primary view camera and the other cameras in the multi-view human image set (excluding the primary view camera), at least one associated camera is dynamically determined for the primary view camera from the multi-view human image set; image data corresponding to the primary view camera and image data corresponding to each associated camera are determined from the multi-view image set, the clean foreground mask corresponding to each image, and the camera pose corresponding to each image; based on the image data corresponding to the primary view camera and the image data corresponding to each associated camera, the model parameters of the 3D Gaussian splash model are jointly updated using gradients to obtain the updated model parameters under the current iteration. The updated model parameters under the current iteration optimization are used as the model parameters in the next iteration optimization, and the optimization steps are continued until the number of iterations is reached to obtain the three-dimensional Gaussian representation model of the target human body.

[0131] According to the static human body reconstruction device 600 provided by the present invention, the reconstruction module 630 is further used for: Based on the number of 3D points jointly observed by the main-view camera and each of the other cameras, the correlation strength between the main-view camera and each of the other cameras is calculated. Based on the aforementioned correlation strengths, a subset of candidate correlated cameras is selected for the main-view camera; Determine the angle between the normal direction of the main view camera and the normal directions of each camera in the candidate associated camera subset; Each associated camera is determined from the subset of candidate associated cameras based on the size of each included angle.

[0132] According to the static human body reconstruction device 600 provided by the present invention, the reconstruction module 630 is further used for: Based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, the multi-view joint optimization strategy and the joint supervision mechanism of transparency mask supervision and random background perturbation are used to optimize the three-dimensional Gaussian splash model and reconstruct the three-dimensional Gaussian representation model of the target human body.

[0133] According to the static human body reconstruction device 600 provided by the present invention, the joint supervision mechanism of transparency mask supervision and random background perturbation includes: Using the clean foreground mask corresponding to each of the images as a supervision signal, the first loss is calculated for the opacity image generated by the model rendering. The random background color of each of the images is dynamically generated. The random background color of each of the images is combined with the image to form a new target image using the clean foreground mask corresponding to each of the images. Based on the new target image, a second loss is calculated for the color image generated by the model rendering. Based on the first loss and the second loss, determine the total loss function; Based on the total loss function, the model parameters of the three-dimensional Gaussian splash model are adjusted.

[0134] According to the static human body reconstruction device 600 provided by the present invention, the reconstruction module 630 is further used for: Using the clean foreground mask corresponding to each image as a supervision signal, the first loss of the opacity image is calculated using the binary cross-entropy loss function.

[0135] According to the static human body reconstruction device 600 provided by the present invention, the data preprocessing module 620 is specifically used for: Perform foreground segmentation based on background subtraction on each of the images in the multi-view image set to obtain each initial mask; The maximum connected component is extracted from each of the initial masks to obtain the clean foreground mask corresponding to each of the images; By combining the prior pose information of the camera with the clean foreground mask corresponding to each image, the multi-view image set is bundled and adjusted to obtain the pose of each camera and the sparse point cloud of each human body.

[0136] According to the present invention, a static human body reconstruction device 600 is provided, the device further comprising a rendering module; The rendering module is used for: The target camera pose is input into the three-dimensional Gaussian representation model of the target human body; the three-dimensional Gaussian representation model of the target human body is composed of multiple three-dimensional Gaussian ellipsoids, each of which has the attributes of position, covariance, opacity and spherical harmonic coefficients; Based on the target camera pose, the three-dimensional Gaussian ellipsoids in the three-dimensional Gaussian representation model of the target human body are filtered and sorted. Based on the sorted three-dimensional Gaussian ellipsoids, a two-dimensional rendered image of the target human body under the current rendering viewpoint corresponding to the pose of the target camera is generated through parallel rasterization and alpha hybrid synthesis.

[0137] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a static human body reconstruction method, which includes: Acquire a multi-view image set of a target human body captured by a multi-camera array; the multi-view image set includes images captured by a telephoto lens and images captured by a short-focus lens; The multi-view image set is preprocessed to obtain the clean foreground mask, accurate camera pose, and sparse human point cloud corresponding to each image in the multi-view image set. Based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is used to optimize the three-dimensional Gaussian splash model and reconstruct the three-dimensional Gaussian representation model of the target human body.

[0138] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the static human body reconstruction method provided by the above methods, the method comprising: Acquire a multi-view image set of a target human body captured by a multi-camera array; the multi-view image set includes images captured by a telephoto lens and images captured by a short-focus lens; The multi-view image set is preprocessed to obtain the clean foreground mask, accurate camera pose, and sparse human point cloud corresponding to each image in the multi-view image set. Based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is used to optimize the three-dimensional Gaussian splash model and reconstruct the three-dimensional Gaussian representation model of the target human body.

[0140] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the static human body reconstruction method provided by the methods described above, the method comprising: Acquire a multi-view image set of a target human body captured by a multi-camera array; the multi-view image set includes images captured by a telephoto lens and images captured by a short-focus lens; The multi-view image set is preprocessed to obtain the clean foreground mask, accurate camera pose, and sparse human point cloud corresponding to each image in the multi-view image set. Based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is used to optimize the three-dimensional Gaussian splash model and reconstruct the three-dimensional Gaussian representation model of the target human body.

[0141] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0142] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A static human body reconstruction method, characterized in that, include: Acquire a multi-view image set of the target human body captured by a multi-camera array; The multi-view image set includes images acquired through a telephoto lens and images acquired through a short-focus lens; The multi-view image set is preprocessed to obtain the clean foreground mask, accurate camera pose, and sparse human point cloud corresponding to each image in the multi-view image set. Based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, a multi-view joint optimization strategy is used to optimize the three-dimensional Gaussian splash model and reconstruct the three-dimensional Gaussian representation model of the target human body.

2. The static human body reconstruction method according to claim 1, characterized in that, The method involves optimizing the 3D Gaussian splash model using a multi-view joint optimization strategy based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, to reconstruct a 3D Gaussian representation model of the target human body, including: In any iteration of optimization, the following optimization steps are performed: A primary view camera is selected in the current iteration; based on the geometric observation relationship and viewpoint complementarity between the primary view camera and the other cameras in the multi-view human image set (excluding the primary view camera), at least one associated camera is dynamically determined for the primary view camera from the multi-view human image set; from the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse human point cloud corresponding to each image, the image data corresponding to the primary view camera and the image data corresponding to each associated camera are determined; based on the image data corresponding to the primary view camera and the image data corresponding to each associated camera, the model parameters of the 3D Gaussian splash model are jointly updated using gradients to obtain the updated model parameters under the current iteration. The updated model parameters under the current iteration optimization are used as the model parameters in the next iteration optimization, and the optimization steps are continued until the number of iterations is reached to obtain the three-dimensional Gaussian representation model of the target human body.

3. The static human body reconstruction method according to claim 2, characterized in that, The step of dynamically determining at least one associated camera for the main-view camera from the multiple cameras corresponding to the multi-view human image set, based on the geometric observation relationship and viewpoint complementarity between the main-view camera and the other cameras in the multi-view human image set (excluding the main-view camera), includes: Based on the number of 3D points jointly observed by the main-view camera and each of the other cameras, the correlation strength between the main-view camera and each of the other cameras is calculated. Based on the correlation strengths, a subset of candidate correlated cameras is selected for the main view camera; Determine the angle between the normal direction of the main view camera and the normal directions of each camera in the candidate associated camera subset; Each associated camera is determined from the subset of candidate associated cameras based on the size of each included angle.

4. The static human body reconstruction method according to claim 1, characterized in that, The method involves optimizing the 3D Gaussian splash model using a multi-view joint optimization strategy based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, to reconstruct a 3D Gaussian representation model of the target human body, including: Based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the sparse point cloud of the human body corresponding to each image, the multi-view joint optimization strategy and the joint supervision mechanism of transparency mask supervision and random background perturbation are adopted to optimize the three-dimensional Gaussian splash model and reconstruct the three-dimensional Gaussian representation model of the target human body.

5. The static human body reconstruction method according to claim 4, characterized in that, The joint supervision mechanism of transparency mask supervision and random background perturbation includes: Using the clean foreground mask corresponding to each of the images as a supervision signal, the first loss is calculated for the opacity image generated by the model rendering. The random background color of each of the images is dynamically generated. The random background color of each of the images is combined with the image to form a new target image using the clean foreground mask corresponding to each of the images. Based on the new target image, a second loss is calculated for the color image generated by the model rendering. Based on the first loss and the second loss, determine the total loss function; Based on the total loss function, the model parameters of the three-dimensional Gaussian splash model are adjusted.

6. The static human body reconstruction method according to claim 5, characterized in that, The step of using the clean foreground mask corresponding to each of the images as a supervision signal to calculate the first loss for the opacity image generated by the model rendering includes: Using the clean foreground mask corresponding to each image as a supervision signal, the first loss of the opacity image is calculated using the binary cross-entropy loss function.

7. The static human body reconstruction method according to any one of claims 1-6, characterized in that, The preprocessing of the multi-view image set to obtain the clean foreground mask, accurate camera pose, and sparse human point cloud corresponding to each image in the multi-view image set also includes: Perform foreground segmentation based on background subtraction on each of the images in the multi-view image set to obtain each initial mask; The maximum connected component is extracted from each of the initial masks to obtain the clean foreground mask corresponding to each of the images; By combining the prior pose information of the camera with the clean foreground mask corresponding to each image, the multi-view image set is bundled and adjusted to obtain the pose of each camera and the sparse point cloud of each human body.

8. The static human body reconstruction method according to any one of claims 1-6, characterized in that, The method further includes: The target camera pose is input into the three-dimensional Gaussian representation model of the target human body; the three-dimensional Gaussian representation model of the target human body is composed of multiple three-dimensional Gaussian ellipsoids, each of which has the attributes of position, covariance, opacity and spherical harmonic coefficients; Based on the target camera pose, the three-dimensional Gaussian ellipsoids in the three-dimensional Gaussian representation model of the target human body are filtered and sorted. Based on the sorted three-dimensional Gaussian ellipsoids, a two-dimensional rendered image of the target human body under the current rendering viewpoint corresponding to the pose of the target camera is generated through parallel rasterization and alpha hybrid synthesis.

9. A static human body reconstruction device, characterized in that, include: The acquisition module is used to acquire a set of multi-view images of the target human body captured by a multi-camera array; The multi-view image set includes images acquired through a telephoto lens and images acquired through a short-focus lens; The data preprocessing module is used to preprocess the multi-view image set to obtain the clean foreground mask, accurate camera pose, and human sparse point cloud corresponding to each image in the multi-view image set. The reconstruction module is used to optimize the three-dimensional Gaussian splash model based on the multi-view image set, the clean foreground mask corresponding to each image, the camera pose corresponding to each image, and the human body sparse point cloud corresponding to each image, and to reconstruct the three-dimensional Gaussian representation model of the target human body.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the static human body reconstruction method as described in any one of claims 1 to 8.

Images