A method and system for 3D reconstruction and automatic segmentation of bald patches based on video images

Abstract

This invention relates to a method and system for 3D reconstruction and automatic segmentation of alopecia areata regions based on video images, belonging to the field of computer vision and medical image processing technology. Addressing the problems of strong subjectivity, large geometric errors, and lack of 3D information in existing alopecia areata assessments that rely on 2D photographs, this invention acquires a video surrounding the head, uses a deep learning model to generate temporally consistent semantic masks and optimizes frame data, and constructs a high-precision 3D head model using planar-constrained Gaussian splashing technology. By introducing parametric head model priors and a neural deformation pyramid, non-rigid registration is achieved, automatically extracting the scalp region. Multidimensional vectors are constructed by fusing color and geometric features, adaptively segmenting alopecia areata lesions and calculating the true surface area. This invention achieves full automation from video acquisition to quantitative diagnosis, significantly improving the geometric fidelity of scalp reconstruction and the objectivity of lesion measurement, providing accurate visual evidence for clinical diagnosis and treatment.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and medical image processing technology, and relates to a method and system for three-dimensional reconstruction and automatic segmentation of alopecia areata regions based on video images. Background Technology

[0002] Alopecia areata is a common autoimmune skin disease, clinically characterized by sudden, localized patchy hair loss on the scalp, which can progress to total or universal alopecia. Accurate assessment of the area, distribution pattern, and dynamic evolution of alopecia areata is crucial for developing personalized treatment plans, determining disease severity, and evaluating drug efficacy. Currently, clinical assessment methods for alopecia areata primarily rely on visual observation and two-dimensional photographic measurement by physicians, which have significant limitations.

[0003] Traditional assessment methods are highly subjective and lack consistency. Doctors' visual estimations of lesion area exhibit significant inter-observer variability, leading to inconsistencies in treatment plans as different physicians assess the same patient's condition. Furthermore, the scalp is a three-dimensional surface with complex curvature, and two-dimensional photographic measurements completely ignore these geometric characteristics, resulting in severe projection errors. Alopecia areata on the top and sides of the head show different distortion ratios in two-dimensional imaging, making it impossible for a single photograph to reflect the true total area of ​​hair loss. In addition, two-dimensional photographs struggle to record the precise spatial distribution of lesions on a three-dimensional head model. During treatment cycles lasting several months, physicians cannot accurately determine minute changes in lesion edges by comparing photographs taken under different lighting conditions.

[0004] Existing 3D reconstruction technologies face significant bottlenecks when applied to scalp and hair scenes. Multi-view stereo vision-based methods rely on feature point matching, but the uniform texture of the scalp region leads to matching failures. The hair region, with its complex structure and severe self-occlusion, causes traditional stereo matching algorithms to fail in the hair area, resulting in reconstructions filled with noise or large areas of voids, failing to meet the accuracy requirements of medical measurements. While neural radiation field technology can synthesize realistic new perspective images, it relies on implicit neural networks to represent the scene, making it difficult to directly derive high-quality explicit meshes for medical measurements. Furthermore, when processing scalp specular reflections and fine hair structures, it easily bakes specular information into color channels, causing visual artifacts when lighting or viewing angles changes, and making accurate segmentation of geometric surfaces difficult. 3D Gaussian splashing, as an emerging explicit point cloud representation method, performs excellently in terms of rendering speed and quality. However, the original algorithm primarily optimizes visual effects rather than geometric accuracy, generating chaotic point clouds that struggle to extract smooth, continuous scalp surfaces conforming to anatomical structures. It particularly fails to meet the high geometric fidelity required for the scalp region and lacks semantic information, making it unable to automatically distinguish between hair and scalp. While parametric head models can provide standard topological and semantic information, they can only fit the general shape and cannot capture the subtle geometric features of individual patients. They have a very weak ability to fit hair areas and cannot reflect the true state of hair loss areas.

[0005] Medical settings present unique challenges for assessing alopecia areata. Beyond 3D reconstruction, automatic lesion segmentation and quantification are also required. Existing general-purpose reconstruction algorithms cannot automatically identify scalp areas, typically requiring doctors to manually delineate them on the 3D model—a process that is inefficient and prone to fatigue errors. Furthermore, the physical differences between scalp and hair are often confused in current reconstructions, leading to texture distortion and impacting subsequent image-based lesion identification. Therefore, there is an urgent clinical need for an end-to-end solution capable of automatically reconstructing a high-precision 3D scalp model from readily available video data, combined with anatomical semantic automatic segmentation and measurement of alopecia areata regions. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a method and system for three-dimensional reconstruction and automatic segmentation of alopecia areata regions based on video images.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for 3D reconstruction and automatic segmentation of bald patches based on video images includes the following steps: The video data of the head surrounding the object to be detected is acquired. The image quality assessment model is used to score the sharpness of the sampled frames. Grounded SAM 2 is introduced to generate temporally consistent semantic masks of the head and scalp. Low-quality frames are removed by combining the mask ratio analysis and a multi-view effective image sequence is constructed. The structure-of-motion reconstruction technique is used to extract features and estimate poses of the multi-view effective image sequence to generate a sparse point cloud. A Gaussian splash scene representation model based on planar constraints is established. The Gaussian unit is initialized using the sparse point cloud. Iterative optimization is performed by combining the human head semantic mask with the camera pose. The initial 3D mesh is extracted by unbiased depth rendering and truncated symbolic distance field fusion. The initial 3D mesh is subjected to cascaded optimization processing. A point cloud convolutional network is introduced to refine the mesh geometry in depth. The high-frequency skin texture is recovered by implicit occupancy field inference, and a high-precision human head mesh is generated. Texture mapping is performed on the high-precision head mesh, and a texture completion algorithm based on a diffusion model is used to semantically fill texture holes, generating a three-dimensional head model with complete texture. A parametric head model FLAME is introduced to extract the topological prior of head anatomy. A neural deformation pyramid model is constructed to achieve non-rigid registration between the parametric model and the three-dimensional head model. Based on the registration results, anatomical semantics are projected and the target scalp region is extracted. A multidimensional feature vector is constructed, and an adaptive clustering algorithm is used to segment the alopecia areata lesions within the target scalp region. The true surface area of ​​the alopecia areata region is calculated based on the three-dimensional surface integral.

[0008] Furthermore, the establishment of a Gaussian splash scene representation model based on planar constraints specifically includes: The 3D scene is represented as a set of Gaussian ellipsoids with learnable position, rotation, scaling, opacity, and spherical harmonics. By introducing a planar regularization constraint, Gaussian elements are forced to conform to the geometry of the scalp surface by penalizing the non-planar distribution of Gaussian elements in the local neighborhood. Construct an unbiased depth rendering pipeline, calculate pixel-level depth by accumulating transmissivity-weighted depth values, and optimize Gaussian geometric parameters using the consistency constraints between the depth map and the normal map. By combining the exposure compensation mechanism, affine transformation modeling is performed on the illumination changes under different viewpoints.

[0009] Furthermore, the plane regularization constraint term is:

[0010] in, For plane regularization constraints, It is a set of surface Gaussian elements. For the first i The smallest eigenvalue of the covariance matrix of Gaussian units. For the first i The largest eigenvalue of the covariance matrix of Gaussian units For the first i The covariance matrix of Gaussian units.

[0011] Furthermore, the introduction of a point cloud convolutional network for in-depth refinement of mesh geometric details specifically includes: The optimized mesh is converted into a dense point cloud as network input; Multi-scale geometric features are extracted using point cloud-based convolutional operations to construct a spatial implicit occupancy field; The query points in space are subjected to feature interpolation and multilayer perceptron classification to determine the probability that they are located inside or outside an object. The moving cube algorithm is used to extract isosurfaces from high-resolution implicit fields.

[0012] Furthermore, the construction of the neural deformation pyramid model to achieve non-rigid registration specifically includes: A hierarchical multilayer perceptron network architecture is constructed, with each layer corresponding to a sinusoidal position code of different frequencies, to simulate a multi-scale motion field from global rigid motion to local non-rigid deformation. Input the source mesh vertex coordinates into the network to predict the displacement vector field, and construct a total loss function that includes chamfer distance loss, as rigid as possible ARAP regularization term and Laplace smoothing regularization term; A progressive training strategy from low frequency to high frequency is adopted, first optimizing global shape alignment and then optimizing local detail fitting; Using the registered vertex index mapping relationship, the predefined scalp semantic boundary in the FLAME model is projected onto the target reconstruction mesh as a seed constraint for region growing.

[0013] Furthermore, the segmentation of alopecia areata lesions within the target scalp region using an adaptive clustering algorithm specifically includes: The texture image of the target scalp mesh is converted to the CIELAB color space, and the luminance features of the L channel and the chrominance features of the a and b channels are extracted. The variance of the K-neighborhood normals of the mesh vertices is calculated as a roughness geometric feature; Constructing feature vectors Clustering algorithms are used to divide vertices into hair-covered regions and hair loss patch regions; Morphological opening operations and connected component analysis are performed on the classification results to remove noise regions with an area smaller than a preset threshold, thus establishing the final alopecia areata lesions. Traverse all triangular faces within the alopecia areata region and calculate the area of ​​each triangle. And by summing them up, the precise surface area of ​​the alopecia areata lesion is obtained, where Let the area be the triangular facet. Let be the vector from vertex A to vertex B of the triangular facet. Let be the vector from vertex A to vertex C of the triangular facet.

[0014] Furthermore, the introduction of Grounded SAM 2 to generate temporally consistent semantic masks for the human head and scalp specifically includes: Use a text encoder to convert prompts into semantic embedding vectors; Multi-scale features of video frames are extracted using a hierarchical image encoder. A memory bank and a memory attention module are introduced to store and retrieve the mask features and image features of the preceding frame, and the occlusion and viewpoint change problems in the video are solved by temporal context fusion. By combining semantic embedding vectors with fused image features using a mask decoder, a temporally consistent instance segmentation mask sequence is output.

[0015] Furthermore, the extraction of the target scalp region also includes boundary curve smoothing processing: Establish a local coordinate system with the top of the head as the Z-axis, and project the registered scalp boundary points onto the cylindrical coordinate system; Divide the angle domain into N intervals, and select the candidate point with the lowest height in each interval as the boundary point; The extracted boundary point sequence is smoothed using spline interpolation or Gaussian smoothing filtering to generate a continuous closed scalp contour line.

[0016] A system for 3D reconstruction and automatic segmentation of alopecia areata based on video images includes a processor, an input device, an output device, and a memory, which are interconnected. The memory stores a computer program, which includes program instructions. The processor is configured to call the program instructions to execute the method for 3D reconstruction and automatic segmentation of alopecia areata based on video images.

[0017] The beneficial effects of this invention are as follows: (1) By introducing Gaussian splashing technology with planar constraints and a multi-level geometric optimization strategy, this invention can generate a watertight and high-fidelity 3D head mesh. Compared with traditional multi-view stereo vision methods, this scheme significantly suppresses point cloud noise and artifacts, resulting in a smoother surface that conforms to anatomical structure. At the same time, by combining point cloud convolutional networks and implicit occupancy field inference, this invention restores the microscopic geometric details at the hair follicle level while maintaining the accuracy of the macroscopic shape of the scalp, greatly improving the model's precision and realism in medical diagnosis.

[0018] (2) By introducing a parametric head model as an anatomical topological prior and using a neural deformation pyramid to achieve non-rigid registration, this invention successfully maps standard scalp regions onto a high-precision reconstructed model of the individual patient. This deep learning-based registration method can adapt to large deformations of different head shapes, avoiding the local optimum problem that traditional algorithms are prone to when fitting complex surfaces, thus achieving fully automatic and high-precision extraction of the scalp region and completely eliminating the need for tedious manual drawing on the three-dimensional model.

[0019] (3) By integrating color space features and local geometric roughness features, this invention can adaptively distinguish between hair-covered areas and alopecia areata lesion areas, unaffected by changes in lighting. More importantly, the area integration calculation method based on three-dimensional curved surface mesh completely eliminates the geometric distortion caused by curved surface projection in traditional two-dimensional photographic measurements, providing clinically accurate lesion area data. This provides solid quantitative support for doctors to determine the severity of the disease, formulate personalized treatment plans, and track the evolution of treatment efficacy.

[0020] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the overall technical process of the method of the present invention. Figure 2 This is a schematic diagram illustrating the principles of data preprocessing and semantic mask generation in this invention; Figure 3 This is a flowchart illustrating the entire process of high-fidelity geometric reconstruction and refinement in this invention. Figure 4 This is a schematic diagram of the non-rigid registration process between the FLAME model based on the Neural Deformation Pyramid (NDP) and the reconstructed mesh in this invention. Figure 5 This is a flowchart illustrating the logical process of alopecia areata feature analysis and quantitative evaluation in this invention. Detailed Implementation

[0022] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0023] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0024] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0025] Figure 1This is a flowchart of the overall technical process of the present invention, showing the complete S1-S10 steps from video input to alopecia areata analysis; Figure 2 This is a schematic diagram illustrating the principle of data preprocessing and semantic mask generation in this invention, showing how Grounded SAM2 combines text prompts to generate a temporally consistent mask. Figure 3 This is a flowchart of the high-fidelity geometric reconstruction and refinement process in this invention, showing a schematic diagram of the dense reconstruction based on PGSR, TSDF fusion, and point cloud convolutional network mesh refinement process. Figure 4 This is a schematic diagram of the non-rigid registration process between the FLAME model based on the Neural Deformation Pyramid (NDP) and the reconstructed mesh in this invention. Figure 5 This is a flowchart illustrating the logical process of alopecia areata feature analysis and quantitative evaluation in this invention, showing the complete processing chain from target scalp grid input to final score output.

[0026] Example 1: A method for 3D scalp reconstruction and alopecia areata quantification based on implicit geometric refinement and neural deformation pyramid. The method flow of this embodiment is as follows: Figure 1 As shown, the aim is to recover a 3D model of the scalp with micron-level geometric details from handheld monocular videos and automatically perform semantic segmentation and quantization of lesions. The overall process includes three core stages: high-fidelity geometric reconstruction (module 1), semantic injection and region extraction (module 2), and lesion feature analysis and quantization (module 3).

[0027] Module 1: High-fidelity head model reconstruction and geometric refinement The core of this module is to solve the reconstruction defects of traditional multi-view stereo vision (MVS) in areas with weak texture (scalp) and complex structure (hair). By introducing temporal semantic constraints, planar regularized Gaussian fields and implicit point cloud convolutional networks, a watertight 3D mesh that retains hair follicle-level details is constructed.

[0028] Step S1: Data Acquisition and Semantic Parsing Based on Temporal Memory Non-uniform sampling strategy: Acquiring the RGB video stream surrounding the head A content-aware non-uniform sampling strategy is adopted: a lightweight face detector is used to analyze frame content, and a low sampling rate is set for frontal facial frames (e.g., ...). For frames on the scalp and back of the head, a high sampling rate is set (e.g.) (and used the HyperIQA network to calculate the sharpness score) Remove To construct an effective frame sequence from blurred frames. .

[0029] Temporally consistent mask generation: The Grounded SAM2 model is introduced to generate the foreground mask.

[0030] Prompt encoding: Input text prompt To the text encoder, generate semantic embedding vectors. .

[0031] Memory Bank Mechanism: The model maintains a first-in, first-out (FIFO) memory bank. Store the mask features of the past $K$ frames and image features For the current frame Use the memory attention module to retrieve historical information:

[0032] in, This is the query vector for the current frame. This mechanism leverages temporal context to address local occlusion (such as hand obstruction) and abrupt changes in viewpoint during surround shooting, outputting a temporally consistent binary mask sequence. .

[0033] Step S2: Sparse Reconstruction and Construction of the Basic Dense Field PGSR Pose calculation: Using COLMAP to calculate the sequence Feature extraction (SIFT) and matching are performed, and the reprojection error is minimized using bundle adjustment to calculate the camera intrinsic parameters. and external references And generate sparse point clouds. .

[0034] Planar constrained Gaussian splash PGSR initialization: with Initialize a 3D Gaussian set .

[0035] Planar regularization optimization: To suppress volumetric noise on the scalp surface, the Gaussian covariance matrix is ​​optimized. When introducing planar constraint terms The penalty is the thickness of the Gaussian ellipsoid in the local normal direction (i.e., the minimum eigenvalue). ):

[0036] This will force the Gosskiy unit to flatten and conform to the scalp cutting plane.

[0037] Unbiased depth rendering: Calculates pixel-level depth using a weighted average of cumulative transmissivity. And combined with normal consistency loss Optimize geometry:

[0038]

[0039] in, The geometric normal is obtained by depth map subtraction. To render the normals, the base mesh is finally extracted through TSDF (Truncation Symbolic Distance Field) fusion. .

[0040] Step S3: Implicit Geometric Refinement Based on Point Cloud Convolution against To address the issues of tiny holes and smooth details, a pre-trained implicit surface refinement network is introduced (which directly manipulates the point cloud without voxelization).

[0041] Point cloud feature encoding: Convert to dense surface point cloud Multi-scale geometric features are extracted using a fully convolutional point network (F-PConv). The $l$-th convolutional layer is defined as:

[0042] This operation aggregates neighborhood features in a continuous space while preserving the original resolution.

[0043] For any query point in space (For constructing higher resolution isosurfaces), the network does not use simple trilinear interpolation, but instead aggregates them through an attention mechanism. Nearest input point Features :

[0044]

[0045] Implicit field decoding and isosurface extraction: aggregating features Input decoder prediction occupancy probability Construct a high-resolution grid for batch inference. Finally, the Marching Cubes algorithm is used to extract... The isosurfaces are used to obtain a high-fidelity mesh containing the micro-undulations of hair follicles. .

[0046] Step S4: Texture Mapping and Completion right Seamless UV unwrapping using xatlas is performed. Texture maps are generated through multi-view weighted projection. For texture holes caused by occlusion, a generative model based on Latent Diffusion Inpainting is used to perform semantic completion based on the context texture.

[0047] Module 2: Semantic Injection and Automated Scalp Region Extraction This module aims to solve the challenge of "accurately identifying scalp from semantically undefined meshes". It utilizes the FLAME parametric model as a topological prior and achieves semantic transfer through non-rigid registration.

[0048] Step S5: Parametric model fitting Optimize the shape parameters of the FLAME model using the detected 2D facial key points. (Shape), attitude parameters (Pose) and facial expression parameters This aligns the projection with the video frame, generating a source mesh with a standard topology. .

[0049] Step S6: Construct the Neural Deformation Pyramid (NDP) for non-rigid registration. In order to make smooth (Source) and fine (Target Alignment, constructing hierarchical neural deformation fields .

[0050] Build A hierarchical pyramid network. Input coordinates of layer network After a frequency of Sine code:

[0051] set up Low-frequency layer Capture global rigid displacement (head alignment), high-frequency layer Capture localized non-rigid deformations (scalp undulations).

[0052] Cascaded Displacement Prediction: Total Displacement Field It is formed by summing the outputs of each layer:

[0053] Optimization objective: Minimize Chamfer Distance and regularization terms.

[0054] in, To ensure the constraints are as rigid as possible (As-Rigid-As-Possible) and prevent mesh topology tearing.

[0055] Step S7: Semantic Projection and Boundary Optimization Semantic transfer: Utilizing the registered correspondence, The predefined set of "scalp" vertex indices Projected onto target mesh This yields a rough estimate of the scalp area. .

[0056] Bucket Filtering: Establishes a boundary smoothing mechanism based on the overhead normal. Local cylindrical coordinate system of the axis .Will Divided into Each interval (bucket) is selected. Within each interval, select... The boundary point with the smallest value is used as the key point. A B-spline curve is used to connect the key points to generate a smooth and closed hairline outline. .

[0057] Precise cropping: Using the center of the top of the head as the seed point, perform a geodesic region growing algorithm until the outline is reached. To obtain the final target scalp grid .

[0058] Module 3: Bald Spot Region Segmentation and Quantization Based on Feature Manifolds This module uses an unsupervised adaptive clustering algorithm to identify lesions on the extracted scalp mesh.

[0059] Step S8: Construct a multidimensional feature manifold for Each vertex Calculate eigenvectors : Chromaticity characteristics: Convert the texture to the CIELAB color space and extract... The components are used to separate brightness and skin tone information.

[0060] Microscopic geometric features: computational local normal variance of the neighborhood The alopecia areata area is smooth. ), with rough areas ( ).

[0061] Manifold Construction: Constructing a Weighted Adjacency Graph edge weight .

[0062] Step S9: Adaptive Pruning and Clustering Adaptive thresholding: The histogram distribution of edge weights across the entire graph is statistically analyzed, and the optimal segmentation threshold is automatically calculated using the Otsu algorithm. This threshold corresponds to the natural boundary between "skin" and "hair" in the feature space.

[0063] Image pruning: Remove all The edge. This operation severs the connection between the lesion and the normal area in the feature space.

[0064] Region extraction: Perform connected component analysis on the pruned graph to obtain a set of candidate regions. .

[0065] Semantic classification: Calculate each region average characteristics The region category is determined based on preset rules (high brightness and low roughness):

[0066] Step S10: Surface area integration and report generation Actual area calculation: Traverse all triangles classified as "Alopecia" Calculate the surface area integral:

[0067] SALT score: Calculates the lesion area as a percentage of the total scalp area. The percentage is calculated, and a scoring report is automatically generated based on the standard SALT partition weights.

[0068] Example 2: A 3D Reconstruction and Automatic Segmentation System for Bald Areas Based on Video Images System Hardware Architecture Acquisition terminal: Smartphone (such as iPhone 14 Pro), using ARKit to obtain the initial pose and shoot a 4K resolution / 60fps video stream.

[0069] Compute workstation: Equipped with an Intel Core i9 processor, 64GB of RAM, and an NVIDIA RTX 4090 graphics card (24GB of VRAM), designed to support high-performance computing models such as 3DGS, SAM 2, and NDP.

[0070] System software modules Data interface module: responsible for video stream decoding, frame synchronization, and IQA quality screening.

[0071] Semantic analysis module: integrates the Grounded SAM 2 model to output the mask stream for each frame in real time; integrates the FLAME fitting algorithm to generate semantic priors.

[0072] Reconstruction Engine: The core module, implemented using PyTorch. It includes the COLMAP interface (sparse reconstruction), the PGSR solver (dense reconstruction), and a point cloud convolutional network inference engine (mesh optimization).

[0073] Lesion analysis module: Runs NDP registration algorithm and feature clustering algorithm to automatically output binarized mask and area data of alopecia areata region.

[0074] Visual UI: Displays a 3D scalp model via a web-based or native GUI. Supports user interaction, such as manually correcting the boundaries of alopecia areata (Human-in-the-loop), and supports exporting PDF diagnostic reports.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for three-dimensional reconstruction and automatic segmentation of bald patches based on video images, characterized in that: Includes the following steps: The video data of the head surrounding the object to be detected is acquired. The image quality assessment model is used to score the sharpness of the sampled frames. Grounded SAM 2 is introduced to generate temporally consistent semantic masks of the head and scalp. Low-quality frames are removed by combining the mask ratio analysis and a multi-view effective image sequence is constructed. The structure-of-motion reconstruction technique is used to extract features and estimate poses of the multi-view effective image sequence to generate a sparse point cloud. A Gaussian splash scene representation model based on planar constraints is established. The Gaussian unit is initialized using the sparse point cloud. Iterative optimization is performed by combining the human head semantic mask with the camera pose. The initial 3D mesh is extracted by unbiased depth rendering and truncated symbolic distance field fusion. The initial 3D mesh is subjected to cascaded optimization processing. A point cloud convolutional network is introduced to refine the mesh geometry in depth. The high-frequency skin texture is recovered by implicit occupancy field inference, and a high-precision human head mesh is generated. Texture mapping is performed on the high-precision head mesh, and a texture completion algorithm based on a diffusion model is used to semantically fill texture holes, generating a three-dimensional head model with complete texture. A parametric head model FLAME is introduced to extract the topological prior of head anatomy. A neural deformation pyramid model is constructed to achieve non-rigid registration between the parametric model and the three-dimensional head model. Based on the registration results, anatomical semantics are projected and the target scalp region is extracted. A multidimensional feature vector is constructed, and an adaptive clustering algorithm is used to segment the alopecia areata lesions within the target scalp region. The true surface area of ​​the alopecia areata region is calculated based on the three-dimensional surface integral.

2. The method for three-dimensional reconstruction and automatic segmentation of bald patches based on video images according to claim 1, characterized in that: The establishment of a Gaussian splash scene representation model based on planar constraints specifically includes: The 3D scene is represented as a set of Gaussian ellipsoids with learnable position, rotation, scaling, opacity, and spherical harmonics. By introducing a planar regularization constraint, Gaussian elements are forced to conform to the geometry of the scalp surface by penalizing the non-planar distribution of Gaussian elements in the local neighborhood. Construct an unbiased depth rendering pipeline, calculate pixel-level depth by accumulating transmissivity-weighted depth values, and optimize Gaussian geometric parameters using the consistency constraints between the depth map and the normal map. By combining the exposure compensation mechanism, affine transformation modeling is performed on the illumination changes under different viewpoints.

3. The method for three-dimensional reconstruction and automatic segmentation of bald patches based on video images according to claim 2, characterized in that: The plane regularization constraint term is: in, For plane regularization constraints, It is a set of surface Gaussian elements. For the first i The smallest eigenvalue of the covariance matrix of Gaussian units. For the first i The largest eigenvalue of the covariance matrix of Gaussian units For the first i The covariance matrix of Gaussian units.

4. The method for three-dimensional reconstruction and automatic segmentation of bald patches based on video images according to claim 1, characterized in that: The introduction of a point cloud convolutional network for in-depth refinement of mesh geometric details specifically includes: The optimized mesh is converted into a dense point cloud as network input; Multi-scale geometric features are extracted using point cloud-based convolutional operations to construct a spatial implicit occupancy field; The query points in space are subjected to feature interpolation and multilayer perceptron classification to determine the probability that they are located inside or outside an object. The moving cube algorithm is used to extract isosurfaces from high-resolution implicit fields.

5. The method for three-dimensional reconstruction and automatic segmentation of bald patches based on video images according to claim 1, characterized in that: The construction of the neural deformation pyramid model to achieve non-rigid registration specifically includes: A hierarchical multilayer perceptron network architecture is constructed, with each layer corresponding to a sinusoidal position code of different frequencies, to simulate a multi-scale motion field from global rigid motion to local non-rigid deformation. Input the source mesh vertex coordinates into the network to predict the displacement vector field, and construct a total loss function that includes chamfer distance loss, as rigid as possible ARAP regularization term and Laplace smoothing regularization term; A progressive training strategy from low frequency to high frequency is adopted, first optimizing global shape alignment and then optimizing local detail fitting; Using the registered vertex index mapping relationship, the predefined scalp semantic boundary in the FLAME model is projected onto the target reconstruction mesh as a seed constraint for region growth.

6. The method for three-dimensional reconstruction and automatic segmentation of bald patches based on video images according to claim 1, characterized in that: The process of segmenting alopecia areata lesions within the target scalp region using an adaptive clustering algorithm specifically includes: The texture image of the target scalp mesh is converted to the CIELAB color space, and the luminance features of the L channel and the chrominance features of the a and b channels are extracted. The variance of the K-neighborhood normals of the mesh vertices is calculated as a roughness geometric feature; Constructing feature vectors Clustering algorithms are used to divide vertices into hair-covered regions and hair loss patch regions; Morphological opening operations and connected component analysis are performed on the classification results to remove noise regions with an area smaller than a preset threshold, thus establishing the final alopecia areata lesions. Traverse all triangular faces within the alopecia areata region and calculate the area of ​​each triangle. And by summing them up, the precise surface area of ​​the alopecia areata lesion is obtained, where Let the area of ​​the triangular facet be... Let be the vector from vertex A to vertex B of the triangular facet. Let be the vector from vertex A to vertex C of the triangular facet.

7. The method for three-dimensional reconstruction and automatic segmentation of bald patches based on video images according to claim 1, characterized in that: The introduction of Grounded SAM 2 to generate temporally consistent semantic masks for the human head and scalp specifically includes: Use a text encoder to convert prompts into semantic embedding vectors; Multi-scale features of video frames are extracted using a hierarchical image encoder. A memory bank and a memory attention module are introduced to store and retrieve the mask features and image features of the preceding frame, and the occlusion and viewpoint change problems in the video are solved by temporal context fusion. By combining semantic embedding vectors with fused image features using a mask decoder, a temporally consistent instance segmentation mask sequence is output.

8. The method for three-dimensional reconstruction and automatic segmentation of bald patches based on video images according to claim 1, characterized in that: The extraction of the target scalp region also includes boundary curve smoothing: Establish a local coordinate system with the top of the head as the Z-axis, and project the registered scalp boundary points onto the cylindrical coordinate system; Divide the angle domain into N intervals, and select the candidate point with the lowest height in each interval as the boundary point; The extracted boundary point sequence is smoothed using spline interpolation or Gaussian smoothing filtering to generate a continuous closed scalp contour line.

9. A system for three-dimensional reconstruction and automatic segmentation of bald patches based on video images, characterized in that: The device includes a processor, an input device, an output device, and a memory, which are interconnected. The memory stores a computer program, which includes program instructions. The processor is configured to invoke the program instructions to execute the method for three-dimensional reconstruction and automatic segmentation of alopecia areata based on video images as described in any one of claims 1 to 8.

Images