3D digital portrait automatic modeling method based on image feature fusion

Abstract

The invention discloses a 3D digital portrait automatic modeling system based on image feature fusion, and relates to the technical field of image data processing. The system comprises a multi-modal image acquisition module, an image preprocessing module, a layered feature extraction module, a dynamic weight feature fusion module, a 3D geometric reconstruction module, a detail optimization module and a model output module. The core of the method is that portrait multi-dimensional data is acquired through multi-modal image acquisition, color, texture and depth features are extracted in a layered manner after preprocessing, feature adaptive fusion is realized by adopting a dynamic weight algorithm, 3D geometric reconstruction and detail optimization are completed based on fused features, and finally a high-precision 3D digital portrait model is output. The system does not need a large amount of manual intervention, can solve the problems of rough feature fusion, detail loss and low automation degree in the prior art, remarkably improves the precision and efficiency of 3D portrait modeling, and is suitable for virtual fitting, film and television production and other scenes.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, specifically to an automated 3D digital portrait modeling method based on image feature fusion. Background Technology

[0002] With the rapid development of virtual reality, augmented reality, and digital entertainment industries, the demand for 3D digital human portrait modeling is increasing. Its core lies in transforming two-dimensional human portrait information into a realistic and high-precision three-dimensional model through image data processing technology. However, existing 3D digital human portrait modeling technology still has many technical bottlenecks, making it difficult to meet the dual requirements of accuracy and efficiency in practical applications. Existing technologies mostly rely on single-modal image data for modeling; a single RGB image can only provide color and texture information of a human figure, lacking depth dimension data support, resulting in insufficient accuracy of the 3D human figure's contour after modeling, especially in curved transition areas such as the shoulders and waist, where distortion is prone to occur, and the physiological structural features of the human body cannot be accurately reproduced; although some technologies introduce depth images, they only use depth data as an auxiliary supplement, without forming an effective synergy with the color and texture features of the RGB image, resulting in the separation of the feature correlation between the two types of data, limiting the overall realism of the model; Secondly, existing feature fusion methods have obvious defects. Most technologies adopt a fixed-weight feature fusion strategy, that is, the fusion weights of features such as color, texture, and depth are pre-set and remain unchanged during the modeling process. However, the importance of features varies in different regions of a human face. Fixed weights cannot adapt to this regional difference, which can easily lead to the weakening of important features and the overemphasis of secondary features, thus causing feature conflicts. For example, facial texture blurring and contour deformation occur at the same time, increasing the complexity of subsequent model adjustments. Furthermore, the existing system has a low degree of automation and requires a lot of manual intervention. After feature extraction, the effectiveness of the features needs to be judged manually and the fusion parameters need to be adjusted. After the 3D geometric reconstruction is completed, the model's details need to be corrected manually. This manual mode not only prolongs the modeling cycle, but also causes poor consistency of 3D portrait models in different batches due to human error, making it difficult to meet the needs of batch modeling scenarios. Furthermore, existing technologies are not robust enough to image noise and lighting interference. In actual acquisition scenarios, uneven ambient lighting can cause color deviations in RGB images, and device precision limitations can cause noise in depth images. Existing preprocessing technologies can only perform simple noise reduction or brightness adjustment, and cannot specifically compensate for the impact of lighting and noise on feature extraction, resulting in a decrease in the accuracy of subsequent feature extraction, which is ultimately reflected in the loss of details in the 3D model. In summary, existing 3D digital human portrait modeling technologies have significant shortcomings in terms of data modality collaboration, feature fusion adaptability, automation level, and anti-interference ability. There is an urgent need for a modeling system that can achieve efficient fusion of multimodal features, full automation, and high robustness to break through the current technical bottlenecks and promote the large-scale application of 3D digital human portraits in various fields. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides an automated 3D digital portrait modeling method based on image feature fusion, which solves the problems mentioned in the background section.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solution: an automated 3D digital portrait modeling method based on image feature fusion, comprising a multimodal image acquisition module, an image preprocessing module, a hierarchical feature extraction module, a dynamic weighted feature fusion module, a 3D geometric reconstruction module, a detail optimization module, and a model output module electrically connected in sequence to the detail optimization module; The multimodal image acquisition module is used to simultaneously acquire multi-dimensional image data of the target human image; the image preprocessing module is used to perform noise suppression and illumination correction on the acquired image data; the hierarchical feature extraction module is used to extract color features, texture features, and depth features from the preprocessed image in layers; the dynamic weighted feature fusion module is used to calculate the dynamic fusion weight of each feature based on the feature importance evaluation results and complete feature fusion; the 3D geometric reconstruction module is used to generate a preliminary 3D human image geometric model based on the fused feature data; the detail optimization module is used to optimize the microscopic details and contour accuracy of the preliminary geometric model; and the model output module is used to convert the optimized 3D human image model into a standard format and output it.

[0005] Optionally, the multimodal image acquisition module includes an RGB camera, a structured light depth camera, and a near-infrared camera, which achieve synchronized data acquisition through a synchronization triggering unit. The RGB camera is used to acquire color and texture images of the portrait, with a resolution of not less than 4K. The structured light depth camera is used to acquire depth images of the portrait, with a depth measurement error of no more than 2mm. The near-infrared camera is used to acquire skin texture images of the portrait, and can penetrate slight obstructions to obtain information about the underlying skin texture. The synchronization triggering unit controls the shutter trigger time difference of the three cameras to not exceed 1ms through a hardware trigger signal, avoiding multimodal image misalignment caused by slight movements of the portrait.

[0006] Optionally, the image preprocessing module includes a noise suppression unit and an illumination correction unit; the noise suppression unit employs differentiated algorithms for different modal images: for RGB images, a bilateral filtering algorithm is used, the specific formula of which is as follows: ; in For the input image, To output the image, For filtering window, For spatial domain standard deviation, The standard deviation of the grayscale range. The normalization coefficient is used; a median filtering algorithm is used for the depth image, and the window size is set to 3×3 to preserve the depth edge information; the illumination correction unit uses the Retinex algorithm to decompose the illumination component and reflection component of the image to eliminate the uneven illumination caused by backlight and shadow, so that the gray mean of the corrected RGB image is controlled between 120-150 and the gray variance is controlled between 40-60.

[0007] Optionally, the hierarchical feature extraction module includes a color feature extraction unit, a texture feature extraction unit, and a depth feature extraction unit. These three units process the preprocessed multimodal image in parallel. The color feature extraction unit extracts HSV color space features from the RGB image by calculating the H (hue), S (saturation), and V (brightness) values ​​of each pixel to generate an HSV histogram. The histogram has 180 H channels, 256 S channels, and 256 V channels to preserve subtle color differences. The texture feature extraction unit extracts texture features from the RGB and near-infrared images: for the RGB image, a local binary mode algorithm is used to calculate the gray-level difference encoding of pixels within a 3×3 neighborhood to generate an LBP histogram; for the near-infrared image, a gray-level co-occurrence matrix algorithm is used to calculate four texture statistics—contrast, correlation, energy, and homogeneity—at distance 1 and angles of 0° / 45° / 90° / 135°. The depth feature extraction unit extracts depth gradient and curvature features from the depth image: the depth gradient is calculated using the Sobel operator. , The gradient value in the direction is given by the following formula: ; in For depth images, , They are respectively , Orientation gradient; curvature features are calculated by fitting a quadratic surface of a local region of a depth image to identify the concavity and convexity variations of the human figure contour.

[0008] Optionally, the dynamic weighted feature fusion module includes a feature importance evaluation unit and a feature fusion unit; the feature importance evaluation unit calculates the weight of each feature based on feature entropy and feature confidence: feature entropy The formula used to measure the information richness of a feature is as follows: ; in Features The The probability of each component; feature confidence. The confidence level is measured by comparing the consistency of multimodal features in the same region; a higher matching degree results in a higher confidence level. (Weight) The specific calculation formula is as follows: ; in The total number of features participating in the fusion; the feature fusion unit uses a weighted summation method to achieve feature fusion. After normalizing the dimensions of different features, they are superimposed according to their weights to generate a fused feature vector. The specific formula is as follows: ; in For the first Normalized vectors of features, To fuse feature vectors; and during the fusion process, after processing each portrait region, the system automatically recalculates the feature weights of that region, achieving adaptive region fusion.

[0009] Optionally, the 3D geometric reconstruction module includes a point cloud generation unit and a mesh construction unit. The point cloud generation unit maps the fused feature vectors into three-dimensional spatial points: using the depth value in the depth feature as the z-axis coordinate and the pixel coordinates (x, y) of the RGB image as the two-dimensional plane coordinates, and combining color and texture features to assign color and texture attributes to each spatial point to generate an initial point cloud. The mesh construction unit uses the Poisson reconstruction algorithm to mesh the initial point cloud: by constructing the indicator function of the point cloud, the Poisson equation is solved to obtain a three-dimensional implicit surface, and then a triangular mesh model is generated by extracting isosurfaces. During the reconstruction process, a dense mesh is used for the facial region of the human face, and a regular mesh is used for the torso region to balance model accuracy and computational efficiency.

[0010] Optionally, the detail optimization module includes a micro-detail repair unit and a contour accuracy adjustment unit. The micro-detail repair unit repairs details of the mesh model: for skin pores, it transfers texture features extracted from near-infrared images to the mesh surface and uses a texture mapping algorithm to fill in the missing pore areas; for hair strands, it analyzes the direction and density of hair strands in RGB images to generate a three-dimensional line segment model of hair strands, which is then fused with the head mesh model. The contour accuracy adjustment unit fine-tunes the model contour based on depth gradient features and curvature features: for curved areas such as shoulders and waist, it corrects the smoothness of the contour by fitting a quadratic curve; for facial features, it adjusts the positional deviation of the contour by comparing the edge information after feature fusion, so that the proportions of the facial features are consistent with a real human image. All optimization operations are completed automatically by the algorithm without the need for manual input of adjustment parameters.

[0011] Optionally, the model output module includes a format conversion unit and a model verification unit. The format conversion unit supports converting the optimized 3D human portrait model into various standard formats, including OBJ, STL, and FBX. The OBJ format retains the model's texture coordinates and material information, the STL format is suitable for 3D printing scenarios, and the FBX format supports skeletal binding information. The model verification unit determines whether the model's accuracy meets the standard by calculating the feature matching degree between the model and the original multimodal image: if the matching degree is ≥90%, the model is directly output; if the matching degree is <90%, it automatically returns to the detail optimization module for re-optimization until the matching degree meets the standard. The verification result generates a log file, recording the model's accuracy parameters and processing time, which is convenient for subsequent traceability and parameter optimization.

[0012] Optionally, the multimodal image acquisition module further includes an image alignment unit. After acquisition, this unit uses the RGB image as a reference and finds the corresponding feature points of the depth image, near-infrared image and RGB image through the SIFT feature matching algorithm. Then, it uses homography matrix transformation to achieve pixel-level alignment of the three images, with an alignment error of no more than 1 pixel. During the alignment process, if the number of matched feature points is less than 30, the system will automatically trigger re-acquisition to avoid subsequent feature extraction errors due to alignment failure.

[0013] Optionally, the dynamic weighted feature fusion module further includes a feature conflict resolution unit. When different modal features conflict, this unit calculates the confidence level of the conflicting features and retains the feature with higher confidence. For example, if the RGB color feature of the shadow area has low confidence and the near-infrared texture feature has high confidence, then the near-infrared texture feature is retained first, and the deviation of the RGB color feature is corrected based on this feature. Through feature conflict resolution, the fused feature vector is not distorted due to conflicting features, thereby further improving the accuracy of 3D geometric reconstruction.

[0014] This invention provides an automated 3D digital portrait modeling method based on image feature fusion, which has the following beneficial effects: This automated 3D digital portrait modeling method based on image feature fusion effectively solves the core pain points of existing 3D portrait modeling technologies through multi-dimensional technological innovation. It has significant advantages in both technical performance and practical application value, specifically in the following aspects: The system overcomes the limitations of existing technologies that rely on single-modal images by employing a multimodal image acquisition module to simultaneously acquire RGB, depth, and near-infrared images. Furthermore, it ensures the consistency of multimodal data through synchronous triggering and image alignment units. The introduction of multimodal data not only compensates for the lack of depth information in single RGB images but also acquires skin texture (such as pores and fine wrinkles) from near-infrared images, providing comprehensive data support for subsequent high-precision modeling. The image alignment and re-acquisition mechanism further avoids feature extraction errors caused by data misalignment, ensuring the quality of modeling data from the source. Furthermore, the system innovatively proposes a dynamic weight fusion strategy, which has stronger adaptability and accuracy compared to the fixed weight of existing technologies. The dynamic weight is calculated in real time based on feature entropy and feature confidence, and automatically adjusts the weight for different regions of the portrait (such as face and torso). This allows important features (such as facial texture and contour depth) to dominate the fusion process, while secondary features provide reasonable assistance, effectively avoiding the problem of "weakened important features and overemphasis on secondary features" caused by fixed weights. At the same time, the addition of a feature conflict resolution unit can actively identify and handle conflicts of multimodal features, ensuring that the fused feature vector truly reflects the portrait features and provides high-quality feature input for 3D geometric reconstruction. Meanwhile, the system achieves full automation from image acquisition to model output, completely changing the existing technology's reliance on manual intervention. Differential noise suppression and illumination correction in the preprocessing stage, parallel processing of feature extraction, automatic parameter adjustment for detail optimization, and automatic verification and re-optimization of model output are all completed automatically by algorithms, without the need for manual input to adjust parameters or correct model defects. This fully automated mode not only significantly shortens the modeling cycle but also avoids human error, significantly improving model consistency in batch modeling scenarios. The system also achieves high precision and realism in 3D human portrait models through layered feature extraction and targeted optimization strategies. The layered extraction of color, texture, and depth features can completely preserve the multi-dimensional information of the human portrait, and the dynamically fused feature vectors provide a precise basis for geometric reconstruction. Micro-detail repair (such as pore texture migration and hair generation) and contour precision adjustment (such as curve area smoothing and facial feature proportion correction) are also achieved.

[0015] Furthermore, the system boasts excellent compatibility and practicality. The model output module supports multiple standard formats, adapting to various downstream application scenarios, and the FBX format includes a reserved bone binding interface, facilitating subsequent animation production. The model verification unit and log recording function not only ensure the accuracy of the output model but also allow users to trace the modeling process, optimize algorithm parameters, and improve system maintainability. In summary, this system achieves breakthroughs in data collaboration, feature fusion, automation, and accuracy restoration, providing an efficient and reliable technical solution for 3D digital human portrait modeling. It can powerfully promote the large-scale application of 3D digital human portraits in fields such as the virtual economy, digital entertainment, and medical aesthetics (e.g., virtual plastic surgery effect preview). Detailed Implementation

[0016] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0017] This invention provides a technical solution: an automated 3D digital portrait modeling method based on image feature fusion, comprising a multimodal image acquisition module, an image preprocessing module, a hierarchical feature extraction module, a dynamic weighted feature fusion module, a 3D geometric reconstruction module, a detail optimization module, and a model output module electrically connected in sequence to the detail optimization module; The multimodal image acquisition module is used to synchronously acquire multi-dimensional image data of the target human image. The multimodal image acquisition module includes an RGB camera, a structured light depth camera, and a near-infrared camera, which are synchronized for data acquisition through a synchronization trigger unit. The RGB camera is used to acquire color and texture images of the human image, with a resolution of no less than 4K. The structured light depth camera is used to acquire depth images of the human image, with a depth measurement error of no more than 2mm. The near-infrared camera is used to acquire skin texture images of the human image, and can penetrate slight obstructions (such as thin clothing) to obtain information about the underlying skin texture. The synchronization trigger unit controls the shutter trigger time difference of the three cameras to be no more than 1ms through hardware trigger signals to avoid multimodal image misalignment caused by slight movements of the human image. The multimodal image acquisition module also includes an image alignment unit. After acquisition, this unit uses the RGB image as a reference and finds the corresponding feature points of the depth image, near-infrared image and RGB image through the SIFT feature matching algorithm. Then, it uses homography matrix transformation to achieve pixel-level alignment of the three images with an alignment error of no more than 1 pixel. During the alignment process, if the number of matched feature points is less than 30 (i.e., the image is severely misaligned), the system will automatically trigger re-acquisition to avoid subsequent feature extraction errors due to alignment failure. The image preprocessing module is used to perform noise suppression and illumination correction on the acquired image data. The image preprocessing module includes a noise suppression unit and an illumination correction unit. The noise suppression unit employs differentiated algorithms for different modalities of the image: for RGB images, a bilateral filtering algorithm is used, with the specific formula as follows: ; in For the input image, To output the image, For filtering window, For spatial domain standard deviation, The standard deviation of the grayscale range. The normalization coefficient is used; the median filtering algorithm is used for the depth image, and the window size is set to 3×3 to preserve the depth edge information; the illumination correction unit uses the Retinex algorithm, which decomposes the illumination component and reflection component of the image to eliminate the uneven illumination caused by backlight and shadow, so that the gray mean of the corrected RGB image is controlled between 120-150 and the gray variance is controlled between 40-60. The hierarchical feature extraction module is used to extract color, texture, and depth features from the preprocessed image in layers. This module includes a color feature extraction unit, a texture feature extraction unit, and a depth feature extraction unit, which process the preprocessed multimodal image in parallel. The color feature extraction unit extracts HSV color space features from the RGB image by calculating the H (hue), S (saturation), and V (brightness) values ​​of each pixel to generate an HSV histogram. The histogram bins are set to 180 H channels, 256 S channels, and 256 V channels to preserve color space. The system retains subtle color differences. The texture feature extraction unit extracts texture features from RGB and near-infrared images: for RGB images, the Local Binary Mode (LBP) algorithm is used to calculate the gray-level difference encoding of pixels within a 3×3 neighborhood, generating an LBP histogram; for near-infrared images, the Gray-Level Co-occurrence Matrix (GLCM) algorithm is used to calculate four texture statistics: contrast, correlation, energy, and homogeneity at distance 1 and angles of 0° / 45° / 90° / 135°. The depth feature extraction unit extracts depth gradient and curvature features from the depth image: the depth gradient is calculated using the Sobel operator. , The gradient value in the direction is given by the following formula: ; in For depth images, , They are respectively , Orientation gradient; curvature features are calculated by fitting the quadratic surface of a local region of the depth image to identify the concavity and convexity of the human figure contour (such as a protruding bridge of the nose and sunken eye sockets). The dynamic weighted feature fusion module is used to calculate the dynamic fusion weights of each feature and complete feature fusion based on the feature importance evaluation results. The dynamic weighted feature fusion module includes a feature importance evaluation unit and a feature fusion unit. The feature importance evaluation unit calculates the weights of each feature based on feature entropy and feature confidence: feature entropy... The formula used to measure the information richness of a feature is as follows: ; in Features The The probability of each component; feature confidence. The confidence level is measured by comparing the consistency of multimodal features in the same region (e.g., the matching degree between RGB textures and near-infrared textures); the higher the matching degree, the higher the confidence level. (Weight) The calculation formula is as follows: ; in The total number of features participating in the fusion; the feature fusion unit uses a weighted summation method to achieve feature fusion. After normalizing the dimensions of different features (normalized to the [0, 1] interval), the fused feature vector is generated by superimposing the weights. The specific formula is as follows: ; in For the first Normalized vectors of features, To fuse feature vectors; and during the fusion process, after processing each portrait region (such as face, torso, limbs), the system automatically recalculates the feature weights of that region to achieve adaptive region fusion; The dynamic weighted feature fusion module also includes a feature conflict resolution unit. When different modal features conflict (e.g., an RGB image shows a shadow on a face, while a near-infrared image shows no shadow in that area), this unit calculates the confidence level of the conflicting features and retains the feature with higher confidence. For example, if the RGB color feature of the shadow area has low confidence and the near-infrared texture feature has high confidence, then the near-infrared texture feature is retained first, and the deviation of the RGB color feature is corrected based on this feature. Through feature conflict resolution, the fused feature vector is not distorted due to conflicting features, further improving the accuracy of 3D geometric reconstruction. The 3D geometry reconstruction module is used to generate a preliminary 3D geometric model of a human face based on the fused feature data. The module includes a point cloud generation unit and a mesh construction unit. The point cloud generation unit maps the fused feature vectors into three-dimensional spatial points: using the depth value from the depth features as the z-axis coordinate and the pixel coordinates (x, y) of the RGB image as the two-dimensional plane coordinates, it assigns color and texture attributes to each spatial point by combining color and texture features, thus generating an initial point cloud. The mesh construction unit uses the Poisson reconstruction algorithm to mesh the initial point cloud: by constructing an indicator function for the point cloud, the Poisson equation is solved to obtain a three-dimensional implicit surface, and then isosurfaces are extracted to generate a triangular mesh model. During the reconstruction process, a dense mesh (mesh side length not exceeding 0.5mm) is used for the facial region of the human face, while a regular mesh (mesh side length not exceeding 1mm) is used for the torso region to balance model accuracy and computational efficiency. The detail optimization module is used to optimize the microscopic details and contour accuracy of the initial geometric model. This module includes a microscopic detail repair unit and a contour accuracy adjustment unit. The microscopic detail repair unit addresses detail defects in the mesh model: for skin pores, it transfers texture features extracted from near-infrared images to the mesh surface and uses a texture mapping algorithm to fill in missing pore areas; for hair strands, it analyzes the direction and density of hair strands in RGB images to generate a 3D line segment model of the hair strands, which is then fused with the head mesh model. The contour accuracy adjustment unit fine-tunes the model contour based on depth gradient and curvature features: for curved areas such as the shoulders and waist, it corrects the smoothness of the contour by fitting quadratic curves; for facial features (such as eye sockets and lip lines), it adjusts the positional deviation of the contour by comparing the edge information after feature fusion, ensuring that the proportions of the facial features are consistent with a real human image. All optimization operations are completed automatically by the algorithm, requiring no manual input of adjustment parameters. The model output module converts the optimized 3D human portrait model into a standard format and outputs it. The module includes a format conversion unit and a model verification unit. The format conversion unit supports converting the optimized 3D human portrait model into various standard formats, including OBJ, STL, and FBX. OBJ format retains the model's texture coordinates and material information, STL format is suitable for 3D printing scenarios, and FBX format supports skeletal binding information (reserving an interface for subsequent animation production). The model verification unit calculates the feature matching degree between the model and the original multimodal image to determine if the model's accuracy meets the standard: if the matching degree is ≥90%, the model is directly output; if the matching degree is <90%, it automatically returns to the detail optimization module for re-optimization until the matching degree meets the standard. The verification results generate a log file, recording the model's accuracy parameters (such as feature matching degree and mesh density) and processing time, facilitating subsequent traceability and parameter optimization. Case 1: 3D avatar modeling of users on a virtual fitting platform. The application scenario is as follows: An e-commerce platform launched a virtual try-on feature, which required the rapid generation of high-precision 3D human portrait models for platform users. After uploading their models, users could try on virtual clothing on the platform and intuitively view the matching effect of the clothing's cut, color, and themselves. This scenario required a short modeling cycle (≤5 minutes per user), high model accuracy (able to reproduce body contours and skin texture), and a high degree of automation (users do not need to have professional modeling knowledge). Traditional modeling techniques could not meet these requirements, so the system of this invention was adopted. The specific steps of system implementation include: S1. Multimodal Image Acquisition: The platform deploys multimodal acquisition equipment (including a 4K RGB camera, a structured light depth camera, and a near-infrared camera) in offline experience stores. Users stand in the acquisition area (with a solid-color backdrop to avoid interference). The system controls the three cameras to acquire images simultaneously through a synchronous trigger unit, with an acquisition time of approximately 2 seconds. After acquisition, the image alignment unit uses the RGB image as a reference and finds more than 300 corresponding feature points through SIFT feature matching. It then uses homography matrix transformation to achieve pixel-level alignment of the three images, with the alignment error controlled within 1 pixel to ensure data consistency. S2. Apply bilateral filtering to the RGB image. Environmental noise was removed, and slight backlighting during acquisition was eliminated using the Retinex algorithm, stabilizing the image grayscale mean at 130±2. A 3×3 median filter was applied to the depth image to preserve the depth edge information of the shoulders and waist. The same bilateral filtering parameters as the RGB image were applied to the near-infrared image to remove device noise and highlight skin texture. S3. Three feature extraction units process the preprocessed image in parallel. The color feature extraction unit generates an HSV histogram (180 bins in the H channel and 256 bins in the S / V channel) to accurately capture the difference between the user's clothing color and skin tone. The texture feature extraction unit calculates the LBP histogram (3×3 neighborhood) for the RGB image and calculates the GLCM statistics (contrast 15.2, correlation 0.8, energy 0.3, homogeneity 0.7) for the near-infrared image to completely extract skin pores and clothing fabric texture. The depth feature extraction unit calculates the depth gradient using the Sobel operator. The mean values ​​are 8.5 and 7.2 respectively. By fitting a quadratic surface, the principal curvature values ​​are obtained (the principal curvature of the bridge of the nose is positive and that of the eye socket area is negative), and the concave and convex changes of the body contour are identified. S4. Dynamic weighted feature fusion: The feature importance evaluation unit calculates the weight of each feature; in the facial region, the entropy of near-infrared texture features. =7.8 and confidence level =0.95, all values ​​are high, weight is set to 0.4; RGB color feature entropy =6.5, confidence level =0.9, weight set to 0.3; deep feature entropy =5.2, Confidence level =0.85, weight set to 0.3; confidence level of depth features in the torso region. =0.92 height (clear contour information), weight set to 0.5; RGB color feature weight 0.3, near-infrared texture feature weight 0.2; the feature fusion unit superimposes the normalized feature vector according to the weight, while the feature conflict resolution unit handles one conflict (the RGB image shows a slight shadow on the shoulder, while the near-infrared image has no shadow), retains the near-infrared texture feature with higher confidence, and corrects the RGB color deviation. S5, 3D geometric reconstruction: The point cloud generation unit uses depth as the z-axis and RGB pixel coordinates as the x / y-axis to generate an initial point cloud containing 500,000 spatial points, with each point assigned color and texture attributes; The mesh building unit uses Poisson reconstruction, with a mesh edge length of 0.4mm (refined) for the facial region and 1mm (normal) for the torso region, generating a triangular mesh model with approximately 800,000 vertices and approximately 1.5 million faces; The S6 micro-detail repair unit transfers near-infrared texture to the facial mesh, fills in areas lacking pores, and generates a 3D hair model based on the hair direction of the RGB image (hair diameter 0.05mm, density approximately 100 hairs / mm²); the contour accuracy adjustment unit fits the quadratic curve of the shoulder, corrects the contour smoothness, compares the lip line edge information, and adjusts the lip line position deviation (from 0.3mm to 0.1mm) to ensure accurate facial proportions; S7. Model Output: The format conversion unit converts the model to OBJ format (preserving texture and material). The model verification unit calculates the feature matching degree to be 94% (≥90%) and outputs the model directly. At the same time, a log file is generated to record the matching degree of 94% and the mesh density of 0.4~1mm.

[0018] Users import OBJ format models into the virtual fitting system and try on a dress: the model can accurately fit the dress shape (with no obvious gaps or tightness at the shoulders and waist), and the skin pores and fabric texture of the dress are naturally superimposed. Users can view the fitting effect from 360° and intuitively judge whether the length and waist of the dress fit well.

[0019] Case Study 2: 3D Character Modeling for Film and Television. A film company needed to create a 3D digital avatar of the protagonist for a realistic film (for post-production special effects scenes, such as the protagonist interacting with virtual scenes). The model was required to highly reproduce the actor's facial details and body contours, and support subsequent skeletal rigging and animation production. This system was used for implementation. During the acquisition phase, higher-precision equipment (8K RGB camera, 1mm depth camera error) was used to capture multi-angle (front, side, 45°) multimodal images of the actors. Feature extraction focused on optimizing facial features (such as wrinkles and eye bag texture), with the weight of facial texture features in the dynamic weighting increased to 0.5. For detail optimization, unique features of the actors (such as a small scar above the left eyebrow) were addressed, and the scar's shape (3mm in length, 0.2mm in width) was accurately reconstructed using edge information after feature fusion. Output was converted to FBX format while retaining the bone binding interface. The final model achieved a 96% feature matching rate with the actors, allowing the post-production team to complete bone binding in just one day. During animation production, the model's movements closely matched the actors' real movements, resulting in realistic visual effects and significantly shortening the post-production cycle.

[0020] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An automated 3D digital portrait modeling method based on image feature fusion, comprising a multimodal image acquisition module, an image preprocessing module, a hierarchical feature extraction module, a dynamic weighted feature fusion module, a 3D geometric reconstruction module, a detail optimization module, and a model output module electrically connected in sequence to the detail optimization module; The multimodal image acquisition module is used to simultaneously acquire multi-dimensional image data of the target human image; the image preprocessing module is used to perform noise suppression and illumination correction on the acquired image data; the layered feature extraction module is used to extract color features, texture features and depth features from the preprocessed image in layers; The dynamic weighted feature fusion module is used to calculate the dynamic fusion weight of each feature and complete the feature fusion based on the feature importance evaluation results. The 3D geometry reconstruction module is used to generate a preliminary 3D human portrait geometric model based on the fused feature data; the detail optimization module is used to optimize the microscopic details and contour accuracy of the preliminary geometric model; and the model output module is used to convert the optimized 3D human portrait model into a standard format and output it.

2. The system according to claim 1, characterized in that, The multimodal image acquisition module includes an RGB camera, a structured light depth camera, and a near-infrared camera, which achieve synchronized data acquisition through a synchronization triggering unit. The RGB camera is used to acquire color and texture images of the portrait, with a resolution of no less than 4K. The structured light depth camera is used to acquire depth images of the portrait, with a depth measurement error of no more than 2mm. The near-infrared camera is used to acquire skin texture images of the portrait, and can penetrate slight obstructions to obtain information about the underlying skin texture. The synchronization triggering unit controls the shutter trigger time difference of the three cameras to be no more than 1ms through a hardware trigger signal to avoid multimodal image misalignment caused by slight movements of the portrait.

3. The system according to claim 1, characterized in that, The image preprocessing module includes a noise suppression unit and an illumination correction unit; the noise suppression unit employs differentiated algorithms for different modal images: for RGB images, a bilateral filtering algorithm is used, the specific formula of which is as follows: ； in For the input image, To output the image, For filtering window, For spatial domain standard deviation, The standard deviation of the grayscale range. The normalization coefficient is used; a median filtering algorithm is used for the depth image, and the window size is set to 3×3 to preserve the depth edge information; the illumination correction unit uses the Retinex algorithm to decompose the illumination component and reflection component of the image to eliminate the uneven illumination caused by backlight and shadow, so that the gray mean of the corrected RGB image is controlled between 120-150 and the gray variance is controlled between 40-60.

4. The system according to claim 1, characterized in that, The hierarchical feature extraction module includes a color feature extraction unit, a texture feature extraction unit, and a depth feature extraction unit. These three units process the preprocessed multimodal image in parallel. The color feature extraction unit extracts HSV color space features from the RGB image by calculating the H (hue), S (saturation), and V (brightness) values ​​of each pixel to generate an HSV histogram. The histogram bins are set to 180 H channels, 256 S channels, and 256 V channels to preserve subtle color differences. The texture feature extraction unit extracts texture features from the RGB and near-infrared images. For RGB images, a Local Binary Mode (LBP) algorithm is used to calculate the gray-level difference encoding of pixels within a 3×3 neighborhood, generating an LBP histogram. For near-infrared images, a Gray-Level Co-occurrence Matrix (GLCM) algorithm is used to calculate four texture statistics: contrast, correlation, energy, and homogeneity at distance 1 and angles of 0° / 45° / 90° / 135°. The depth feature extraction unit extracts depth gradient and curvature features from the depth image: the depth gradient is calculated using the Sobel operator. , The gradient value in the direction is given by the following formula: ； in For depth images, , They are respectively , Orientation gradient; Curvature features are used to calculate principal curvature values ​​by fitting a quadratic surface to a local region of a depth image, in order to identify the concavity and convexity variations of the human figure's contour.

5. The system according to claim 1, characterized in that, The dynamic weighted feature fusion module includes a feature importance evaluation unit and a feature fusion unit; The feature importance evaluation unit calculates the weight of each feature based on feature entropy and feature confidence: feature entropy The formula used to measure the information richness of a feature is as follows: ； in Features The The probability of each component; feature confidence. The confidence level is measured by comparing the consistency of multimodal features in the same region; a higher matching degree results in a higher confidence level. (Weight) The specific calculation formula is as follows: ； in The total number of features participating in the fusion; the feature fusion unit uses a weighted summation method to achieve feature fusion. After normalizing the dimensions of different features, they are superimposed according to their weights to generate a fused feature vector. The specific formula is as follows: ； in For the first Normalized vectors of features, To fuse feature vectors; and during the fusion process, after processing each portrait region, the system automatically recalculates the feature weights of that region, achieving adaptive region fusion.

6. The system according to claim 1, characterized in that, The 3D geometric reconstruction module includes a point cloud generation unit and a mesh construction unit. The point cloud generation unit maps the fused feature vectors into three-dimensional spatial points: using the depth value in the depth feature as the z-axis coordinate and the pixel coordinates (x, y) of the RGB image as the two-dimensional plane coordinates, and combining color and texture features to assign color and texture attributes to each spatial point to generate an initial point cloud. The mesh construction unit uses the Poisson reconstruction algorithm to mesh the initial point cloud: by constructing the indicator function of the point cloud, the Poisson equation is solved to obtain a three-dimensional implicit surface, and then a triangular mesh model is generated by extracting isosurfaces. During the reconstruction process, a dense mesh is used for the facial area of ​​the portrait, and a regular mesh is used for the torso area to balance model accuracy and computational efficiency.

7. The system according to claim 1, characterized in that, The detail optimization module includes a micro-detail repair unit and a contour accuracy adjustment unit; the micro-detail repair unit repairs the detail defects of the mesh model: for skin pores, the texture features extracted from near-infrared images are transferred to the mesh surface, and a texture mapping algorithm is used to fill the missing areas of the pores; For hair strands, a 3D line segment model of the hair strands is generated by analyzing the direction and density of the hair strands in the RGB image, and then fused with the head mesh model. The contour accuracy adjustment unit fine-tunes the model contour based on depth gradient features and curvature features: for curved areas such as shoulders and waist, the smoothness of the contour is corrected by fitting a quadratic curve; for facial features, the positional deviation of the contour is adjusted by comparing the edge information after feature fusion, so that the proportions of the facial features are consistent with the real human image; and all optimization operations are completed automatically by the algorithm without the need for manual input of adjustment parameters.

8. The system according to claim 1, characterized in that, The model output module includes a format conversion unit and a model verification unit. The format conversion unit supports converting the optimized 3D human portrait model into various standard formats, including OBJ, STL, and FBX. OBJ format retains the model's texture coordinates and material information, STL format is suitable for 3D printing scenarios, and FBX format supports skeletal binding information. The model verification unit determines whether the model's accuracy meets the standard by calculating the feature matching degree between the model and the original multimodal image: if the matching degree is ≥90%, the model is directly output; if the matching degree is <90%, it automatically returns to the detail optimization module for re-optimization until the matching degree meets the standard. The verification results generate a log file, recording the model's accuracy parameters and processing time, facilitating subsequent traceability and parameter optimization.

9. The system according to claim 2, characterized in that, The multimodal image acquisition module also includes an image alignment unit. After acquisition, this unit uses the RGB image as a reference, finds the corresponding feature points of the depth image, near-infrared image and RGB image through the SIFT feature matching algorithm, and then uses homography matrix transformation to achieve pixel-level alignment of the three images with an alignment error of no more than 1 pixel. Furthermore, during the alignment process, if the number of matching feature points is less than 30, the system will automatically trigger a re-acquisition to avoid subsequent feature extraction errors due to alignment failure.

10. The system according to claim 5, characterized in that, The dynamic weighted feature fusion module also includes a feature conflict resolution unit. When different modal features conflict, this unit calculates the confidence level of the conflicting features and retains the feature with higher confidence. For example, if the RGB color feature of the shadow area has low confidence and the near-infrared texture feature has high confidence, then the near-infrared texture feature is retained first, and the deviation of the RGB color feature is corrected based on this feature. Through feature conflict resolution, the fused feature vector is not distorted due to conflicting features, thereby further improving the accuracy of 3D geometric reconstruction.

Images