Nipple protection area three-dimensional reconstruction method and system based on super-resolution

Abstract

The invention discloses a nipple protection area three-dimensional reconstruction method and system based on super-resolution, and relates to the technical field of image processing, and the method comprises the steps: obtaining a two-dimensional slice image of a nipple protection area through a medical image device; performing preliminary three-dimensional structure estimation on the two-dimensional slice image to obtain spatial geometric constraints; constructing a super-resolution network based on three-dimensional structure guidance for the two-dimensional image according to geometric constraints; and dense high-precision point clouds are generated through multi-view stereo matching for three-dimensional reconstruction. According to the method, two-dimensional image enhancement and three-dimensional structure constraint are fused through a three-dimensional structure guided super-resolution method, so that the detail information of the image is enhanced while the three-dimensional reconstruction precision of the nipple protection area is improved; and by adopting innovative structure consistency loss and depth constraint loss, the structure continuity and the space consistency of the reconstruction model are effectively improved, and the problem of three-dimensional structure distortion possibly occurring in a traditional method is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a three-dimensional reconstruction method and system for the nipple protection area based on super-resolution. Background Art

[0002] With the rapid development of medical imaging technology, three-dimensional reconstruction technology has gradually become an important auxiliary means for medical diagnosis, surgical planning and postoperative evaluation, and is particularly widely used in fields such as breast surgery and plastic surgery. At present, most of the three-dimensional reconstruction methods for medical images are processed based on traditional two-dimensional medical images (such as MRI, CT, ultrasound) or two-dimensional photo sequences, and then computer vision technology is used to construct three-dimensional models. However, due to the fine structure and subtle anatomical features of medical images, traditional imaging devices are limited by device resolution and scanning accuracy during the imaging process, resulting in insufficient clarity of the obtained two-dimensional images and blurred tissue boundaries, which affect the accuracy and realism of three-dimensional reconstruction. Therefore, in order to improve image clarity, some studies have tried to first enhance the resolution of two-dimensional images through image super-resolution algorithms and then perform three-dimensional reconstruction, achieving certain results. However, traditional super-resolution algorithms mostly ignore the guiding role of three-dimensional spatial structure information, resulting in the lack of spatial consistency in the enhanced images and the problems of structural distortion and detail loss in the reconstructed three-dimensional models.

[0003] Specifically, when the prior art performs super-resolution enhancement on medical images, it generally does not consider three-dimensional structure constraint information and is only limited to pixel-level optimization of two-dimensional images, resulting in difficulty in ensuring the spatial consistency of the images after super-resolution. In addition, most of the existing three-dimensional reconstruction technologies adopt the method of directly performing dense reconstruction from two-dimensional images. However, due to the lack of effective three-dimensional structure prior guidance, the spatial geometric relationship cannot be fully utilized to improve the reconstruction accuracy, resulting in insufficient refinement of the three-dimensional models of medical images and unclear expression of local details (such as blood vessels, tissue edges, skin textures). This will have an adverse impact on the application of diagnosis and surgical planning in the medical field and is difficult to meet the requirements of precision medicine for model accuracy and authenticity. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed.

[0005] Therefore, the technical problem to be solved by the present invention is: how to solve the problems of insufficient image spatial consistency and low model accuracy existing in the three-dimensional reconstruction process of medical images.

[0006] To solve the above technical problem, the present invention provides the following technical solution: A three-dimensional reconstruction method for the nipple protection area based on super-resolution, including: Obtaining two-dimensional slice images of the nipple protection area through a medical imaging device; Perform preliminary three-dimensional structure estimation on two-dimensional slice images to obtain spatial geometric constraints; Build a super-resolution network guided by three-dimensional structure for two-dimensional images according to the geometric constraints; Perform three-dimensional reconstruction by generating dense and high-precision point clouds through multi-view stereo matching.

[0007] As a preferred scheme of the three-dimensional reconstruction method of the nipple protection area based on super-resolution according to the present invention, wherein: the two-dimensional slice images include collecting low-resolution images of the nipple protection area from multiple angles and viewpoints to obtain several groups of two-dimensional image data sequences with different viewpoints; Preprocess the collected images, including denoising, deblurring, and illumination normalization, preliminarily calibrate the internal and external parameters of the camera, and correct the distortion; output the processed low-resolution image sequence.

[0008] As a preferred scheme of the three-dimensional reconstruction method of the nipple protection area based on super-resolution according to the present invention, wherein: the performing of the preliminary three-dimensional structure estimation includes using the preprocessed low-resolution image sequence, adopting a structure from motion recovery algorithm, and extracting feature points of each view image; Obtain reliable matching points through feature matching methods; Based on the reliable matching points, estimate the internal and external parameters of each view camera, and obtain a preliminary coarse-grained three-dimensional sparse point cloud structure; Use the initial coarse-grained three-dimensional structure as the spatial constraint information for super-resolution processing.

[0009] As a preferred scheme of the three-dimensional reconstruction method of the nipple protection area based on super-resolution according to the present invention, wherein: the building of the super-resolution network guided by three-dimensional structure includes, Prepare the network input, and the input data is the preliminary coarse-grained three-dimensional structure and the preprocessed multi-view low-resolution two-dimensional images; Use a shallow convolutional network to extract basic image features from the low-resolution image sequence and output a basic two-dimensional feature map ; Introduce the preliminary three-dimensional structure information into the network, and fuse the three-dimensional structure features and the two-dimensional image features through an attention mechanism; Apply spatial constraint information to the two-dimensional features and output a structure-guided fused feature map ; Take the fused feature map as the input, and perform efficient upsampling processing using the sub-pixel convolution method to output a super-resolution enhanced two-dimensional high-quality image; Train the super-resolution network, and select the real high-resolution images of the nipple protection area as the supervised data for the training set; Define a hybrid loss function, and use the Adam optimizer to iteratively optimize the parameters of the super-resolution network until convergence.

[0010] As a preferred embodiment of the three-dimensional reconstruction method for the nipple protection area based on super-resolution according to the present invention, wherein: the defined hybrid loss function includes content loss, perceptual loss, structural consistency loss, and depth constraint loss; The specific definition includes that the content loss measures the pixel-level gap of the image; The perceptual loss supervises the semantic features of the reconstructed image; The structural consistency loss constrains the continuity of the pixel feature space of different perspectives through the consistency of corresponding points in the three-dimensional space; The depth constraint loss provides additional constraints for network training by using the depth information of the preliminary three-dimensional structure.

[0011] As a preferred embodiment of the three-dimensional reconstruction method for the nipple protection area based on super-resolution according to the present invention, wherein: the generation of a dense and high-precision point cloud includes performing multi-view stereo matching using the camera parameters generated in the preliminary three-dimensional structure estimation and the high-resolution image enhanced by the super-resolution network; Utilize the disparity estimation algorithm to calculate the disparity, generate the depth map, and fuse the multi-view depth information to obtain a high-density three-dimensional point cloud of the nipple protection area; Denoise, filter, and optimize the obtained high-density point cloud to remove clutter and isolated points and improve the clarity and accuracy of the point cloud.

[0012] As a preferred embodiment of the three-dimensional reconstruction method for the nipple protection area based on super-resolution according to the present invention, wherein: the three-dimensional reconstruction includes performing meshing processing on the optimized high-density point cloud using the Poisson reconstruction algorithm to generate a fine three-dimensional surface mesh model of the nipple protection area; Further optimize the mesh model, including: Mesh smoothing and removing redundant vertices, repairing holes and irregular edges, and streamlining the model structure; Perform high-precision texture mapping optimization on the high-resolution image enhanced based on super-resolution using the multi-view texture mapping method.

[0013] As a preferred embodiment of the three-dimensional reconstruction system for the nipple protection area based on super-resolution according to the present invention, wherein: it includes a data acquisition module, a preliminary estimation module, a 3D-guided super-resolution processing module, and a three-dimensional reconstruction module; The data acquisition module includes a data acquisition unit and an image processing unit; it is used to collect multi-angle low-resolution two-dimensional images and depth scan data around the nipple protection area and complete the preliminary processing of the nipple protection area data; The preliminary estimation module includes a feature extraction unit, a parameter estimation unit, and an initial structure generation unit; it is used to realize the estimation of the preliminary three-dimensional structure and output the coarse-grained three-dimensional space prior information; The 3D-guided super-resolution processing module is used to perform super-resolution processing on the two-dimensional image of the nipple protection area by using three-dimensional structure information as a guide; The three-dimensional reconstruction module includes a reconstruction unit and a texture mapping unit, and is used to perform high-precision three-dimensional reconstruction of the nipple protection area based on the super-resolution image and output a refined three-dimensional model.

[0014] A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the three-dimensional reconstruction method of the nipple protection area based on super-resolution.

[0015] A computer-readable storage medium stores a computer program thereon. When the computer program is executed by a processor, the steps of the three-dimensional reconstruction method of the nipple protection area based on super-resolution are implemented.

[0016] The beneficial effects of the present invention: Through the super-resolution method guided by three-dimensional structure, the two-dimensional image enhancement and three-dimensional structure constraint are fused, which not only improves the accuracy of three-dimensional reconstruction of the nipple protection area, but also enhances the detail information of the image; The innovative structure consistency loss and depth constraint loss are adopted to effectively improve the structural continuity and spatial consistency of the reconstruction model, and solve the problem of three-dimensional structure distortion that may occur in traditional methods. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is the overall flowchart of a three-dimensional reconstruction method of the nipple protection area based on super-resolution provided by the first embodiment of the present invention. Detailed Embodiments

[0019] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the following will describe the specific embodiments of the present invention in detail with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative work shall fall within the scope of protection of the present invention.

[0020] Example 1, referring to Figure 1 , which is an embodiment of the present invention, provides a three-dimensional reconstruction method of the nipple protection area based on super-resolution, including: S1: Obtain the two-dimensional slice images of the nipple protection area through a medical imaging device.

[0021] Further, the two-dimensional slice images include collecting low-resolution images of the nipple protection area from multiple angles and viewpoints to obtain several groups of two-dimensional image data sequences with different viewpoints. Furthermore, preprocess the collected images, including denoising, deblurring, and illumination normalization, preliminarily calibrate the internal and external parameters of the camera, and correct the distortion; output the processed low-resolution image sequence.

[0022] Furthermore, use image denoising methods such as Gaussian filtering and median filtering to remove noise for image denoising; perform illumination normalization to balance the brightness and contrast differences between images from different viewpoints.

[0023] Furthermore, use an image correction algorithm to correct the distortion and perform a preliminary estimation of the internal and external parameters of the camera; output the preprocessed low-resolution image sequence to form a standardized data set.

[0024] It should be noted that the collected two-dimensional images are systematically preprocessed, including using methods such as Gaussian filtering and non-local means filtering for denoising to eliminate noise points caused by equipment imaging or environmental factors; and removing motion blur and sampling interference through image deblurring methods, and applying histogram equalization or illumination normalization methods to standardize the image brightness and contrast; calibrate the internal and external parameters of the acquisition device and correct the image distortion according to a known calibration board or reference object; finally, obtain the processed standardized low-resolution two-dimensional image sequence, laying a foundation for the next step of preliminary three-dimensional structure estimation.

[0025] S2: Perform a preliminary three-dimensional structure estimation on the two-dimensional slice images to obtain spatial geometric constraints.

[0026] Further, the preliminary three-dimensional structure estimation includes using the preprocessed low-resolution image sequence and adopting a structure from motion recovery algorithm to extract feature points of each viewpoint image.

[0027] Extract the feature points of each image from the preprocessed low-resolution two-dimensional image sequence. Use a feature matching algorithm to match the feature points, and use the random sample consensus algorithm to remove the mismatched points to ensure the reliability of the feature point matching.

[0028] Based on the reliable feature matching points, adopt a structure from motion recovery algorithm to estimate the preliminary three-dimensional sparse point cloud structure of the nipple protection area and obtain the camera parameters at the same time.

[0029] The formula for preliminary feature extraction is expressed as: ; Obtain reliable matching points through feature matching methods, where, represents the i-th low-resolution image, and the number of images is N. represents the SIFT feature extraction operation on the image. represents from the image the set of feature points extracted, including information such as the position, scale, direction, and feature descriptor of the key points.

[0030] Based on the reliable matching points, estimate the internal and external parameters of each perspective camera, and obtain a preliminary coarse-grained three-dimensional sparse point cloud structure.

[0031] Use the obtained coarse-grained three-dimensional structure as the spatial constraint information for the next super-resolution network. The construction formula of the preliminary three-dimensional structure SfM is expressed as: ; where, represents the preliminary three-dimensional sparse point cloud structure (SparsePointCloud), , represents the internal and external parameters of the camera corresponding to the i-th image.

[0032] It should be noted that the Structure from Motion (SfM) algorithm, based on the feature matching relationship and the initial internal and external parameter estimation method of camera calibration, gradually optimizes and calculates the camera poses and pose information corresponding to each perspective image, and constructs the initial coarse-grained three-dimensional sparse point cloud structure of the nipple protection area; finally, output the coarse-grained sparse three-dimensional structure information in the form of a spatial structure prior to guide the two-dimensional super-resolution image enhancement based on the deep learning network in the next stage, and further improve the three-dimensional reconstruction accuracy.

[0033] S3: Build a three-dimensional structure-guided super-resolution network for the two-dimensional image according to geometric constraints.

[0034] Furthermore, the building of the three-dimensional structure-guided super-resolution network includes preparing the network input, and the input data is the preliminary coarse-grained three-dimensional structure and the preprocessed multi-perspective low-resolution two-dimensional images.

[0035] For the input low-resolution image : ; where, represents the feature map extracted from the i-th low-resolution image; σ represents the non-linear activation function; W and b represent the convolutional network parameters.

[0036] Project the preliminary three-dimensional structure information onto the two-dimensional view to obtain depth-guided features , specifically as follows: ; Use a shallow convolutional network to extract basic image features from the low-resolution image sequence and output a basic two-dimensional feature map. . Use the two-dimensional convolutional operation Conv2D for local feature extraction, including convolution, activation, and feature normalization. The convolution kernel size is selected as 3×3, the stride is 1, and the shallow basic features of the image are extracted.

[0037] Introduce the preliminary three-dimensional structure information into the network and fuse the three-dimensional structure features with the two-dimensional image features through the attention mechanism.

[0038] Apply spatial constraint information to the two-dimensional features and output a structure-guided fused feature map. .

[0039] Using the fused feature map as the input, adopt the sub-pixel convolution method for efficient upsampling processing and output a two-dimensional high-quality image enhanced by super-resolution.

[0040] Fuse the two-dimensional features and the depth-guided features through the attention mechanism. The spatial attention weight formula is expressed as: ; Fused feature representation: ; where represents the Sigmoid function; represents the convolution operation; [] represents feature concatenation; represents element-wise multiplication.

[0041] First, perform global pooling on the feature channels, use the FC layer to obtain the channel attention weights, and perform channel weighted fusion on the fused features to achieve 3D guidance. Use the spatial attention module to fuse the spatial structure, obtain the spatial attention weight map using the two-dimensional convolutional layer, calculate the correlation between the two-dimensional features and the 3D structure information, and highlight the feature weights in the structure area.

[0042] The formula for realizing feature map upsampling using sub-pixel convolution is expressed as: ; Obtain the high-resolution output image: ; where pixelshuffle represents the sub-pixel rearrangement operation; represents the fused feature map of the i-th frame view.

[0043] Train a super-resolution network, and select the real high-resolution images in the nipple protection area as the supervised data for the training set.

[0044] Define a hybrid loss function, and use the Adam optimizer to iteratively optimize the parameters of the super-resolution network until convergence.

[0045] The defined hybrid loss function includes content loss, perceptual loss, structural consistency loss, and depth constraint loss.

[0046] The specific definition is as follows. The content loss measures the pixel-level difference of the images. The formula for the content loss is expressed as: ; where represents the real high-resolution image.

[0047] The perceptual loss is to supervise the semantic features of the reconstructed image. The formula for the perceptual loss is expressed as: ; where is the feature extraction function of the prediction training network (such as VGG).

[0048] The structural consistency loss constrains the continuous pixel feature space of different perspectives through the consistency of corresponding points in the three-dimensional space. Constrained by the consistency of the corresponding features after projection into the three-dimensional space, let P be the projection function from a two-dimensional point to the three-dimensional space. The formula is expressed as: ; where is the coordinate of the super-resolution result of the image projected into the three-dimensional space, represents the corresponding three-dimensional space point under another perspective, is the projection function.

[0049] The depth constraint loss provides additional constraints for network training by using the depth information of the preliminary three-dimensional structure. The constraint formula based on the depth information provided by the coarse-grained three-dimensional structure is expressed as: ; where represents the estimated depth map corresponding to the super-resolution image reconstruction process, represents the coarse-grained depth prior map. represents the square of the Euclidean distance (the square of the L2 norm), indicating the pixel-by-pixel difference between the estimated depth and the initial depth.

[0050] It should be noted that the surface reconstruction algorithm performs surface meshing on the point cloud to generate a fine three-dimensional surface mesh model of the nipple protection area; further refine and optimize the generated mesh model, including mesh smoothing, redundant vertex deletion, and mesh hole filling, to improve the continuity and smoothness of the three-dimensional model; finally, using the enhanced high-resolution two-dimensional image sequence, map the fine texture information to the three-dimensional mesh surface through the multi-view texture mapping method to generate a fine three-dimensional reconstruction model of the nipple protection area with accurate spatial structure and realistic texture, and output in a standard format for medical analysis or auxiliary diagnosis.

[0051] S4: Generate a dense and high-precision point cloud through multi-view stereo matching for 3D reconstruction.

[0052] Furthermore, the generation of the dense and high-precision point cloud includes performing multi-view stereo matching using the camera parameters generated in the preliminary 3D structure estimation and the high-resolution image enhanced by the super-resolution network; Utilize the disparity estimation algorithm to calculate the disparity, generate the depth map, and fuse the multi-view depth information to obtain a high-density three-dimensional point cloud of the nipple protection area; Denoise, filter, and optimize the obtained high-density point cloud to remove clutter and isolated points and improve the clarity and accuracy of the point cloud.

[0053] The 3D reconstruction includes performing meshing on the optimized high-density point cloud using the Poisson reconstruction algorithm to generate a fine three-dimensional surface mesh model of the nipple protection area; Further optimize the mesh model, including: Mesh smoothing and removal of redundant vertices, repair of holes and irregular edges, and refinement of the model structure; Perform high-precision texture mapping optimization on the high-resolution image enhanced based on super-resolution using the multi-view texture mapping method.

[0054] It should be noted that a depth constraint loss is introduced, and the depth supervision signal is calculated through the difference between the depth map predicted by the network and the initial coarse-grained depth structure to further enhance the depth accuracy of the two-dimensional image super-resolution processing; finally, through iterative training with the Adam optimizer, the network weights are optimized with the mixed loss function, so that the final output result of the network simultaneously has high-precision spatial consistency, accurate depth contours, and good perceptual quality, effectively improving the detail accuracy of the three-dimensional reconstruction result of the nipple protection area.

[0055] Embodiment 2 is an embodiment of the present invention, which provides a three-dimensional reconstruction method for the nipple protection area based on super-resolution. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0056] First, the nipple protection area was systematically scanned using advanced medical imaging equipment to collect low-resolution two-dimensional slice images from different angles and viewpoints. In the experiment, a combined scheme of high-resolution CT and MRI was adopted to ensure that the acquired images had a certain degree of spatial diversity and information redundancy during the acquisition process. After the acquired images were transmitted to the image processing terminal, preprocessing operations were first performed. The preprocessing process included denoising the images using Gaussian filtering and non-local mean filtering algorithms, and at the same time, using adaptive histogram equalization to normalize the illumination of the images to eliminate the brightness and contrast differences caused by different acquisition angles and equipment parameters. This step also included preliminarily calibrating the internal and external parameters of the camera using a preset calibration board, and then correcting the image distortion, thereby outputting the processed low-resolution image sequence.

[0057] Next, the structure-from-motion (SfM) algorithm was used to perform a preliminary three-dimensional structure estimation on the preprocessed image sequence. Specifically, key feature points were extracted from the images of each perspective using the SIFT algorithm, and the RANSAC method was used to eliminate the mismatched points. Based on the results of multi-view feature matching, the internal and external parameter information of the cameras for each perspective was estimated, and a preliminary coarse-grained three-dimensional sparse point cloud structure was reconstructed. This coarse-grained point cloud not only retained the main spatial distribution characteristics of the nipple protection area but also provided spatial constraint information for subsequent super-resolution processing.

[0058] After obtaining the preliminary three-dimensional structure, it was jointly input into a three-dimensional structure-guided super-resolution network together with the low-resolution image sequence. The network first used a shallow convolutional network to extract the basic features of each low-resolution image to form a basic two-dimensional feature map. At the same time, the preliminary three-dimensional structure was projected onto the two-dimensional image plane to generate a corresponding depth-guided feature map. An attention mechanism module was designed to fuse the depth-guided features with the two-dimensional image features, and different weights were assigned to the features at different positions through a spatial attention mechanism, thereby achieving efficient focusing on the key structure areas in the image and outputting a structure-guided fused feature map.

[0059] Subsequently, the sub-pixel convolution (Sub-pixel Convolution) technology was used to upsample the fused feature map to generate a high-quality two-dimensional image enhanced by super-resolution. During the network training stage, the true high-resolution image of the nipple protection area was used as the supervised data, and a hybrid loss function including content loss, perceptual loss, structural consistency loss, and depth constraint loss was adopted. The network parameters were iteratively optimized using the Adam optimizer until the loss converged.

[0060] After the super-resolution network is trained, the disparity is calculated by the multi-view stereo matching (MVS) method using the high-resolution images output by the network and the camera parameters obtained from the preliminary 3D structure estimation. In the experiment, the PatchMatch algorithm is used to generate depth maps corresponding to each view, and the multi-view depth information is fused to form a high-density 3D point cloud within the nipple protection area. Thereafter, statistical filtering and radius filtering are used to denoise the generated point cloud data, removing isolated points and noise points to ensure the high precision and high continuity of the point cloud data. Finally, the Poisson reconstruction algorithm is used to mesh the point cloud data to generate a fine 3D surface mesh model of the nipple protection area. After mesh reconstruction, the model is smoothed, redundant vertices are removed, and holes are repaired to ensure the continuous structure and rich details of the model. Finally, based on the multi-view texture mapping method, the high-resolution image texture enhanced by super-resolution is mapped onto the surface of the 3D model, thereby obtaining a 3D reconstruction model of the nipple protection area with realistic texture and accurate structure.

[0061] The 3D reconstruction method of the nipple protection area based on super-resolution shows significant advantages in all key indicators. First of all, the image acquisition resolution remains between 150 dpi and 170 dpi for all test subjects, demonstrating the performance stability of the acquisition device. In terms of image preprocessing, the noise reduction rate for all test subjects reaches over 85%, up to 89% at most, indicating that the preprocessing algorithm can effectively eliminate noise and blur effects in the images, ensuring the high quality of the basic data for subsequent 3D reconstruction. This is particularly important for refined 3D reconstruction because the improvement of the original image quality is directly related to the accuracy of subsequent structure estimation and super-resolution reconstruction.

[0062] In terms of the error of the preliminary 3D reconstruction, the error for all test subjects is less than 0.75 mm, and the minimum error is 0.65 mm. This shows that by using the structure-from-motion recovery algorithm and feature matching strategy, when extracting the preliminary 3D structure from low-resolution images, this method can achieve high precision while ensuring speed, thus providing a reliable spatial constraint prior for the subsequent super-resolution network. The super-resolution magnification factor is 3.0 times for all test subjects, which means that after being processed by the super-resolution network, the resolution of the 2D image is significantly improved, resulting in a significant improvement in the presentation of image details.

[0063] The point cloud density, as an important parameter for measuring the quality of 3D reconstruction, shows relatively high values in all test objects, ranging from 2.10 points / mm³ to 2.50 points / mm³, indicating that in the multi-view stereo matching and disparity fusion steps of this method, it is capable of generating high-density and high-precision 3D point clouds. This high point cloud density is conducive to subsequent mesh reconstruction and texture mapping, enabling the final generated 3D model to have a higher ability to restore details. At the same time, the texture mapping accuracy remains above 92% in all test objects, reaching a maximum of 95%, which fully demonstrates that the high-resolution images generated after super-resolution enhancement can accurately reflect the actual tissue texture and details during the texture mapping process, enhancing the realism and diagnostic value of the model.

[0064] Embodiment 3 is an embodiment of the present invention, which provides a 3D reconstruction system for the nipple protection area based on super-resolution, including a data acquisition module, a preliminary estimation module, a 3D-guided super-resolution processing module, and a 3D reconstruction module.

[0065] The data acquisition module includes a data acquisition unit and an image processing unit; it is used to collect multi-angle low-resolution two-dimensional images and depth scan data around the nipple protection area, and complete the preliminary processing of the nipple protection area data.

[0066] The preliminary estimation module includes a feature extraction unit, a parameter estimation unit, and an initial structure generation unit; it is used to estimate the preliminary 3D structure and output coarse-grained 3D spatial prior information.

[0067] The 3D-guided super-resolution processing module is used to perform super-resolution processing on the two-dimensional images of the nipple protection area by using the 3D structure information as a guide.

[0068] The 3D reconstruction module includes a reconstruction unit and a texture mapping unit, which are used to achieve high-precision 3D reconstruction of the nipple protection area based on the super-resolution image and output a refined 3D model.

[0069] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0070] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, apparatus, or device.

[0071] More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection parts with one or more wirings (electronic devices), portable computer disk cartridges (magnetic devices), random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memories), fiber optic devices, and portable compact disc read-only memories (CDROM). Additionally, a computer-readable medium can even be paper or other suitable media on which a program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting, or otherwise processing it as appropriate, and then storing it in a computer memory.

[0072] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc. It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

[0073] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A three-dimensional reconstruction method for the nipple protection area based on super-resolution, characterized in that, Including: Obtaining two-dimensional slice images of the nipple protection area through a medical imaging device; Performing preliminary three-dimensional structure estimation on the two-dimensional slice images to obtain spatial constraints; Constructing a three-dimensional structure-guided super-resolution network for the two-dimensional images according to the spatial constraints; Performing three-dimensional reconstruction by generating a dense and high-precision point cloud through multi-view stereo matching.

2. The three-dimensional reconstruction method of the nipple protection area based on super-resolution according to claim 1, characterized in that: The two-dimensional slice images include collecting low-resolution images of the nipple protection area from multiple angles and viewpoints to obtain several groups of two-dimensional image data sequences with different viewpoints; Preprocessing the collected images, including denoising, deblurring, and illumination normalization, preliminarily calibrating the internal and external parameters of the camera, and correcting distortion; Outputting the processed low-resolution image sequence.

3. The three-dimensional reconstruction method of the nipple protection area based on super-resolution according to claim 2, characterized in that: The performing of preliminary three-dimensional structure estimation includes using the preprocessed low-resolution image sequence, adopting a structure from motion recovery algorithm, and extracting feature points of each-view image; Obtaining reliable matching points through a feature matching method; Based on the reliable matching points, estimating the internal and external parameters of each-view camera and obtaining a preliminary coarse-grained three-dimensional sparse point cloud structure; Taking the initial coarse-grained three-dimensional structure as the spatial constraint information for super-resolution processing.

4. The three-dimensional reconstruction method of the nipple protection area based on super-resolution according to claim 3, characterized in that: The constructing of a three-dimensional structure-guided super-resolution network includes: Preparing network inputs, where the input data is the preliminary coarse-grained three-dimensional structure and the preprocessed multi-view low-resolution two-dimensional images; Extract basic image features from a low-resolution image sequence using a shallow convolutional network and output a basic two-dimensional feature map ; Introducing the preliminary three-dimensional structure information into the network and fusing the three-dimensional structure features and two-dimensional image features through an attention mechanism; Apply spatial constraint information to two-dimensional features and output a structure-guided fused feature map ; Using the fused feature map as the input, perform upsampling processing using the sub-pixel convolution method to output a two-dimensional high-quality image after super-resolution enhancement; Training the super-resolution network, and selecting the real high-resolution images of the nipple protection area as the supervised data for the training set; Defining a hybrid loss function, and using an Adam optimizer to iteratively optimize the parameters of the super-resolution network until convergence.

5. The three-dimensional reconstruction method of the nipple protection area based on super-resolution according to claim 4, wherein: The defining of the hybrid loss function includes content loss, perceptual loss, structure consistency loss, and depth constraint loss; Specifically defined including that the content loss measures the pixel-level difference of images; the perceptual loss supervises the semantic features of the reconstructed images; the structure consistency loss constrains the continuity of pixel feature spaces of different viewpoints through the consistency of three-dimensional space corresponding points; the depth constraint loss provides additional constraints for network training using the depth information of the preliminary three-dimensional structure.

6. The three-dimensional reconstruction method of the nipple protection area based on super-resolution according to claim 5, characterized in that: The generating of a dense and high-precision point cloud includes performing multi-view stereo matching using the camera parameters generated in the preliminary three-dimensional structure estimation and the high-resolution images enhanced by the super-resolution network; Using a disparity estimation algorithm to implement disparity calculation, depth map generation, and fusing multi-view depth information to obtain a high-density three-dimensional point cloud of the nipple protection area; Performing denoising, filtering, and optimization on the obtained high-density point cloud to remove clutter and isolated points and improve the clarity and precision of the point cloud.

7. The three-dimensional reconstruction method of the nipple protection area based on super-resolution according to claim 6, characterized in that: The performing of three-dimensional reconstruction includes performing meshing processing on the optimized high-density point cloud using a Poisson reconstruction algorithm to generate a fine three-dimensional surface mesh model of the nipple protection area; Further optimizing the mesh model, including: Mesh smoothing and removing redundant vertices, repairing holes and irregular edges, and streamlining the model structure; Performing high-precision texture mapping optimization on the high-resolution images enhanced based on super-resolution using a multi-view texture mapping method.

8. A system adopting the three-dimensional reconstruction method of the nipple protection area based on super-resolution as described in any one of claims 1 to 7, characterized in that: It includes a data acquisition module, a preliminary estimation module, a 3D-guided super-resolution processing module, and a three-dimensional reconstruction module; The data acquisition module includes a data acquisition unit and an image processing unit; It is used to acquire multi-angle low-resolution two-dimensional images and depth scan data around the nipple protection area and complete the preliminary processing of the nipple protection area data; The preliminary estimation module includes a feature extraction unit, a parameter estimation unit, and an initial structure generation unit; It is used to estimate the preliminary three-dimensional structure and output the coarse-grained three-dimensional spatial prior information; The 3D-guided super-resolution processing module is used to perform super-resolution processing on the two-dimensional images of the nipple protection area by using the three-dimensional structure information as a guide; The three-dimensional reconstruction module includes a reconstruction unit and a texture mapping unit, and is used to achieve high-precision three-dimensional reconstruction of the nipple protection area based on the super-resolution image and output a refined three-dimensional model.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the super-resolution-based three-dimensional reconstruction method of the nipple protection area described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the super-resolution-based three-dimensional reconstruction method of the nipple protection area described in any one of claims 1 to 7.

Images