Throat structure and detail high-resolution image enhancement method based on deep learning

Abstract

The invention relates to the technical field of image enhancement, in particular to a throat structure and detail high-resolution image enhancement method based on deep learning. The method comprises the following steps: acquiring a low-resolution image of a laryngeal endoscope and corresponding laryngeal three-dimensional image data; the method comprises the following steps: extracting features from a low-resolution image based on a ResNet model, extracting features from three-dimensional image data by using a 3D-ResNet model, constructing a time-varying weight matrix modulated by a sine period in the 3D-ResNet model, so that the physiological motion state of the throat can be perceived in real time in an image enhancement process, and fusing the extracted features by using a cross-scale feature pyramid technology; and a U-Net + + model combined with a sub-pixel convolutional layer is utilized to carry out super-resolution reconstruction on the image subjected to feature fusion processing. According to the design, by introducing an anatomical attention module and a dynamic receptive field adjustment technology, the characteristics of clinical key areas (such as vocal cord free edges and anterior joint areas) can be enhanced in a targeted manner.

Description

Technical Field

[0001] The present invention relates to the field of image enhancement technology, and in particular to a method for enhancing high-resolution images of laryngeal structure and details based on deep learning. Background Art

[0002] Traditional models have difficulty effectively capturing tiny but critical features, such as microvessels in areas with a high incidence of cancer. In addition, because the larynx is a dynamic organ, the significant displacement that occurs during breathing or vocalization causes motion artifacts in CT images, seriously affecting the accuracy of clinical assessments such as the depth of tumor infiltration. At the same time, the isotropic processing of standard 3D convolution kernels causes feature loss or blurring under specific viewing angles. Conventional three-dimensional feature extraction ignores the stereoscopic relationship between the laryngeal ventricle, vocal cords, and trachea, and cannot support the effective assessment of dynamic functional disorders such as vocal cord paralysis. In terms of super-resolution reconstruction, existing technologies have insufficient performance of high-definition image details, especially edges and key structures, and are limited in the clarity and contrast of details in the mucosal area, all of which limit the accuracy and reliability of diagnosis. Therefore, a high-resolution image enhancement method for laryngeal structure and details based on deep learning is provided. Summary of the Invention

[0003] The purpose of the present invention is to provide a high-resolution image enhancement method for laryngeal structure and details based on deep learning, so as to solve the problems of traditional laryngeal image processing technology proposed in the above background technology in terms of feature capture, motion artifacts, loss of stereo correlation information and super-resolution reconstruction accuracy.

[0004] To achieve the above objectives, the present invention provides a method for enhancing high-resolution images of laryngeal structure and details based on deep learning, comprising the following steps:

[0005] S1, collecting low-resolution images of the laryngeal endoscope and corresponding three-dimensional image data of the larynx;

[0006] S2. Extract features from low-resolution images using the ResNet model, and from 3D imaging data using the 3D-ResNet model. A sinusoidally modulated time-varying weight matrix is ​​constructed in the 3D-ResNet model, enabling the image enhancement process to perceive the physiological motion state of the larynx in real time. The extracted features are then fused using a cross-scale feature pyramid technique.

[0007] S3. Use the U-Net++ model combined with sub-pixel convolutional layers to perform super-resolution reconstruction on the image that has been processed by feature fusion;

[0008] S4. The deep learning-based segmentation model U-Net performs targeted local enhancement on the mucosal area in the image.

[0009] As a further improvement of the present technical solution, in S2, extracting features from the low-resolution image based on the ResNet model includes the following steps:

[0010] S2.1. Automatically identify and label key anatomical landmarks of the larynx using a pre-trained laryngeal keypoint detection model, and generate a structural heatmap based on these landmarks.

[0011] S2.2, adaptive gamma correction for clinical areas;

[0012] S2.3. Input the image processed in step S2.2 into the improved ResNet model with the anatomical attention module to start the feature extraction process.

[0013] S2.4. Perform the first stage feature extraction of ResNet to obtain a preliminary feature map;

[0014] S2.5, compare the feature map of the current stage with the structural thermal map generated in step S2.1 Figure 1 The two are input to the anatomical attention module, which combines the feature information of the two through convolution operation, generates attention weights, and applies the attention weights to weight the feature maps;

[0015] S2.6. Dynamically adjust the convolution kernel parameters according to the spatial frequency of the current feature map;

[0016] S2.7. Repeat the above steps to obtain multi-level feature output.

[0017] As a further improvement of the present technical solution, in S2, extracting features from the three-dimensional image data based on the 3D-ResNet model includes the following steps:

[0018] S2.8. Collect laryngeal motion sensor data;

[0019] S2.9, constructing a time-varying weight matrix based on the acquired laryngeal motion sensor data;

[0020] S2.10. Decompose the 3D convolution kernel of the 3D-ResNet model into convolution kernels targeting specific directions of the larynx. Use a gating network to assign weights to the feature maps extracted from specific directions of the larynx, and fuse the specific directions of the larynx based on the weights.

[0021] S2.11. The feature maps in different directions obtained through the above steps are sent to the dynamic gating unit for weighted fusion to generate a comprehensive feature representation.

[0022] As a further improvement of the present technical solution, in S2.9, constructing a time-varying weight matrix based on the acquired laryngeal motion sensor data includes the following steps:

[0023] S2.91. Extract key parameters from laryngeal motion sensor data;

[0024] S2.92. Based on the above key parameters, the state is classified and the feature threshold judgment strategy is used to classify the current moment into one of the three states: phonation phase, inspiration phase, and resting phase;

[0025] S2.93, introducing the segmented organ mask in the reference 3D image, and setting the response weight factor of the organ structure according to the physiological state of the different organ structures in each phase;

[0026] S2.94. Initialize a three-dimensional voxel matrix in a voxel space with the same resolution as the input image, and assign values ​​to the voxels of each organ structure according to the organ mask information;

[0027] S2.95. Apply a phase modulation factor to the weight of each organ voxel according to the physiological state of the current frame, and introduce a periodic modulation term in the form of a sine function;

[0028] S2.96, based on the vertical displacement and impedance change, reconstruct the 3D local displacement field at the current moment. To prevent artifacts caused by weight jumps, perform spatial smoothing on the modulated weight matrix.

[0029] S2.97, output the dynamic time-varying weight matrix after spatial smoothing.

[0030] As a further improvement of this technical solution, in S2, the extracted features are fused using a cross-scale feature pyramid technique, including the following steps:

[0031] S2.12, taking the low-resolution 2D endoscopic image and the 3D laryngeal image data obtained by feature extraction as input;

[0032] S2.13. For each scale, the feature map of the endoscopic image is fused with the feature map of the 3D image by splicing;

[0033] S2.14. Construct a feature pyramid based on a top-down path;

[0034] S2.15. Output cross-scale fusion feature pyramid.

[0035] As a further improvement of the present technical solution, in S3, a U-Net++ model combined with a sub-pixel convolution layer is used to perform super-resolution reconstruction on the image that has undergone feature fusion processing, including the following steps:

[0036] S3.1. Taking the fused feature map output by the cross-scale feature pyramid as input, the fused feature map is processed by a deep learning-based semantic segmentation network to generate a vocal cord mask.

[0037] S3.2. Build the encoder part of the U-Net++ model. In the encoder part, feature extraction is performed through three encoding blocks and a bottleneck layer in sequence.

[0038] S3.3. A vocal cord attention gate module is added after each encoding block to enhance the feature area related to the free edge of the vocal cord by fusing the vocal cord mask;

[0039] S3.4. Feed the high-dimensional representation of the encoder output into the bottleneck layer, gradually upsample it using multi-level sub-pixel convolution, and introduce a dynamic kernel strategy in the sub-pixel convolution process;

[0040] S3.5. Using the dense skip connection structure of the U-Net++ model, we connect the feature maps of each level in the encoder with the feature maps of the corresponding scale in the decoder. We also introduce a structure-aware weighting mechanism in the skip connection path of the U-Net++ model to assign weights to the skip connections based on their structural importance.

[0041] S3.6. Add an edge-guided loss term, use the Sobel operator to extract the vocal cord edge gradient, and supervise it with the L1 loss.

[0042] As a further improvement of the present technical solution, in S3.4, upsampling is performed step by step using multi-level sub-pixel convolution, and a dynamic kernel strategy is introduced in the sub-pixel convolution process, including the following steps:

[0043] S3.41. Initialize an empty list to store the upsampled feature maps output at each level. Set the output feature map of the bottleneck layer of the U-Net++ model as the initial input and set the number of upsampling times.

[0044] S3.42, use 1×1 convolution to perform channel compression on the input feature map;

[0045] S3.43. Calculate the gradient amplitude of the compressed feature map and count the proportion of high-frequency areas.

[0046] S3.44, select a convolution kernel according to the dynamic kernel with a high frequency ratio, and convolve the input feature map with the selected convolution kernel;

[0047] S3.45. Perform channel expansion convolution and upsample using the PixelShuffle operation to obtain an upsampled feature map.

[0048] S3.46. Obtain the vocal cord mask at the current resolution, and calculate the average gradient intensity of the corresponding vocal cord mask area in the upsampled feature map;

[0049] S3.47, fusing the upsampled feature map with the same-scale encoded feature map at interval one level;

[0050] S3.48. Save the feature map of the current upsampling output to an empty list and use the feature map as the input of the next upsampling stage;

[0051] S3.49. Repeat steps S3.42 to S3.48 until all upsampling layers are completed, and finally a high-resolution feature map is output.

[0052] As a further improvement to this technical solution, in S3.5, a structure-aware weighting mechanism is introduced into the skip connection path of the U-Net++ model to assign weights to the skip connections according to the structural importance, including the following steps:

[0053] S3.51. Downsample the vocal cord mask to the same spatial size as the skip connection feature map to obtain a structural importance map.

[0054] S3.52. Apply the structure importance map to the skip connection feature map and perform pixel-by-pixel channel weighting.

[0055] S3.53. Concatenate the weighted feature map with the feature map of the corresponding scale in the decoder.

[0056] As a further improvement of the present technical solution, in S4, an attention map is generated according to the anatomical structure, and targeted local enhancement is performed on the mucosal area in the image, including the following steps:

[0057] S4.1. Take the super-resolution reconstructed image as input;

[0058] S4.2. Use the laryngeal mucosal region segmentation model U-Net to process the input image and output a binary mask;

[0059] Among them, the foreground pixels represent the location of the mucosal area, and the background pixels represent the non-mucosal area;

[0060] S4.3. Apply a local enhancement network based on a dynamic filtering mechanism and adaptive weight adjustment to the segmented mucosal region;

[0061] S4.4, reintegrate the locally enhanced mucosal area into the original super-resolution reconstructed image;

[0062] S4.5. Generate the final fused image.

[0063] As a further improvement of the present technical solution, in S4.3, a local enhancement network based on a dynamic filtering mechanism and adaptive weight adjustment is applied to the segmented mucosal area, including the following steps:

[0064] S4.31. Input the segmented mucosal region image into the input layer of the local enhancement network;

[0065] S4.32, extract multi-scale features of the mucosal area through multiple convolutional layers;

[0066] S4.33. Adaptively generate filter kernel weights based on the extracted features, and introduce an attention mechanism to calculate the weight distribution of key texture and lesion features in the mucosal area;

[0067] S4.34. Dynamically adjust the weight distribution of different channels and spatial positions based on the structural information of the mucosal area;

[0068] S4.35. After filtering and weight adjustment, a locally enhanced mucosal area image is generated.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] 1. This deep learning-based method for enhancing high-resolution images of laryngeal structures and details introduces an anatomical attention module and dynamic receptive field adjustment technology. This method can specifically enhance the features of clinically critical areas (such as the free edge of the vocal cords and the anterior commissure), effectively overcoming the feature weakening and multi-scale detail loss problems of traditional ResNet models when processing complex anatomical structures. Furthermore, super-resolution reconstruction using a U-Net++ model combined with sub-pixel convolutional layers places particular emphasis on the reconstruction quality of edge regions. The inclusion of an edge-guided loss term further enhances the clarity of key anatomical structures, helping physicians more accurately assess lesions and improve diagnostic accuracy.

[0071] 2. This deep learning-based method for enhancing high-resolution images of laryngeal structure and details addresses the problem of motion artifacts in CT images caused by significant laryngeal displacement during breathing or phonation. This method constructs a time-varying weight matrix to compensate for organ displacement. It also employs a three-way separable convolution and dynamic gating network to preserve the anisotropic characteristics of the larynx, enhancing the feature representation of key viewpoints (including the vocal cord level). This method not only reduces image blur and ghosting caused by organ movement, but also maintains the three-dimensional spatial context of the larynx through multi-directional feature gating fusion, supporting the assessment of dynamic functional disorders such as vocal cord paralysis and providing more accurate and reliable imaging evidence for clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 The figure is a flow chart of the overall method of the present invention. DETAILED DESCRIPTION

[0073] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0074] Example: See Figure 1 As shown, this embodiment provides a method for enhancing high-resolution images of laryngeal structure and details based on deep learning, comprising the following steps:

[0075] S1, collecting low-resolution images of the laryngeal endoscope and corresponding three-dimensional images (CT images) of the larynx;

[0076] S2. Extract features from low-resolution images using the ResNet model, and from 3D imaging data using the 3D-ResNet model. A sinusoidally modulated time-varying weight matrix is ​​constructed in the 3D-ResNet model, enabling the image enhancement process to perceive the physiological motion state of the larynx in real time. The extracted features are then fused using a cross-scale feature pyramid technique.

[0077] In this embodiment, extracting features from a low-resolution image based on a ResNet model includes the following steps:

[0078] The ResNet model is a classic convolutional neural network structure that can efficiently extract multi-level features and achieve deep network training stability. Its input is a laryngeal image (single-channel grayscale CT / MRI image, or a three-channel fusion image), and its output is an image feature vector or feature map. The core structure of ResNet is the residual block, which uses skip connections to ensure that features are not easily lost during deep transmission. Each residual block consists of two 3×3 convolutional layers and a shortcut branch.

[0079] Traditional ResNet feature extraction suffers from two major issues: weakened anatomical features and loss of multi-scale details. The laryngeal anatomy is complex (including the free edge of the vocal cords and the anterior commissure). Traditional models treat all pixels equally, causing key features such as microvessels (<0.1 mm) in high-risk areas for cancer to be drowned out by background noise. Fixed-size convolution kernels cannot simultaneously capture both microscopic (mucosal capillaries) and macroscopic (arytenoid cartilage) structures, requiring the cascade of multiple models, increasing computational latency and compromising feature consistency. Using an anatomical attention mechanism and dynamic receptive field adjustment, targeted enhancement of clinically critical areas is achieved.

[0080] The larynx is a complex area of ​​the human body that contains multiple important anatomical structures, such as the free edge of the vocal cords and the anterior commissure. These areas are particularly important for diagnosing diseases such as cancer. However, the traditional ResNet model often cannot effectively distinguish between background noise and key features that are truly clinically significant because it treats all pixels equally. By introducing the anatomical attention module, these key areas can be automatically identified and highlighted, making the lesion features more obvious, which facilitates doctors to make accurate diagnoses; the anatomical structure of the larynx covers multiple scales from micron-level mucosal capillaries to millimeter-level arytenoid cartilages. Fixed-size convolution kernels are difficult to capture information at such a wide range of scales at the same time, which can easily lead to the loss of important details. The anatomical attention mechanism combined with dynamic receptive field adjustment technology can flexibly change the size of the convolution kernel according to the specific needs of different parts, thereby more effectively extracting and representing multi-scale features and improving the overall quality of the image;

[0081] S2.1. Use a pre-trained laryngeal key point detection model to automatically identify and label key anatomical landmarks of the larynx, including but not limited to the free edge of the vocal cords and the anterior ventricular commissure, and generate a structural heat map based on these landmarks.

[0082] S2.2. Adaptive gamma correction is performed for clinical areas (abnormal mucosa or lesions) to enhance the contrast and detail of these areas.

[0083] S2.3. Input the image processed in step S2.2 into the improved ResNet model with the anatomical attention module to start the feature extraction process.

[0084] S2.4. Perform the first stage feature extraction of ResNet to obtain a preliminary feature map;

[0085] S2.5, compare the feature map of the current stage with the structural thermal map generated in step S2.1 Figure 1 The two are input to the anatomical attention module, which combines the feature information of the two through convolution operations and generates an attention weight between 0 and 1. The attention weight is applied to the feature map to highlight the features of important areas while suppressing the information of non-critical areas.

[0086] S2.6. Dynamically adjust the convolution kernel parameters based on the spatial frequency of the current feature map to effectively capture features of different scales (for high-frequency features, a smaller convolution kernel (3×3) is used, while for low-frequency features, a dilated convolution or a larger convolution kernel (5×5) may be used).

[0087] S2.7. Repeat the above steps (feature extraction → attention weighting → dynamic receptive field adjustment) to obtain multi-level feature output, covering different levels of information from microscopic blood vessels to macroscopic cartilage structures.

[0088] Furthermore, extracting features from 3D image data based on the 3D-ResNet model includes the following steps:

[0089] The 3D-ResNet model is a three-dimensional extension of ResNet. All convolution and pooling operations are performed in the time-space three-dimensional domain, and each residual block contains two or three convolution layers.

[0090] The larynx undergoes significant displacement during breathing and phonation. Traditional 3D-ResNet treats dynamic organs as static voxels, resulting in motion artifacts in CT images (including blurred vocal cord edges and cartilage ghosting), which seriously affects the assessment of tumor invasion depth. By dynamically adjusting voxel values ​​through a phase weight matrix, the dynamic laryngoscope sensor data is synchronized with the 3D image, compensating for organ displacement during the phonation and inhalation phases and reducing motion artifacts.

[0091] Clinical CT slice thickness is much greater than the intra-slice resolution. Standard 3D convolution kernel isotropic processing results in feature loss in the coronal plane (a key perspective for assessing vocal cord symmetry) and blurred details in the sagittal plane (for observing epiglottal mobility). Three-way separable convolution is used to specifically extract anatomical plane features, and a dynamic gating network is used to automatically enhance the weights of clinically critical directions.

[0092] Conventional 3D feature extraction ignores the stereoscopic relationship between the laryngeal chamber, vocal cords, and trachea (such as the 3D rotation angle of the arytenoid cartilage), and cannot support the assessment of dynamic functional disorders such as vocal cord paralysis. Multi-directional feature gating fusion preserves the 3D spatial context of the larynx.

[0093] S2.8. Collect laryngeal motion sensor data (via laryngoscope or accelerometer) that can reflect changes in laryngeal position under different conditions.

[0094] S2.9. Build a time-varying weight matrix based on acquired laryngeal motion sensor data (this matrix adjusts CT values ​​according to the three phases of inspiration, phonation, and rest). Dynamically adjust the CT values ​​in the 3D image to compensate for organ displacement caused by breathing and phonation. For image data with slice thickness greater than 1 mm, use depth-aware interpolation along the Z axis to improve the spatial resolution of the image.

[0095] The process of constructing a time-varying weight matrix based on the acquired laryngeal motion sensor data includes the following steps:

[0096] The internal structure of the larynx will undergo significant deformation under different physiological states such as phonation, inhalation and rest. In particular, the position and morphology of tissues such as the vocal cords and epiglottis will change dynamically in a short period of time. If these changes are ignored and only static images or fixed attention areas are used for image enhancement, problems such as structural mismatch, blurring and edge artifacts will occur, seriously affecting the clarity and diagnostic value of key anatomical areas. In addition, it is difficult to maintain temporal consistency during the image enhancement process, which can easily cause frame jumps and make the enhancement effect unstable. The introduction of a time-varying weight matrix constructed based on laryngeal motion sensor data enables the image enhancement process to perceive the physiological motion state of the larynx in real time and dynamically adjust the weight response of different anatomical regions, thereby accurately focusing on the detailed reconstruction of key structures (the free edge of the vocal cords). At the same time, by introducing smoothing mechanisms such as sinusoidal periodic modulation and spatial interpolation, inter-frame noise and weight jumps are effectively suppressed, the structural consistency and temporal stability of the enhanced image are improved, and the accuracy and clinical adaptability of the overall super-resolution reconstruction are significantly improved.

[0097] S2.91. Extract key parameters from laryngeal motion sensor data, including the glottal impedance waveform (reflecting the phonation state) , vertical displacement of the throat , air pressure change rate ;

[0098] The sensor data stream is: ;

[0099] S2.92. Based on the above key parameters, the state is classified and the feature threshold judgment strategy is used to classify the current moment into one of the three states: phonation phase, inspiration phase, and resting phase;

[0100] The feature threshold judgment strategy is:

[0101] ;

[0102] Indicates the current time Physiological state category; Indicates the glottal impedance threshold (determining whether to speak); Indicates the upper limit of the vertical displacement of the throat, in mm; Indicates a threshold value of the air pressure change rate, in mm; Indicates the lower limit of the vertical displacement of the throat, in mm;

[0103] S2.93. Introduce the segmented organ masks (including glottis, laryngeal ventricle, vocal cords, tracheal cartilage, etc.) in the reference 3D image and set the response weight factors of the organ structures according to their physiological states in each time phase.

[0104] S2.94. Initialize a 3D voxel matrix with all 1s in the voxel space with the same resolution as the input image. , ( is the height of the image, is the width of the image, is the depth of the image), and the voxels of each organ structure are assigned values ​​according to the organ mask information: , represents the organ mask function (the organ number corresponding to the voxel position), Represents the three-dimensional coordinates in the image data, that is, the position index in the width, height, and depth directions. Represents a set of static weight values ​​defined based on different organ structure types;

[0105] S2.95, according to the physiological state of the current frame, apply a phase modulation factor to the weight of each organ voxel (i.e., the magnification / reduction coefficient of the weight in different physiological stages), and introduces a periodic modulation term in the form of a sine function (i.e., dynamically adjust the time function of the weight) to more realistically simulate the effects of respiratory rhythm;

[0106] Among them, the periodic modulation term in the form of a sine function for:

[0107] ;

[0108] Where, represents the average respiratory rate, Indicates the amplitude adjustment factor (default 0.1~0.3);

[0109] S2.96. Reconstruct the current 3D local displacement field based on vertical displacement and impedance changes. (This describes the direction and magnitude of motion of each voxel. This is to model the spatial deformation of the laryngeal anatomy during real physiological motion into the image enhancement process, ensuring temporal consistency, structural continuity, and functional responsiveness of the enhanced results.) To prevent artifacts from weight jumps, spatially smooth the modulated weight matrix, applying bilinear interpolation transitions in particular to the organ boundary layer (2 mm thickness).

[0110] Among them, reconstructing the 3D local displacement field at the current moment for:

[0111] ;

[0112] Where, is the organ displacement coefficient (range 0.8~1.2), is the impedance-displacement coupling coefficient (range 0.05~0.2), Indicates the rate of change of glottal impedance, which is calculated through the time series data of the laryngeal motion sensor and reflects the instantaneous state of glottal movement. is the glottal impedance over time The function of the change of (the instantaneous pressure difference on both sides of the glottis divided by the instantaneous air volume passing through the glottis); Indicates the displacement along the patient's left-right axis (normal range is between, When the vocal cords are fixed; laryngeal spasm); Indicates the displacement along the patient's anteroposterior axis (normal range is between, When unilateral vocal cord paralysis occurs; when the tumor compresses); Indicates displacement along the patient's superior-inferior axis (normal range is between, When the arytenoid cartilage is dislocated; crocaloarytenoid arthritis);

[0113] The formula for spatial smoothing of the modulated weight matrix is:

[0114] ;

[0115] ;

[0116] ;

[0117] Where, is the final dynamic weight after smoothing, is the distance to the organ boundary, represents the mixing coefficient, which is , varies linearly with distance; is the weight value obtained by combining dynamic weights with directional adjustments based on the displacement field and its gradient, Represents a dynamic weight value that combines organ type, phase, and respiratory rhythm. is the thickness of the transition zone at the organ boundary, is the adjustment coefficient of the displacement field on the weight change (dimensionless), which controls the influence of the anatomical structure displacement on the weight gradient;

[0118] S2.97, output smoothed dynamic time-varying weight matrix ,

[0119] , where Indicates the height direction (number of rows); Indicates the width direction (number of columns); Indicates the depth direction (number of slices).

[0120] In this embodiment, the dynamic time-varying weight matrix output by S2.97 is Corresponding 3D laryngeal image data Perform voxel-by-voxel multiplication to obtain the corrected three-dimensional image data ,in Denotes the Hadamard product, and the corrected Used as input to the 3D-ResNet model for subsequent feature extraction;

[0121] S2.10. Decompose the 3D convolution kernel of the 3D-ResNet model into convolution kernels for specific laryngeal directions, including axial, coronal, and sagittal convolutions, to reduce computational complexity while preserving the anisotropic characteristics of the laryngeal region. Use a learnable gating network to assign weights to feature maps extracted from specific laryngeal directions and fuse them based on the weights. This process enhances feature representation at key viewpoints (including the vocal cord level).

[0122] S2.11. The feature maps in different directions obtained through the above steps are sent to the dynamic gating unit for weighted fusion to generate a comprehensive feature representation.

[0123] Furthermore, the extracted features are fused using the cross-scale feature pyramid technology, which includes the following steps:

[0124] S2.12. Using the low-resolution 2D endoscopic image obtained through feature extraction and the 3D laryngeal image data as input, extract a 2D feature map at a specific viewpoint from the 3D image and resize it to match the size of the 2D endoscopic image. This step ensures that the images from the two different sources can be compared and fused at the same spatial resolution.

[0125] S2.13. For each scale, the feature map of the adjusted endoscopic image is fused with the feature map of the 3D image by splicing, and the fused features of the three scales are obtained (denoted as F1, F2 and F3);

[0126] S2.14. Build a feature pyramid using a top-down approach (starting with the smallest-scale feature map F3, upsampling (bilinear interpolation) to obtain a feature map, then adding it to F2, upsampling the added feature map and adding it to F1). Ensure that the two feature maps match in size after each upsampling and before adding them to the larger-scale feature map. If necessary, further resize the feature maps to ensure accurate pixel-level alignment.

[0127] S2.15. Output the cross-scale fusion feature pyramid, which contains information of different scales from fine-grained to coarse-grained.

[0128] S3. Use the U-Net++ model combined with sub-pixel convolutional layers to perform super-resolution reconstruction on the image that has been processed by feature fusion;

[0129] In this embodiment, U-Net++ is an enhanced version of the U-Net model that improves semantic segmentation accuracy through dense skip connections and multi-scale feature fusion. When processing images, U-Net is first used for preliminary semantic segmentation, generating features such as vocal cord masks, which are used to locate and identify key areas in the image. Traditional upsampling methods (including bilinear interpolation or transposed convolution) can cause blurring or checkerboard effects. Sub-pixel convolution (also known as Pixel Shuffle) is a more effective upsampling technique that intelligently rearranges the channel data of a low-resolution image to produce a high-resolution output, preserving more detail. This method can restore high-definition image detail while reducing computational cost. In the U-Net++ model, a dynamic kernel strategy is introduced during the upsampling process, adaptively adjusting the convolution kernel weights based on the current local structure. This means that for areas with more high-frequency information (such as edges and textures), convolution kernels with smaller receptive fields can be selected to capture details. For areas with smoother structures, dilated convolutions may be used to cover a larger area. This strategy helps enhance the ability to express details in sub-pixel reconstruction.

[0130] Using the U-Net++ model combined with sub-pixel convolutional layers, super-resolution reconstruction is performed on the image that has undergone feature fusion processing. The following steps are included:

[0131] S3.1. The fused feature map output by the cross-scale feature pyramid is used as input and processed by a deep learning-based semantic segmentation network (U-Net) to generate a vocal cord mask (the backbone network (ResNet-34) is pre-trained on ImageNet, and the segmentation head is fine-tuned on the laryngoscope dataset (Kvasir-SEG)). This mask is used to locate the area related to the free edge of the vocal cords in the image.

[0132] S3.2. Construct the encoder portion of the U-Net++ model. In this portion, feature extraction is performed sequentially through three encoding blocks and a bottleneck layer. The encoder consists of three encoding blocks and a bottleneck layer (with 64, 128, and 256 channels, respectively, and 512 output channels). Each encoding block includes two layers of convolution and a ReLU activation function to gradually extract the structural features of the image and perform spatial downsampling to capture feature information at different levels.

[0133] S3.3. A vocal cord attention gate module is added after each encoding block to enhance the feature regions related to the free edge of the vocal cords by fusing the vocal cord masks. (By fusing the pre-generated vocal cord masks to emphasize the feature regions related to the free edge of the vocal cords, these regions that are critical for diagnosis are made more prominent. This method helps overcome the problem of traditional convolutional neural networks treating all pixels equally when processing complex anatomical structures, ensuring that key details are not drowned out by background noise. This module can specifically increase the weights of important regions such as the vocal cords during network training, making the information of these regions more easily captured by subsequent layers and more effectively propagating gradient information backward, helping the model to better learn the feature representations of these regions.) This improves the weights of key regions and their gradient propagation capabilities, helping to improve reconstruction quality, especially the clarity of important anatomical structures.

[0134] S3.4. The high-dimensional representation output by the encoder is fed into the bottleneck layer, where the output dimension is (20 × 15 × 2048). Multi-level sub-pixel convolution is used for progressive upsampling (to restore high-definition image details). A dynamic kernel strategy is introduced during the sub-pixel convolution process to adaptively generate convolution kernel weights based on the current local structure to enhance the ability to express details during sub-pixel reconstruction.

[0135] Among them, multi-level sub-pixel convolution is used to gradually perform upsampling (for restoring high-definition image details), and a dynamic kernel strategy is introduced in the sub-pixel convolution process, including the following steps:

[0136] S3.41. Initialize an empty list to store the upsampled feature maps output at each level. Set the output feature map of the bottleneck layer of the U-Net++ model as the initial input and set the number of upsampling times (3).

[0137] S3.42. Use 1×1 convolution to perform channel compression on the input feature map to reduce the amount of computation and standardize the feature distribution;

[0138] S3.43. Calculate the gradient magnitude of the compressed feature map (implemented using the Sobel operator) and count the proportion of high-frequency regions.

[0139] S3.44. Select a convolution kernel based on the dynamic kernel of the high-frequency ratio (if the high-frequency ratio is greater than a (a represents the ratio threshold of the high-frequency area, ranging from 0.2 to 0.4), it indicates that the high-frequency information is rich, and a convolution kernel with a smaller receptive field is selected; if the high-frequency ratio is less than or equal to a, it indicates a smooth structure, and a dilated convolution is selected). Then use the selected convolution kernel to convolve the input feature map;

[0140] S3.45. Perform channel-dilated convolution and upsample using the PixelShuffle operation (the PixelShuffle operation achieves efficient image upsampling by rearranging the channel dimensions in the feature map into spatial dimensions) to obtain the upsampled feature map.

[0141] S3.46. Obtain the vocal cord mask at the current resolution and calculate the average gradient strength of the corresponding vocal cord mask area in the upsampled feature map (by extracting the gradient amplitude of the area corresponding to the vocal cord mask in the upsampled feature map and averaging its pixel gradients to measure the detail strength of the area). This is used for structural weighting of subsequent skip connections.

[0142] S3.47. Fuse the upsampled feature map with the same-scale encoded feature map one level apart (e.g., the previous layer) (first adjust the number of channels and spatial size of the same-scale encoded feature map, calculate the structural weighting factor (calculate the attention based on the comparison of the average gradient strength and the full-image gradient), and then perform weighted fusion).

[0143] S3.48. Save the feature map of the current upsampling output to an empty list and use the feature map as the input of the next upsampling stage;

[0144] S3.49. Repeat steps S3.42 to S3.48 until all upsampling layers (N=3) are completed, and finally a high-resolution feature map is output.

[0145] S3.5. Using the dense skip connection structure of the U-Net++ model (a dense skip connection structure is a mechanism designed to enhance information flow and feature reuse. Through dense skip connections, U-Net++ can better combine information from different levels, including both detailed low-level features and abstract high-level features, which is very important for accurately segmenting complex and delicate anatomical structures), each level of the feature map in the encoder is converted to ( Indicates the network level, The CNN connects the densely connected features (denoting the depth of dense connections) with the feature maps of the corresponding scale in the decoder to achieve the fusion of semantic and spatial information. This allows for the control of redundant information transfer and the focus on key structures. Furthermore, a structure-aware weighting mechanism is introduced into the skip connection path of the U-Net++ model (a technique used in deep learning models to optimize feature transfer and enhance the recognition of specific anatomical structures). This mechanism assigns weights to skip connections based on structural importance (whether they belong to the vocal cords) to achieve more targeted reconstruction adjustments.

[0146] Furthermore, a structure-aware weighting mechanism is introduced into the skip connection path of the U-Net++ model to assign weights to skip connections based on their structural importance (whether they belong to the vocal cords). The following steps are included:

[0147] S3.51. Downsample the vocal cord mask to the skip connection feature map The same spatial size, get the structural importance map , each position value of which reflects whether the position is in the vocal cord area or adjacent to the boundary;

[0148] Structural Importance Diagram Expressed as:

[0149] ;

[0150] Where, ; Represents the spatial coordinate position in the image; It means that the vocal cord area is fully present; represents the weight of the non-vocal cord area;

[0151] S3.52, the structural importance diagram Applied to skip connection feature map , perform pixel-by-pixel channel weighting, that is, retain the feature intensity of the vocal cord area and suppress the redundant feature transfer of non-critical areas;

[0152] The pixel-by-pixel channel weighting is as follows:

[0153] ;

[0154] Where, is the weighted feature map, Represents the channel of the feature map;

[0155] S3.53, the weighted feature map It is spliced ​​with the feature map of the corresponding scale in the decoder, so that the decoder can focus more on the restoration of key structural areas when restoring image details.

[0156] S3.6. In order to particularly emphasize and improve the reconstruction quality of edge areas, an edge-guided loss term is added, the Sobel operator is used to extract the vocal cord edge gradient, and L1 loss is used for supervision (the Sobel operator is used to calculate the gradient map of the predicted image and the true label image respectively, the two gradient maps are differenced pixel by pixel, and the average value of the difference is calculated using the L1 loss function. The loss is added to the total loss as an edge-guided term to optimize the model's edge reconstruction ability).

[0157] S4, based on the deep learning segmentation model U-Net, performs targeted local enhancement on the mucosal area in the image;

[0158] In this embodiment, an attention map is generated based on the anatomical structure, and targeted local enhancement is performed on the mucosal area in the image, including the following steps:

[0159] S4.1. Take the super-resolution reconstructed image as input;

[0160] S4.2. Use the laryngeal mucosal region segmentation model U-Net to process the input image and output a binary mask;

[0161] Among them, the foreground pixels represent the location of the mucosal area, and the background pixels represent the non-mucosal area;

[0162] S4.3. Apply a local enhancement network based on a dynamic filtering mechanism and adaptive weight adjustment to the segmented mucosal region;

[0163] Furthermore, the dynamic filtering mechanism can automatically adjust the filter parameters according to the specific characteristics of different mucosal regions to achieve the best image enhancement effect. This mechanism can effectively remove noise interference while retaining or even enhancing important tissue structural information. The mechanism also allows different filtering strategies to be adopted for different types of mucosa (such as inflammation, ulcers, etc.), thereby improving the visualization of specific lesion features. The adaptive weight adjustment technology can automatically assign different weight values ​​according to the different characteristics within the mucosal region during the processing process, ensuring that key areas receive more attention. This technology can enhance sensitivity to subtle structural changes, including subtle morphological changes in early cancer, making these changes easier to detect and further improving diagnostic efficiency.

[0164] The segmented mucosal area is subjected to a local enhancement network based on a dynamic filtering mechanism and adaptive weight adjustment, including the following steps:

[0165] S4.31. Input the segmented mucosal region image into the input layer of the local enhancement network as the high-interest region to be processed;

[0166] S4.32. Multi-scale features of the mucosal region are extracted through multiple convolutional layers, capturing the texture features, edge information, and fine structure of the mucosal region, ensuring multi-level expression of detail information.

[0167] S4.33. Adaptively generate filter kernel weights based on the extracted features. The filter kernel dynamically adjusts to the local structure, enhancing details rather than simply smoothing. An attention mechanism is introduced to calculate the weight distribution of key texture and lesion features in the mucosal area, strengthening attention to important areas and improving the response strength of these areas.

[0168] S4.34. Combining the structural information of the mucosal area (including edge strength and texture complexity), dynamically adjust the weight distribution of different channels and spatial positions to achieve differentiated processing of different positions and channels, and highlight important details;

[0169] S4.35, after filtering and weight adjustment, generating a locally enhanced mucosal region image;

[0170] S4.4, reintegrate the locally enhanced mucosal area into the original super-resolution reconstructed image;

[0171] S4.5. Generate the final fused image.

[0172] The basic principles, main features, and advantages of the present invention are shown and described above. It should be understood by those skilled in the art that the present invention is not limited to the above-described embodiments. The above-described embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and modifications fall within the scope of the invention claimed.

Claims

1. A deep learning-based high-resolution image enhancement method for laryngeal structure and details, characterized by: The following steps are involved: S1, collecting low-resolution images of the laryngeal endoscope and corresponding three-dimensional image data of the larynx; S2. Extract features from low-resolution images using the ResNet model, and from 3D imaging data using the 3D-ResNet model. A sinusoidally modulated time-varying weight matrix is ​​constructed in the 3D-ResNet model, enabling the image enhancement process to perceive the physiological motion state of the larynx in real time. The extracted features are then fused using a cross-scale feature pyramid technique. S3. Use the U-Net++ model combined with sub-pixel convolutional layers to perform super-resolution reconstruction on the image that has been processed by feature fusion; S4. The deep learning-based segmentation model U-Net performs targeted local enhancement on the mucosal area in the image.

2. The method for enhancing laryngeal structure and details using deep learning according to claim 1, characterized in that: In S2, features are extracted from the low-resolution image based on the ResNet model, including the following steps: S2.

1. Automatically identify and label key anatomical landmarks of the larynx using a pre-trained laryngeal keypoint detection model, and generate a structural heatmap based on these landmarks. S2.2, adaptive gamma correction for clinical areas; S2.

3. Input the image processed in step S2.2 into the improved ResNet model with the anatomical attention module to start the feature extraction process. S2.

4. Perform the first stage feature extraction of ResNet to obtain a preliminary feature map; S2.

5. Input the feature map of the current stage and the structural heat map generated in step S2.1 into the anatomical attention module, combine the feature information of the two through convolution operation, generate attention weights, and apply the attention weights to weight the feature map; S2.

6. Dynamically adjust the convolution kernel parameters according to the spatial frequency of the current feature map; S2.

7. Repeat the above steps to obtain multi-level feature output.

3. The method for enhancing high-resolution images of laryngeal structure and details based on deep learning according to claim 2, characterized in that: In S2, extracting features from the three-dimensional image data based on the 3D-ResNet model includes the following steps: S2.

8. Collect laryngeal motion sensor data; S2.9, constructing a time-varying weight matrix based on the acquired laryngeal motion sensor data; S2.

10. Decompose the 3D convolution kernel of the 3D-ResNet model into convolution kernels targeting specific directions of the larynx. Use a gating network to assign weights to the feature maps extracted from specific directions of the larynx, and fuse the specific directions of the larynx based on the weights. S2.

11. The feature maps in different directions obtained through the above steps are sent to the dynamic gating unit for weighted fusion to generate a comprehensive feature representation.

4. The method for enhancing high-resolution images of laryngeal structure and details based on deep learning according to claim 3, characterized in that: In S2.9, constructing a time-varying weight matrix based on the acquired laryngeal motion sensor data includes the following steps: S2.

91. Extract key parameters from laryngeal motion sensor data; S2.

92. Based on the above key parameters, the state is classified and the feature threshold judgment strategy is used to classify the current moment into one of the three states: phonation phase, inspiration phase, and resting phase; S2.93, introducing the segmented organ mask in the reference 3D image, and setting the response weight factor of the organ structure according to the physiological state of the different organ structures in each phase; S2.

94. Initialize a three-dimensional voxel matrix in a voxel space with the same resolution as the input image, and assign values ​​to the voxels of each organ structure according to the organ mask information; S2.

95. Apply a phase modulation factor to the weight of each organ voxel according to the physiological state of the current frame, and introduce a periodic modulation term in the form of a sine function; S2.96, based on the vertical displacement and impedance change, reconstruct the 3D local displacement field at the current moment. To prevent artifacts caused by weight jumps, perform spatial smoothing on the modulated weight matrix. S2.97, output the dynamic time-varying weight matrix after spatial smoothing.

5. The method for enhancing high-resolution images of laryngeal structure and details based on deep learning according to claim 3, characterized in that: In S2, the extracted features are fused using the cross-scale feature pyramid technology, including the following steps: S2.12, taking the low-resolution 2D endoscopic image and the 3D laryngeal image data obtained by feature extraction as input; S2.

13. For each scale, the feature map of the endoscopic image is fused with the feature map of the 3D image by splicing; S2.

14. Construct a feature pyramid based on a top-down path; S2.

15. Output cross-scale fusion feature pyramid.

6. The method for enhancing laryngeal structure and details with high resolution based on deep learning according to claim 1, characterized in that: In S3, a U-Net++ model combined with a sub-pixel convolutional layer is used to perform super-resolution reconstruction on the image that has undergone feature fusion processing, including the following steps: S3.

1. Taking the fused feature map output by the cross-scale feature pyramid as input, the fused feature map is processed by a deep learning-based semantic segmentation network to generate a vocal cord mask. S3.

2. Build the encoder part of the U-Net++ model. In the encoder part, feature extraction is performed through three encoding blocks and a bottleneck layer in sequence. S3.

3. A vocal cord attention gate module is added after each encoding block to enhance the feature area related to the free edge of the vocal cord by fusing the vocal cord mask; S3.

4. Feed the high-dimensional representation of the encoder output into the bottleneck layer, gradually upsample it using multi-level sub-pixel convolution, and introduce a dynamic kernel strategy in the sub-pixel convolution process; S3.

5. Using the dense skip connection structure of the U-Net++ model, we connect the feature maps of each level in the encoder with the feature maps of the corresponding scale in the decoder. We also introduce a structure-aware weighting mechanism in the skip connection path of the U-Net++ model to assign weights to the skip connections based on their structural importance. S3.

6. Add an edge-guided loss term, use the Sobel operator to extract the vocal cord edge gradient, and supervise it with the L1 loss.

7. The method for enhancing laryngeal structure and details high-resolution images based on deep learning according to claim 6, characterized in that: In S3.4, multi-level sub-pixel convolution is used to gradually perform upsampling, and a dynamic kernel strategy is introduced in the sub-pixel convolution process, including the following steps: S3.

41. Initialize an empty list to store the upsampled feature maps output at each level. Set the output feature map of the bottleneck layer of the U-Net++ model as the initial input and set the number of upsampling times. S3.42, use 1×1 convolution to perform channel compression on the input feature map; S3.

43. Calculate the gradient amplitude of the compressed feature map and count the proportion of high-frequency areas. S3.44, select a convolution kernel according to the dynamic kernel with a high frequency ratio, and convolve the input feature map with the selected convolution kernel; S3.

45. Perform channel expansion convolution and upsample using the PixelShuffle operation to obtain an upsampled feature map. S3.

46. Obtain the vocal cord mask at the current resolution, and calculate the average gradient intensity of the corresponding vocal cord mask area in the upsampled feature map; S3.47, fusing the upsampled feature map with the same-scale encoded feature map at interval one level; S3.

48. Save the feature map of the current upsampling output to an empty list and use the feature map as the input of the next upsampling stage; S3.

49. Repeat steps S3.42 to S3.48 until all upsampling layers are completed, and finally a high-resolution feature map is output.

8. The method for enhancing high-resolution images of laryngeal structure and details based on deep learning according to claim 6, characterized in that: In S3.5, a structure-aware weighting mechanism is introduced into the skip connection path of the U-Net++ model to assign weights to skip connections based on structural importance, including the following steps: S3.

51. Downsample the vocal cord mask to the same spatial size as the skip connection feature map to obtain a structural importance map. S3.

52. Apply the structure importance map to the skip connection feature map and perform pixel-by-pixel channel weighting. S3.

53. Concatenate the weighted feature map with the feature map of the corresponding scale in the decoder.

9. The method for enhancing laryngeal structure and details high-resolution images based on deep learning according to claim 1, characterized in that: In S4, an attention map is generated according to the anatomical structure, and a targeted local enhancement is performed on the mucosal area in the image, including the following steps: S4.

1. Take the super-resolution reconstructed image as input; S4.

2. Use the laryngeal mucosal region segmentation model U-Net to process the input image and output a binary mask; Among them, the foreground pixels represent the location of the mucosal area, and the background pixels represent the non-mucosal area; S4.

3. Apply a local enhancement network based on a dynamic filtering mechanism and adaptive weight adjustment to the segmented mucosal region; S4.4, reintegrate the locally enhanced mucosal area into the original super-resolution reconstructed image; S4.

5. Generate the final fused image.

10. The method for enhancing high-resolution images of laryngeal structure and details based on deep learning according to claim 9, characterized in that: In S4.3, applying a local enhancement network based on a dynamic filtering mechanism and adaptive weight adjustment to the segmented mucosal area includes the following steps: S4.

31. Input the segmented mucosal region image into the input layer of the local enhancement network; S4.32, extract multi-scale features of the mucosal area through multiple convolutional layers; S4.

33. Adaptively generate filter kernel weights based on the extracted features, and introduce an attention mechanism to calculate the weight distribution of key texture and lesion features in the mucosal area; S4.

34. Dynamically adjust the weight distribution of different channels and spatial positions based on the structural information of the mucosal area; S4.

35. After filtering and weight adjustment, a locally enhanced mucosal area image is generated.

Images