Laryngoscope image super-resolution improving method, system and equipment based on deep learning

Abstract

The invention provides a laryngoscope image super-resolution improving method, system and device based on deep learning. The system comprises an image preprocessing module, a shallow feature extraction module, a deep feature extraction module and an image reconstruction module. The image preprocessing module realizes blurred image enhancement based on wavelet transform; the superficial layer feature extraction module is used for extracting superficial layer features from the low-resolution laryngoscope image; the deep feature extraction module is used for capturing deep features from the shallow features, and performing feature fusion on the shallow features and the captured deep features to generate final features; and the image reconstruction module is used for reconstructing a high-resolution laryngoscope image from the learned final features. According to the scheme, the image resolution and image details can be remarkably improved, artifacts and noise interference are effectively reduced, the visual quality of the image is improved, and the image processing efficiency and stability are improved.

Description

Technical Field

[0001] The present invention relates to the fields of medical image processing, artificial intelligence and computer data processing, and in particular to a method, system and device for improving laryngoscope image super-resolution based on deep learning. Background Art

[0002] With the rapid development of medical imaging technology, the demand for high-definition images in the medical field is becoming increasingly urgent, especially in the diagnosis and surgical process, where clear images can significantly improve the accuracy of doctors' diagnoses and the success rate of operations. As an important medical device, laryngoscopes are widely used in the examination and treatment of laryngeal diseases. However, due to the particularity of the laryngoscope's working environment, obtaining high-quality images has always been a challenge.

[0003] Traditional laryngoscope images usually have problems such as low resolution, high noise, and blurred details, which bring many inconveniences to subsequent use. To improve the clarity and resolution of images, traditional methods mainly rely on hardware upgrades and improvements, such as the use of high-resolution cameras and advanced optical systems. However, this method is not only costly, but also limited by the physical space and the size of the instrument, making it difficult to be widely promoted in practical applications. On the other hand, software methods are also widely used in image super-resolution technology, such as bilinear interpolation, bicubic interpolation and other algorithms, which improve image resolution by inserting new pixels between existing image pixels. Although they can improve image quality in some cases, these methods often lead to blurred edges and loss of details because they rely on the global information of the image, making it difficult to restore high-quality detail information. Especially at high magnifications, they are prone to artifacts, noise or image blur, making it difficult to meet the needs of high-precision medical images.

[0004] In recent years, with the development of deep learning technology, especially the successful application of convolutional neural network (CNN) in the field of image processing, image super-resolution technology has ushered in new breakthroughs. By building and training deep neural networks, high-resolution images can be reconstructed from low-resolution images, significantly improving image quality. Deep learning methods can not only learn the global features of images, but also capture local details of images, so they perform well in image reconstruction.

[0005] Image super-resolution methods based on deep learning, such as SRCNN, SRResNet, and SRGAN, have demonstrated high reconstruction effects in natural image super-resolution tasks. SRCNN is a super-resolution model based on convolutional neural networks. It achieves image reconstruction by establishing a direct mapping from low resolution to high resolution. However, the SRCNN network structure is shallow and it is difficult to capture the high-order features of the image, which makes it limited in effect when processing high-precision scenes such as medical images. SRResNet introduced a residual network (ResNet) based on SRCNN. By stacking residual modules to enhance the learning ability of the network, it has achieved remarkable results in the reconstruction of natural images, but its applicability to medical images still needs further verification. SRGAN is a model that combines generative adversarial networks (GAN) for super-resolution reconstruction. By introducing adversarial losses, the generated images are made more natural and realistic. However, the network training of SRGAN is difficult, which can easily lead to unstable training, and artifacts may be generated in medical images, affecting the observation of lesions. Summary of the invention

[0006] In order to solve the above technical problems existing in the prior art, the present invention proposes a laryngoscope image super-resolution enhancement method, system and device based on deep learning neural network, aiming to solve the problems of low image resolution and blurred details existing in the prior art, especially for the characteristics of medical laryngoscope images, to provide an efficient and practical solution. This method can significantly improve the resolution and quality of laryngoscope images. Specifically, the present invention provides the following technical solutions: On the one hand, the present invention provides a laryngoscope image super-resolution enhancement system based on deep learning, the system comprising: an image preprocessing module, a shallow feature extraction module, a deep feature extraction module and an image reconstruction module; The image preprocessing module is used to preprocess the low-resolution laryngoscope image, enhance the local blurred area in the image, obtain a preprocessed laryngoscope image, and input the obtained preprocessed laryngoscope image into the shallow feature extraction module; The shallow feature extraction module is used to extract shallow features from the low-resolution laryngoscope image, and input the obtained shallow features into the deep feature extraction module through multi-scale feature fusion; The deep feature extraction module is used to capture deep features from shallow features, generate final features by fusing the shallow features with the captured deep features, and input these final features into the image reconstruction module; The image reconstruction module is used to reconstruct a complete image from the learned final features, thereby outputting a high-resolution laryngoscope image; The deep feature extraction module includes two paths, one is a deep feature extraction unit and an enhanced convolutional attention unit connected in sequence, wherein the deep feature extraction unit is provided with multiple parallel paths; the other is composed of a graph convolutional neural network unit and two parallel causal convolutional units, the two parallel causal convolutional units are both connected to the graph convolutional neural network unit, the graph convolutional neural network unit receives shallow features, and the two parallel causal convolutional units are both composed of causal convolutional blocks and activation functions.

[0007] Preferably, the shallow feature extraction module includes: three parallel 1×1 convolution and ReLU activation layers, 3×3 convolution and ReLU activation layers, and 5×5 convolution and ReLU activation layers; a primary feature fusion unit, and a 1×1 convolution layer connected to the primary feature fusion unit; The low-resolution laryngoscope image is respectively input into a 1×1 convolution and ReLU activation layer, a 3×3 convolution and ReLU activation layer, and a 5×5 convolution and ReLU activation layer arranged in parallel, and three primary features are respectively extracted; the three primary features are fused by a primary feature fusion unit, and then passed through a 1×1 convolution layer to obtain shallow features.

[0008] Preferably, the deep feature extraction unit comprises five parallel paths, a feature fusion unit and a convolution layer with a convolution kernel of 1×1; The five parallel paths are set as follows: the first path is a 1×1 convolution layer; the second path includes a 1×1 convolution layer and a 3×3 convolution layer connected in sequence; the third path includes a 1×1 convolution layer, a 3×3 convolution layer, and an expansion convolution layer with a dilation value of 3 and a convolution kernel size of 3×3 connected in sequence; the fourth path includes a 1×1 convolution layer, a 5×5 convolution layer, and an expansion convolution layer with a dilation value of 5 and a convolution kernel size of 3×3 connected in sequence; the fifth path is directly connected to the input.

[0009] Preferably, the feature fusion unit fuses the outputs of the first path, the second path, the third path, and the fourth path, and inputs the outputs to a convolution layer with a convolution kernel of 1×1; The output of the convolution layer with a convolution kernel of 1×1 is added to the output of the fifth path.

[0010] Preferably, the image preprocessing module preprocesses the low-resolution laryngoscope image based on wavelet transform, specifically in the following manner: Construct wavelet basis functions and set constraints on the wavelet basis functions; Calculate the square of the wavelet coefficients and get them in order from small to large , where s represents the square value of the corresponding wavelet coefficient, and N represents the number of wavelet coefficients; Calculate the risk value and threshold selection parameter of each wavelet coefficient, and retain the wavelet coefficient whose absolute value of risk value is greater than the threshold selection parameter to obtain the optimized wavelet transform function; The low-resolution laryngoscope image is subjected to wavelet transform using the optimized wavelet transform function to obtain the preprocessed laryngoscope image.

[0011] Preferably, the constructed wavelet basis function is: ;

[0012] Where t represents the time variable of wavelet transform.

[0013] Preferably, the kernel function of the wavelet transform is: ; in, is the scale factor, is the translation factor, .

[0014] Preferably, the constraints are: ; in, express Fourier transform form, R represents the real number set, which means that the calculations and operations in the wavelet transform are all performed in the real number domain, and L() represents the wavelet transform.

[0015] Preferably, the risk value is calculated by calculating the square of the wavelet coefficients and sorting them from small to large to obtain According to the number of wavelet coefficients , calculate the risk value: ;

[0016] Preferably, the threshold selection parameter is calculated as follows: ; in, Represents the mean square error, that is, calculating the wavelet coefficients The mean square error is used as the risk value If , then the wavelet coefficient is retained, otherwise it is deleted, and the optimization of the wavelet transform function is completed. The low-resolution laryngoscope image is processed using the optimized wavelet transform function to improve the overall image quality.

[0017] Preferably, after the wavelet transform, it is further screened whether there is a fuzzy area, and the edge enhancement can be performed on the fuzzy area. The edge information can be enhanced by calculating the average gradient of the fuzzy area: ;

[0018] Among them, the size of the fuzzy area is M×N, The pixel gradient of the blurred area image is represented, and the average gradient G obtained after calculation replaces the pixel value of the original area to achieve the effect of edge enhancement. It should be understood here that gradient enhancement is a preferred additional method, and it is also possible to use only the above-mentioned optimized wavelet transform for processing without gradient enhancement.

[0019] Through the above calculations, after the wavelet transform is improved, the edge information of the local blurred area of ​​the image can be accurately extracted, and the improved wavelet transform is used to effectively enhance the local image blur and improve the overall quality of the image. The results are then input into the shallow feature extraction module.

[0020] Preferably, the blurred area may be, for example, an area of ​​the image that is unclear due to local overexposure, and the blurred area may be defined by manual division or the like. Alternatively, the blurred area may be determined by calculating the average contrast and image quality index of the image area. For example, when the average contrast and image quality index are respectively less than preset thresholds, the blurred area is defined. When screening the blurred area, the image may be divided into areas of a certain unit area and processed block by block, for example, divided into blocks of T pixels.

[0021] Preferably, the average contrast It is used to measure the brightness difference between different areas in the image (the higher the average contrast, the more obvious the brightness difference). The expression is: ; in, represents the average value of contrast, Indicates the strength of the local blur variance of the image.

[0022] Preferably, the image quality index It is the average degree of pixel brightness change in the image area, which is used to evaluate the quality of the image and to guide the enhancement of the local blurred area of ​​the image. The higher the value, the higher the image quality. The calculation method is: ;

[0023] Where x represents the gray value of the pixel in the original low-resolution laryngoscope image, and y represents the gray value of the pixel in the laryngoscope image after wavelet transformation. and They are and The average value of and They are and The second moment of yes and The covariance of each parameter is calculated as follows: ; ; ; ; ;

[0024] Preferably, the enhanced convolutional attention unit comprises a channel attention block and a spatial attention block connected in sequence; The channel attention block performs convolution processing on the shallow features of the input enhanced convolution attention unit, multiplies them with the shallow features, and then inputs them into the spatial attention block; The input of the spatial attention block is multiplied with the output of the spatial attention block to obtain the final features output by the deep feature extraction module.

[0025] Preferably, the channel attention block is provided with two paths: the first path includes a 1×1 convolution layer, a 3×3 convolution layer, a 1×1 convolution layer, a compression and excitation layer, a 1×1 convolution layer, a batch normalization layer and a ReLU activation layer connected in sequence; the second path is directly connected to the input of the channel attention block; The outputs of the first and second paths are summed to obtain the output of the channel attention block.

[0026] Preferably, the spatial attention block is provided with two paths: the first path sequentially passes through a 1×1 convolution layer, a strided convolution layer (Strided Conv), a pooling layer (Pooling), a grouped convolution layer (Conv Groups), an upsampling layer (Upsampling), a 1×1 convolution layer and a Sigmoid activation layer; the second path is directly connected to the input of the spatial attention block; The outputs of the first and second pathways are summed to obtain the output of the spatial attention block.

[0027] Preferably, two paths are arranged between the input and output of the compression and excitation layer: the first path is arranged to be a global pooling layer, a first fully connected layer, a ReLU activation layer, a second fully connected layer and a Sigmoid activation layer connected in sequence; the second path is directly connected to the input of the compression and excitation layer; The output of the first pass is multiplied element-wise with the output of the second pass to obtain the output of the compression and excitation layer.

[0028] Preferably, the graph convolutional neural network unit is calculated as follows: ;

[0029] in, and Respectively represent Layer and The value of the layer, represents the activation function, represents the weight matrix; ; ;

[0030] The shallow features are represented by graph G. , A represents the adjacency matrix, which is a set of real numbers, E represents the edge set, V represents the vertex set, and I represents the identity matrix. Representation Matrix The element value in the i-th row and j-th column of .

[0031] Preferably, the specific derivation process of the graph convolutional neural network unit is as follows: for a graph output by a shallow feature extraction module ,in, represents a set of vertices, i.e. pixels in the image, represents the edge set, is the adjacency matrix. Using the Laplacian operator , establish the dependency relationship between pixels, where Is the degree matrix, indicating the degree of each node. Then the Laplacian matrix is ​​normalized: ; in, is the identity matrix. Then, a convolution kernel is introduced ,in The convolution operation can be expressed as: ,in is the input signal, yes The eigenvector matrix of yes The characteristic value of It can be regarded as a characteristic value To simplify the calculation, Chebyshev polynomials are used to fit : ; in, are Chebyshev polynomials.

[0032] ; in,

[0033] Based on the above formula, the calculation method of the graph convolutional neural network is as follows: ; in, and Respectively represent Layer and The value of the layer, represents the activation function, Represents the weight matrix. Through the above rules, the graph convolutional neural network can aggregate the features of adjacent nodes, thereby establishing long-range dependencies between different nodes and significantly improving the ability of pixels to combine spatial context information.

[0034] Preferably, the causal convolution block is implemented by right alignment to ensure that the convolution only acts on the current time point, focusing on the past state and preventing any information leakage from the future state. In each convolution layer, a gated activation unit is used, which consists of two nonlinear functions: a Sigmoid function and a Tanh function. The output results of the two parallel causal convolution units are fused as follows: ; Among them, * represents the convolution operation, and the Tanh function is defined as , the Sigmoid function is defined as , where the Tanh term is used as a common activation function to constrain the output value to be in the range of [-1,1], and the Sigmoid term is used as a gating mechanism to dynamically activate or deactivate unnecessary time points. The output result is fused with the output result of the enhanced convolutional attention unit.

[0035] Preferably, the image reconstruction module is composed of a first Conv unit, a sub-pixel convolution layer and a second Conv unit connected in sequence; the first Conv unit and the second Conv unit are both composed of a 1×1 convolution layer and a ReLU activation layer.

[0036] In addition, the present invention also provides a laryngoscope image super-resolution enhancement method based on deep learning, which is applied to the system as described above, and comprises: S1. Obtain low-resolution laryngoscope images , performing image preprocessing on the low-resolution laryngoscope image to obtain a preprocessed laryngoscope image, and inputting the preprocessed laryngoscope image into shallow feature extraction; S2, through the shallow feature extraction module, the shallow features of the pre-processed laryngoscope image are proposed; S3, inputting shallow features into a deep feature extraction module to extract deep features; S4, fuse shallow features with deep features to obtain the final features; S5. Input the final features into the image reconstruction module to perform image reconstruction to obtain a high-resolution laryngoscope image. .

[0037] On the other hand, the present invention also provides a laryngoscope image super-resolution enhancement device based on deep learning, which includes a processor and a memory, and the processor calls a computer program stored in the memory to execute the laryngoscope image super-resolution enhancement method based on deep learning as described above.

[0038] Compared with the prior art, this solution has the following beneficial effects: (1) Significant improvement in image resolution and detail Through the improved deep learning network structure, this scheme achieves high-quality reconstruction of low-resolution laryngoscope images, so that the generated high-resolution images can clearly present the detailed information of tissues and lesions, especially effectively enhancing the texture and edge details of tiny lesions.

[0039] (2) Effectively reduce artifacts and noise interference and improve image visual quality Traditional super-resolution methods often introduce artifacts and noise when enlarging images, affecting the diagnostic effect. This scheme significantly reduces artifacts and noise interference in the reconstruction process by introducing a specific loss control mechanism in the network design, making the generated images more natural and with more realistic details.

[0040] (3) Improved image processing efficiency and stability This solution optimizes the deep learning model to achieve higher computational efficiency while maintaining high-quality reconstruction, meeting the needs of real-time clinical applications. At the same time, the stability of the model during training has been improved, and it can maintain consistent reconstruction effects in different image samples and enhancement tasks, ensuring the quality consistency and diagnostic reliability of the generated images.

[0041] (4) Provides a convenient and efficient tool for medical imaging data analysis This solution is also highly adaptable and can be integrated into existing laryngoscope equipment and image analysis systems, and is simple and convenient to operate. In addition, this solution can also be extended to other medical imaging fields, such as endoscopes, fundus scopes, and other low-resolution medical imaging scenarios, and has great application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram of a network architecture according to an embodiment of the present invention; Figure 2 is a structural diagram of a shallow feature extraction module according to an embodiment of the present invention; Figure 3 is a structural diagram of an MRDA unit according to an embodiment of the present invention; Figure 4 4 is a structural diagram of a channel attention block (CAB) according to an embodiment of the present invention; Figure 5 A structural diagram of a compression and excitation layer (SE layer) according to an embodiment of the present invention; Figure 6 4 is a structural diagram of a spatial attention block (SAB) according to an embodiment of the present invention; Figure 7 FIG. 4 is a structural diagram of an image reconstruction module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Those skilled in the art should know that the following specific embodiments or specific implementations are a series of optimized settings listed by the present invention to further explain the specific content of the invention, and these settings can be combined or used in association with each other, unless the present invention clearly states that some or a specific embodiment or implementation cannot be associated or used together with other embodiments or implementations. At the same time, the following specific embodiments or implementations are only used as the most optimized settings, and are not to be understood as limiting the scope of protection of the present invention.

[0046] The purpose of the present invention is to solve the deficiencies of existing laryngoscope images in terms of resolution and detail presentation, to provide higher quality image information for clinical use, and to assist doctors in the accurate observation and diagnosis of laryngeal lesions. Due to the limitations of laryngoscope equipment in hardware imaging, the images obtained are often of low resolution and blurred details. Especially when observing tiny lesions, the image clarity directly affects the doctor's judgment and may lead to missed diagnosis or misdiagnosis. Therefore, improving the clarity and detail restoration of laryngoscope images is an important demand in this field. Based on deep learning super-resolution technology, the present invention proposes a new laryngoscope image super-resolution improvement scheme, which aims to achieve the following technical goals: (1) Enhance image resolution and improve the presentation of lesion details; (2) Reduce artifacts and noise interference and improve image quality; (3) Improve sensitivity to lesions and highlight medical image features; (4) Stable and efficient training process facilitates clinical application.

[0047] The following is a detailed description of this solution in conjunction with specific embodiments: Figure 1 As shown, the laryngoscope image super-resolution enhancement system proposed in this scheme includes three main parts: a shallow feature extraction module, a deep feature extraction module and an image reconstruction module.

[0048] The shallow feature extraction module is used to extract shallow feature data from the low-resolution laryngoscope image, and input the obtained shallow features into the deep feature extraction module through multi-scale feature fusion.

[0049] The deep feature extraction module is used to capture deep features from shallow features, generate richer feature information by fusing shallow features with captured deep features, form final features, and input these final feature data into the image reconstruction module; The deep feature extraction module includes two paths, one of which is a deep feature extraction unit and an enhanced convolutional attention unit connected in sequence, wherein the deep feature extraction unit is provided with multiple parallel paths; the other is composed of a graph convolutional neural network unit and two parallel causal convolutional units, wherein the two parallel causal convolutional units are both connected to the graph convolutional neural network unit, wherein the graph convolutional neural network unit receives shallow features, and the two parallel causal convolutional units are both composed of a causal convolutional block and an activation function. After the features obtained by the two causal convolutional units are multiplied, they are fused with the output of the enhanced convolutional attention unit (i.e., concat operation), and then fused with the shallow features.

[0050] The image reconstruction module is used to reconstruct a complete image from the learned final feature data, thereby outputting a high-resolution laryngoscope image. It should be noted here that the above-mentioned feature fusion can be performed by, for example, a concat operation, which is a common method in the art and will not be described here.

[0051] The following is an elaboration of the structure of each key module.

[0052] 1. Combination Figure 2 As shown, the shallow feature extraction module is used to extract shallow feature data from low-resolution laryngoscope images. The module uses 1×1 convolution + ReLU activation layer, 3×3 convolution + ReLU activation layer and 5×5 convolution + ReLU activation layer to capture primary features, that is, the activation layers corresponding to the above three convolution kernels are arranged in parallel. The shallow features are obtained by fusing the primary features extracted at three scales and then passing through a 1x1 convolution, and the obtained shallow features are input into the deep feature extraction module. Here, the fusion operation of the primary features can be performed by, for example, the concat operation, that is, the primary features output by each channel are spliced.

[0053] 2. The deep feature extraction module consists of a deep feature extraction unit based on a multi-receptive field and dilated attention block (MRDA) and an enhanced convolutional block attention module (eCBAM); Combination Figure 3As shown, in a preferred real-time mode, five parallel paths are set between the input and output of the deep feature extraction unit (i.e., MRDA unit). The first path passes through a 1×1 convolution layer; the second path passes through a 1×1 convolution layer and a 3×3 convolution layer in sequence; the third path passes through a 1×1 convolution layer, a 3×3 convolution layer, and a dilated convolution layer with a dilation value of 3 and a convolution kernel size of 3×3 in sequence; the fourth path passes through a 1×1 convolution layer, a 5×5 convolution layer, and a dilated convolution layer with a dilation value of 5 and a convolution kernel size of 3×3 in sequence; the fifth path is directly connected to the input. The outputs of the first, second, third, and fourth paths are fused (for example, fused by a concat operation), passed through a 1×1 convolution layer, and then added with the output of the fifth path to obtain a fused result, and the result is input to the eCBAM unit. It should be noted here that, in this embodiment, 5 paths are set for the deep feature extraction unit. Of course, those skilled in the art can appropriately adjust the number of paths and the convolution kernel size of the corresponding paths based on the actual image processing requirements and the data conditions of the image object, such as adjusting the number of paths to 3, 2, etc. For example, in the case of setting 3 paths, the first path can be used to pass through a 1×1 convolution layer, a 3×3 convolution layer and an expansion value of 3 and a convolution kernel size of 3×3. The second path passes through a 1×1 convolution layer, a 5×5 convolution layer and a expansion value of 5 and a convolution kernel size of 3×3. The third path is directly connected to the input, etc. These settings should all be deemed to fall within the scope of protection of the present invention.

[0054] The enhanced convolutional attention unit (eCBAM unit) consists of two sub-modules: the channel attention block (CAB) and the spatial attention block (SAB). Figure 4 , Figure 5 , Figure 6As shown, the input of the eCBAM unit is subjected to one-dimensional convolution through the CAB unit, and the convolution result is multiplied with the output result of the MRDA unit and sent to the SAB unit as input for two-dimensional convolution. Then, the result of the two-dimensional convolution is multiplied with the output result of the CAB unit to obtain the final output. Finally, the output is sent to the image reconstruction part. It should be noted here that: the CAB unit itself has two channels set inside, and the fusion result of the two channels of CAB after processing (for example, fusion is performed by concat operation) is multiplied with the output result of the MRDA unit and then input into the SAB unit; similarly, the SAB unit is also set with two channels inside, and the output results of the two channels of SAB are fused (for example, fusion is performed by concat operation), and then multiplied with the input of the SAB unit (that is, the result of multiplying the CAB unit result with the MRDA unit output) to obtain the output.

[0055] Combination Figure 4 As shown in the figure, in this embodiment, two paths are set between the input and output of the channel attention block (i.e., CAB). The first path passes through a 1×1 convolution layer, a 3×3 convolution layer, a 1×1 convolution layer, a Squeeze and Excitation Block (SE), a 1×1 convolution layer, and a batch normalization (Batch Normalization, BN) + ReLU activation layer in sequence; the second path is directly connected to the input. Finally, the output of the first path is added to the output of the second path to obtain the final output of the channel attention block.

[0056] More preferably, combined with Figure 5 As shown in the figure, there are two paths between the input and output of the compression and excitation layer (SE layer). The first path passes through the global pooling layer, the first fully connected layer (FC), the ReLU activation layer, the second fully connected layer and the Sigmoid activation layer. The input feature map of the global pooling layer is of size H×W×C. Through the global pooling operation, a tensor of size 1×1×C is obtained; then, the feature is reduced to , where r is a scaling factor used to reduce the number of parameters and calculations; the reduced-dimensional features pass through the ReLU activation layer to keep the tensor size , and then restore the feature dimension to 1×1×C through the second fully connected layer; finally, the output is limited to the range of [0,1] through the Sigmoid activation layer to obtain a tensor of size 1×1×C, which represents the importance weight of each channel. The second path is directly connected to the input. Finally, each channel of the second path output is multiplied element by element with the corresponding weight, that is, the output of the first path is multiplied element by element with the output of the second path to adjust the feature response of each channel. This process can enhance the features of important channels while suppressing the features of unimportant channels, thereby obtaining the final output of the SE layer.

[0057] Combination Figure 6 As shown, in this embodiment, two paths are set between the input and output of the spatial attention block (SAB). The first path passes through a 1×1 convolution layer, a strided convolution layer (Strided Conv), a pooling layer (Pooling), a grouped convolution layer (Conv Groups), an upsampling layer (Upsampling), a 1×1 convolution layer and a Sigmoid activation layer in sequence. Among them, the strided convolution layer (Strided Conv) can reduce the spatial dimension of the feature map (i.e., downsampling) and extract more abstract features at the same time; the pooling layer (Pooling) further reduces the spatial dimension of the feature map and aggregates global features; the grouped convolution layer (Conv Groups) divides the channels of the feature map into several groups, and each group performs a convolution operation separately to reduce the amount of calculation and capture local spatial information; the upsampling layer (Upsampling) restores the spatial resolution of the feature map to match the size of the original input feature map; the Sigmoid activation layer limits the result to the range of [0,1] to generate the spatial attention weight. Finally, the generated spatial attention weights are multiplied element-wise with the output of the CAB unit to highlight important spatial locations and suppress unimportant parts, thus obtaining the final output of the spatial attention block.

[0058] Combine again Figure 1 As shown, in the deep feature extraction module, the other branch is mainly composed of a graph convolutional neural network unit and two parallel causal convolutional units, the two parallel causal convolutional units are both connected to the graph convolutional neural network unit, the graph convolutional neural network unit receives shallow features, and the two parallel causal convolutional units are both composed of causal convolutional blocks and activation functions. The structural design of this branch is described below.

[0059] For a shallow feature extraction module output ,in, represents a set of vertices, i.e. pixels in the image, represents the edge set, is the adjacency matrix. Using the Laplacian operator , establish the dependency relationship between pixels, where Is the degree matrix, indicating the degree of each node. Then the Laplacian matrix is ​​normalized: ; in, is the identity matrix. Then, a convolution kernel is introduced ,in The convolution operation can be expressed as: ,in is the input signal, yes The eigenvector matrix of yes The characteristic value of It can be regarded as a characteristic value To simplify the calculation, Chebyshev polynomials are used to fit : ; in, are Chebyshev polynomials.

[0060] ; in, ; Based on the above formula, the calculation method of the graph convolutional neural network is as follows:

[0061] in, and Respectively represent Layer and The value of the layer, represents the activation function, Represents the weight matrix. Through the above rules, the graph convolutional neural network can aggregate the features of adjacent nodes, thereby establishing long-range dependencies between different nodes and significantly improving the ability of pixels to combine spatial context information.

[0062] The causal convolution part is implemented by right alignment to ensure that the convolution only acts on the current time point, focusing on the past state and preventing any information leakage from the future state. In each convolution layer, a gated activation unit is used, which consists of two nonlinear functions: a Sigmoid function and a Tanh function. This unit dynamically adjusts the sequence as follows: ; Where * represents the convolution operation. The Tanh function is defined as , the Sigmoid function is defined as , where the Tanh term is used as a common activation function to constrain the output value to be in the range of [-1,1], and the Sigmoid term is used as a gating mechanism to dynamically activate or deactivate unnecessary time points.

[0063] 3. In this embodiment, combined with Figure 7 As shown in the figure, the image reconstruction module is composed of the first Conv unit, the sub-pixel convolution layer, and the second Conv unit. The Conv unit consists of a 1×1 convolution layer and a ReLU activation layer. After being processed by the image reconstruction module, the reconstructed high-resolution laryngoscope image is obtained. .

[0064] 4. In a more preferred embodiment, the scheme can also perform enhancement processing on the original input low-resolution laryngoscope image in advance, so as to facilitate the subsequent further recognition of the image features and improve the image feature recognition and reconstruction effect. Under this setting mode, the scheme also includes an image preprocessing module, which is used to preprocess the low-resolution laryngoscope image, enhance the local blurred area in the image, obtain a preprocessed laryngoscope image, and input the obtained preprocessed laryngoscope image into the shallow feature extraction module.

[0065] Preferably, the image preprocessing module preprocesses the low-resolution laryngoscope image based on wavelet transform, specifically in the following manner: Construct wavelet basis functions and set constraints on the wavelet basis functions; Calculate the square of the wavelet coefficients and get them in order from small to large , where s represents the square value of the corresponding wavelet coefficient, and N represents the number of wavelet coefficients; Calculate the risk value and threshold selection parameter of each wavelet coefficient, and retain the wavelet coefficient whose absolute value of risk value is greater than the threshold selection parameter to obtain the optimized wavelet transform function; The low-resolution laryngoscope image is subjected to wavelet transform using the optimized wavelet transform function to obtain the preprocessed laryngoscope image.

[0066] Preferably, the constructed wavelet basis function is: ;

[0067] Where t represents the time variable of wavelet transform.

[0068] Among them, the kernel function of the wavelet transform is set as: ; in, is the scale factor, is the translation factor, . R represents the set of real numbers.

[0069] Preferably, the constraints of the wavelet transform are set as follows: ; in, express Fourier transform form, R represents the real number set, which means that the calculations and operations in the wavelet transform are all performed in the real number domain, and L() represents the wavelet transform.

[0070] After that, the wavelet transform function is optimized by the risk value method. The square of the wavelet coefficients is calculated and sorted from small to large to obtain According to the number of wavelet coefficients , calculate the risk value: ;

[0071] The threshold selection parameter is further calculated, and the value of the threshold selection parameter is: ;

[0072] in, Represents the mean square error, that is, calculating the wavelet coefficients The mean square error is used as the risk value If , then the wavelet coefficient is retained, otherwise it is deleted, and the optimization of the wavelet transform function is completed. The low-resolution laryngoscope image is processed using the optimized wavelet transform function to improve the overall image quality.

[0073] In addition, it can be further preferred that after the wavelet transform, whether there is a fuzzy area is further screened, and the edge enhancement can be performed on the fuzzy area. The edge information can be enhanced by calculating the average gradient of the fuzzy area: ;

[0074] Among them, the size of the fuzzy area is M×N, The pixel gradient of the blurred area image is represented, and the average gradient G obtained after calculation replaces the pixel value of the original area to achieve the effect of edge enhancement. It should be understood here that gradient enhancement is a preferred additional method, and it is also possible to use only the above-mentioned optimized wavelet transform for processing without gradient enhancement.

[0075] Through the above calculations, after the wavelet transform is improved, the edge information of the local blurred area of ​​the image can be accurately extracted, and the improved wavelet transform is used to effectively enhance the local image blur and improve the overall quality of the image. The results are then input into the shallow feature extraction module.

[0076] In addition, it is further explained that the blurred area can be, for example, an area of ​​the image that is unclear due to local overexposure, and the blurred area can be delineated by manual division or the like. Alternatively, the blurred area is determined by calculating the average contrast and image quality index of the image area. For example, when the average contrast and image quality index are respectively less than preset thresholds, it is defined as a blurred area. When screening the blurred area, the image can be divided into areas of a certain unit area and processed block by block, for example, divided into blocks of T pixels in size. The following is a further explanation using a specific calculation of the average contrast and image quality index as an example: Preferably, the average contrast It is used to measure the brightness difference between different areas in the image. That is, the higher the average contrast, the more obvious the brightness difference, and the better the image quality (clarity). The calculation method can be set as: ; in, represents the average value of contrast, Indicates the strength of the local blur variance of the image.

[0077] Preferably, the image quality index It is the average degree of pixel brightness change in the image area, which is used to evaluate the quality of the image and to guide the enhancement of the local blurred area of ​​the image. The higher the value, the higher the image quality (clarity). The calculation method is: ;

[0078] Where x represents the gray value of the pixel in the original low-resolution laryngoscope image, and y represents the gray value of the pixel in the laryngoscope image after wavelet transformation. and They are and The average value of and They are and The second moment of yes and The covariance of each parameter is calculated as follows: ; ; ; ; ;

[0079] This solution improves the instability of traditional deep learning models during training, solves the problems of high training difficulty and unstable results, and enables the model to maintain high generation quality while training efficiently. This improvement helps to improve the operability and stability of the model in practical applications and provides reliable technical support for clinical promotion. In addition, this solution has good adaptability, can achieve high-quality image improvement in different devices and environments, and has good generalization ability.

[0080] In summary, the present invention achieves high-resolution reconstruction of laryngoscope images through an innovative deep learning super-resolution enhancement method, aiming to improve image details and clarity, reduce noise and artifact interference, enhance sensitivity to lesions, and improve the stability and practicality of the model, thereby helping doctors to diagnose and treat laryngeal diseases more accurately.

[0081] In another embodiment, based on the super-resolution enhancement system disclosed in the above embodiments, the enhanced reconstruction method for laryngoscope images in this solution is as follows: S1. Obtain low-resolution laryngoscope images ,After image preprocessing the low resolution laryngoscope image, the obtained image is input into shallow feature extraction; S2, through the shallow feature extraction module, the shallow features of the low-resolution laryngoscope image are proposed; S3, inputting shallow features into a deep feature extraction module to extract deep features; S4, fusing the shallow features with the deep features (for example, using a concat operation to fuse them) to obtain the final features; S5. Input the final features into the image reconstruction module to perform image reconstruction to obtain a high-resolution laryngoscope image. .

[0082] In another embodiment, the present solution can be implemented by means of a device, which may include a corresponding module for performing each or several steps in each of the above-mentioned embodiments. Therefore, each step or several steps of each of the above-mentioned embodiments can be performed by a corresponding module, and the electronic device may include one or more of these modules. The module may be one or more hardware modules specifically configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination. The device can be implemented using a bus architecture.

[0083] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiment of the present solution includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present solution. The processor performs the various methods and processes described above. For example, the method implementation in the present solution can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0084] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A laryngoscope image super-resolution enhancement system based on deep learning, characterized in that: The system comprises: an image preprocessing module, a shallow feature extraction module, a deep feature extraction module and an image reconstruction module; The image preprocessing module is used to preprocess the low-resolution laryngoscope image, enhance the local blurred area in the image, obtain a preprocessed laryngoscope image, and input the obtained preprocessed laryngoscope image into the shallow feature extraction module; The shallow feature extraction module is used to extract shallow features from the low-resolution laryngoscope image, and input the obtained shallow features into the deep feature extraction module through a multi-scale feature concat operation; The deep feature extraction module is used to capture deep features from shallow features, generate final features by fusing the shallow features with the captured deep features, and input these final features into the image reconstruction module; The image reconstruction module is used to reconstruct a complete image from the learned final features, thereby outputting a high-resolution laryngoscope image; The deep feature extraction module includes two paths, one is a deep feature extraction unit and an enhanced convolutional attention unit connected in sequence, wherein the deep feature extraction unit is provided with multiple parallel paths; the other is composed of a graph convolutional neural network unit and two parallel causal convolutional units, the two parallel causal convolutional units are both connected to the graph convolutional neural network unit, the graph convolutional neural network unit receives shallow features, and the two parallel causal convolutional units are both composed of causal convolutional blocks and activation functions.

2. The system according to claim 1, characterized in that The shallow feature extraction module includes: three parallel 1×1 convolution and ReLU activation layers, 3×3 convolution and ReLU activation layers, and 5×5 convolution and ReLU activation layers; a primary feature fusion unit, and a 1×1 convolution layer connected to the primary feature fusion unit; The low-resolution laryngoscope image is respectively input into a 1×1 convolution and ReLU activation layer, a 3×3 convolution and ReLU activation layer, and a 5×5 convolution and ReLU activation layer arranged in parallel to obtain three primary features respectively; the three primary features are fused through a primary feature fusion unit, and then through a 1×1 convolution layer to obtain shallow features.

3. The system according to claim 1, characterized in that The deep feature extraction unit includes five parallel paths, a feature fusion unit and a convolution layer with a convolution kernel of 1×1; The five parallel paths are set as follows: the first path is a 1×1 convolution layer; the second path includes a 1×1 convolution layer and a 3×3 convolution layer connected in sequence; the third path includes a 1×1 convolution layer, a 3×3 convolution layer, and an expansion convolution layer with a dilation value of 3 and a convolution kernel size of 3×3 connected in sequence; the fourth path includes a 1×1 convolution layer, a 5×5 convolution layer, and an expansion convolution layer with a dilation value of 5 and a convolution kernel size of 3×3 connected in sequence; The fifth path is connected directly to the input.

4. The system according to claim 3, characterized in that The feature fusion unit fuses the outputs of the first path, the second path, the third path, and the fourth path, and inputs the outputs to a convolution layer with a convolution kernel of 1×1; The output of the convolution layer with a convolution kernel of 1×1 is added to the output of the fifth path.

5. The system according to claim 1, characterized in that The image preprocessing module preprocesses the low-resolution laryngoscope image based on wavelet transform, specifically in the following way: Construct wavelet basis functions and set constraints on the wavelet basis functions; Calculate the square of the wavelet coefficients and get them in order from small to large , where s represents the square value of the corresponding wavelet coefficient, and N represents the number of wavelet coefficients; Calculate the risk value and threshold selection parameter of each wavelet coefficient, and retain the wavelet coefficient whose absolute value of risk value is greater than the threshold selection parameter to obtain the optimized wavelet transform function; The low-resolution laryngoscope image is subjected to wavelet transform using the optimized wavelet transform function to obtain the preprocessed laryngoscope image.

6. The system according to claim 1, characterized in that The enhanced convolutional attention unit includes a channel attention block and a spatial attention block connected in sequence; The channel attention block performs convolution processing on the shallow features of the input enhanced convolution attention unit, multiplies them with the shallow features, and then inputs them into the spatial attention block; The input of the spatial attention block is multiplied with the output of the spatial attention block to obtain the final features output by the deep feature extraction module.

7. The system according to claim 6, characterized in that The channel attention block is provided with two paths: the first path includes a 1×1 convolution layer, a 3×3 convolution layer, a 1×1 convolution layer, a compression and excitation layer, a 1×1 convolution layer, a batch normalization layer and a ReLU activation layer connected in sequence; the second path is directly connected to the input of the channel attention block; The outputs of the first and second paths are summed to obtain the output of the channel attention block.

8. The system according to claim 6, characterized in that The spatial attention block is provided with two paths: the first path includes a 1×1 convolution layer, a stride convolution layer, a pooling layer, a grouped convolution layer, an upsampling layer, a 1×1 convolution layer and a sigmoid activation layer connected in sequence; the second path is directly connected to the input of the spatial attention block; The outputs of the first and second pathways are summed to obtain the output of the spatial attention block.

9. A laryngoscope image super-resolution enhancement method based on deep learning, characterized in that: The method is applied to the system according to any one of claims 1 to 8, and the method comprises: S1. Obtain low-resolution laryngoscope images , performing image preprocessing on the low-resolution laryngoscope image to obtain a preprocessed laryngoscope image, and inputting the preprocessed laryngoscope image into shallow feature extraction; S2, through the shallow feature extraction module, the shallow features of the pre-processed laryngoscope image are proposed; S3, inputting shallow features into a deep feature extraction module to extract deep features; S4, fuse shallow features with deep features to obtain the final features; S5. Input the final features into the image reconstruction module to perform image reconstruction to obtain a high-resolution laryngoscope image. .

10. A laryngoscope image super-resolution enhancement device based on deep learning, characterized in that: The device comprises a processor and a memory, and the processor calls a computer program stored in the memory to execute the laryngoscope image super-resolution enhancement method based on deep learning as claimed in claim 9.

Images