Fluorescence image processing method and endoscope system

Abstract

The present invention discloses a fluorescence image processing method and an endoscope system, which relate to the field of image processing. A non-local mean filtering algorithm is used to reduce noise by utilizing the self-similarity of a fluorescence intensity map to generate a denoised fluorescence intensity map. A multi-scale retinal enhancement algorithm is used to perform exponential reduction on the weighted sum of reflection sub-maps of different scales in the logarithmic form of the denoised fluorescence intensity map to generate an enhanced fluorescence intensity map. A fluorescence spectrum map and an enhanced fluorescence intensity map are preprocessed and input into an attention segmentation enhancement network. A shallow feature map and a deep feature map are extracted using convolution and residual blocks and aggregated into a gate input feature map and a gating feature map, respectively. An attention gate is used to transform the gating feature map and the Hadamard product is performed with the gate input feature map to generate a screening feature map. A corrected fluorescence intensity map that provides both structural strength information and functional chemical information is generated by sampling, splicing, convolution and sampling with the gate input feature map, thereby achieving collaborative image denoising and enhancement with unified maps.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a fluorescence image processing method and an endoscope system. Background Art

[0002] Endoscopes are medical devices that play a vital role in medical diagnosis and treatment. However, due to the complexity and limitations of the human internal environment, endoscopic images are often affected by problems such as insufficient lighting, shadows, and blurring, which reduce image quality and usability, and hinder clinical diagnosis and treatment outcomes. To overcome these issues, endoscopic images are typically processed using techniques such as image enhancement, denoising, and brightness adjustment.

[0003] The existing Chinese patent application with publication number CN119228675A discloses a fluorescence image adaptive enhancement and noise reduction method and a fluorescence endoscope imaging system, including: loading a fluorescence endoscope image; dividing the fluorescence endoscope image to obtain D window fluorescence images; extracting the d-th window fluorescence image; performing filtering and noise reduction to generate the d-th window noise reduction image; performing noise reduction verification to generate the d-th window noise reduction verification image; continuing to perform filtering and noise reduction and noise reduction verification on each window fluorescence image within the D window fluorescence images to generate D window noise reduction verification images; performing image fusion on the D window noise reduction verification images to generate a fluorescence endoscope noise reduction image, thereby improving the noise reduction effect.

[0004] However, existing technologies only perform noise reduction and enhancement on fluorescence intensity images, which can only obtain structural information but not functional information. It is easy to ignore subtle differences in tissues, which increases the uncertainty and error of fluorescence intensity image processing. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the present invention proposes a fluorescence image processing method and an endoscope system to realize a collaborative image enhancement technology that combines the fluorescence spectrum map and the fluorescence intensity map, thereby improving the accuracy and credibility of the fluorescence intensity map.

[0006] The technical solution to achieve the purpose of the present invention is:

[0007] A fluorescence image processing method comprises the following specific steps:

[0008] Obtain fluorescence spectrum F S and fluorescence intensity map F I , where the fluorescence spectrum F S Including C wavelength fluorescence intensity sub-graphs, fluorescence intensity graph F I The fluorescence intensity at different spatial positions is recorded, and C is the total number of wavelengths;

[0009] The fluorescence intensity map F is transformed intoI The pixel value of each target point in the image is updated to the product of the pixel values ​​of the remaining comparison points and the similarity weight to generate a denoised fluorescence intensity map. Among them, the comparison point refers to the point other than the target point, and the similarity weight of the comparison point is related to the similarity measure of the comparison local area where the comparison point is located and the target local area where the target point is located;

[0010] A multi-scale retinal enhancement algorithm is used to capture and separate the denoised fluorescence intensity image using Gaussian kernels of different scales. The illumination sub-map of each scale in the logarithmic form is obtained to obtain the reflection sub-map of each scale, and the reflection sub-map of each scale is weighted summed and indexed to restore it to the enhanced fluorescence intensity map F I ';

[0011] The fluorescence spectrum F S Figure 5 and enhanced fluorescence intensity I 'Perform channel splicing and channel normalization and input into the attention segmentation enhancement network, and use parallel convolution and residual blocks to extract shallow feature maps X sup and deep feature map X deep , respectively fuse shallow feature maps X through point-by-point convolution and void space pyramid pooling sup and deep feature map X deep All channels of the gate input feature map X in and gating feature map X se , the attention gate will select the feature map X se Transformed attention coefficient graph W att AND gate input feature map X in The Hadamard product is used as the screening feature map X sift , AND gate input feature map X in Generate a corrected fluorescence intensity map through upsampling, channel concatenation, convolution and upsampling

[0012] Furthermore, the non-local mean filtering algorithm is used to process the fluorescence intensity map F I Generate denoised fluorescence intensity map The specific steps include:

[0013] For the fluorescence intensity map F I The target point in the image is determined, and the target local area centered on the target point is determined;

[0014] Traverse the fluorescence intensity map F I All alignment points in the aligner are counted and the alignment local area centered on each alignment point is determined respectively;

[0015] Calculate the sum of squares of the differences between the pixel values ​​of each compared local area and the target local area and use it as the similarity measure of each compared local area;

[0016] The inverse of the ratio of the similarity measure of each compared local area to the predefined screening control parameter is used as the exponent of the natural exponential function to obtain the exponential similarity measure of each compared local area;

[0017] Normalize the exponential similarity measure of each compared local area and multiply it with the pixel value of the corresponding comparison point to update the pixel value of the target point;

[0018] The fluorescence intensity map F I Each point in the image is updated as a target point to generate a denoised fluorescence intensity map.

[0019] Furthermore, a multi-scale retinal enhancement algorithm was used to process the denoised fluorescence intensity image. Generate enhanced fluorescence intensity map F I ', comprising the following specific steps:

[0020] Denoised fluorescence intensity map Logarithmic transformation was performed to generate a logarithmic fluorescence intensity graph LF I ;

[0021] Set Q Gaussian kernels of different scales to transform the logarithmic fluorescence intensity map LF I With the qth scale σ q Gaussian kernel G q Perform convolution operations to capture the scale σ q Corresponding lighting sub-map LF I,q , q=1,…,Q, Q is the maximum scale;

[0022] The logarithmic fluorescence intensity graph LF I Subtract the qth scale σ q Lighting fraction LF I,q Generate the qth scale σ q Logarithmic reflection plot of

[0023] Count the variance of the logarithmic reflection sub-graphs of Q scales and construct the scale variance vector δ σ , using the Softmax function to normalize the scale contribution vector w σ , the logarithmic reflection sub-graph of each scale is combined with the scale contribution vector w σ The product of the contribution weights of the corresponding scales is accumulated to generate a logarithmic reflection map And it is restored to the enhanced fluorescence intensity map F through the natural index I ′.

[0024] Specifically, the fluorescence spectrum F S Including C wavelength fluorescence intensity sub-graphs, the fluorescence spectrum F S and enhanced fluorescence intensity map FI ' Perform channel splicing to generate a multi-channel fluorescence intensity map F including C+1 channels S+I , multi-channel fluorescence intensity map F S+I The first C channels correspond to the fluorescence intensity maps of C wavelengths, and the C+1th channel corresponds to the enhanced fluorescence intensity map F I ′, the multi-channel intensity map F S+I The two-dimensional image of each channel in is normalized separately to achieve channel normalization and generate a multi-channel normalized intensity map

[0025] Furthermore, the attention segmentation enhancement network processes multi-channel normalized intensity maps Generate corrected fluorescence intensity map The specific steps include:

[0026] Normalize the multi-channel intensity map The two-dimensional image of each channel in the 3D convolution kernel is convolved with the two-dimensional convolution kernel of the corresponding channel to generate shallow feature maps of C+1 channels and spliced ​​into a shallow feature map X sup ;

[0027] The shallow feature map X sup Input residual block, which includes convolution, batch normalization and ReLU function, and the residual block mines the shallow feature map X sup The semantic information feature map in the shallow feature map X sup Superposition generates deep feature map X deep , deep feature map X deep A deep feature map consisting of C+1 channels;

[0028] The shallow feature map X sup Generate a single-channel gate input feature map X by point-by-point convolution of the shallow feature maps of C+1 channels. in ;

[0029] The deep feature map X deep Input void space pyramid pooling, void space pyramid pooling includes point-by-point convolution, 3 different dilated convolutions, global average pooling and a second point-by-point convolution, where the point-by-point convolution transforms the deep feature map X deep Aggregated into the first deep gating feature map of a single channel, three dilated convolutions are used to transform the deep feature map X through three different dilated convolution kernels. deep The second deep gating feature map, the third deep gating feature map and the fourth deep gating feature map are aggregated into a single channel respectively, and the deep feature map X is pooled by global average pooling. deepAverage pooling is performed on each channel and spliced ​​to generate a pooled deep feature map. The output results of point-by-point convolution, dilated convolution and global average pooling are spliced ​​and aggregated into a single-channel gating feature map X using a second point-by-point convolution. se ;

[0030] The attention gate selects the feature map X through the Sigmoid function se Converted into attention coefficient map W att , input the gate into the feature map X in and attention coefficient graph W att Perform Hadamard product to obtain the screening feature map X sift ;

[0031] The feature map X will be filtered sift AND gate input feature map X in The upsampling results are channel stitched and the corrected fluorescence intensity map is generated through convolution and a second upsampling.

[0032] An endoscope system includes an imaging module and an image processing module;

[0033] The imaging module splits the near-infrared light into reference light L R and activation light L A , activate light L A Excite upconversion quantum dots to generate signal light L S And split into the first signal light and the second signal light Capturing the first signal light by tuning the acousto-optic tunable filter The fluorescence intensity at C wavelengths is divided into graphs and the fluorescence spectrum F is generated. S , using quantum correlation imaging, select N sampling positions, and measure the reference light L R Perform M different phase modulations at each sampling position and respectively combine with the second signal light Merge, the infinitesimal accumulation of the sampled interference light intensity of N sampling positions after each phase modulation generates the interference light intensity of each phase modulation and forms the interference light intensity vector I, introduces the compressed sensing technology, and converts the fluorescence intensity map F I The reconstruction problem is transformed into an optimization problem, and the fluorescence intensity map F is iteratively generated by the orthogonal matching algorithm. I , C is the total number of wavelengths, M is the total number of phase modulations, and N is the total number of sampling positions;

[0034] The image processing module uses the non-local mean filtering algorithm to obtain the fluorescence intensity map F I The pixel value of each target point in the image is updated to the product of the pixel values ​​of the remaining comparison points and the similarity weight to generate a denoised fluorescence intensity map. Among them, the similarity weight of the comparison point is related to the similarity measure of the comparison local area where the comparison point is located and the target local area where the target point is located. The multi-scale retinal enhancement algorithm captures and separates the denoised fluorescence intensity image through Gaussian kernels of different scales. The illumination sub-map of each scale in the logarithmic form of is used to obtain the reflection sub-map of each scale, and the weighted sum of the reflection sub-map of each scale is used to generate the logarithmic reflection map. And exponentially reduced to enhanced fluorescence intensity map F I ′, the fluorescence intensity map F I ′ and the obtained fluorescence spectrum F S Perform channel splicing and channel normalization and input the attention segmentation enhancement network, and use parallel convolution and residual blocks to extract shallow feature maps X sup and deep feature map X deep , using point-by-point convolution and void space pyramid pooling to transform the shallow feature map X sup and deep feature map X deep All channels of are aggregated into the gate input feature map X in and gating feature map X se , the attention gate will select the feature map X se Transformed attention coefficient graph W att AND gate input feature map X in The Hadamard product is used as the screening feature map X sift , and input feature map X with the gate in The corrected fluorescence intensity map is generated by upsampling, channel splicing, convolution and upsampling in sequence.

[0035] Furthermore, the imaging module is used to generate a fluorescence spectrum F S and fluorescence intensity map F I , including an optical path splitting unit, a spectral imaging unit and a correlation imaging unit;

[0036] The optical path splitting unit splits the near infrared light into reference light L through the first beam splitter. R and activation light L A , activate light L A Used to excite upconversion quantum dots to generate signal light L carrying fluorescent signals S and transmits the signal light L through the second beam splitter S Split into the first signal light and the second signal light

[0037] The spectral imaging unit adjusts the frequency of the radio frequency driving signal of the acousto-optic tunable filter, based on the acousto-optic effect and the first signal light. Diffract at C wavelengths and measure the first signal light The fluorescence intensity of the fluorescence signal at C wavelengths is divided into graphs and combined to generate a fluorescence spectrum graph F S , C is the total number of wavelengths;

[0038] The correlation imaging unit adopts quantum correlation imaging, selects N sampling positions, and R Perform M different phase modulations at each sampling position and respectively combine with the second signal light Merge, the infinitesimal accumulation of the sampled interference light intensity of N sampling positions after each phase modulation generates the interference light intensity of each phase modulation and forms the interference light intensity vector I. Based on the phase modulation and sampling method, the measurement matrix H is determined, and the compressed sensing technology is introduced to convert the fluorescence intensity map F I The reconstruction problem is transformed into an optimization problem, and the fluorescence intensity map F is generated by iteratively solving the orthogonal matching algorithm. I , M is the total number of phase modulations, and N is the total number of sampling positions.

[0039] Furthermore, the upconversion quantum dots are activated by light L A After excitation, a signal light L carrying a fluorescent signal is generated S include:

[0040] The dopant ions in the upconversion quantum dots absorb the activation light L A The near-infrared photon transitions from the ground state to the excited state;

[0041] The excited dopant ions transfer energy to the adjacent ground-state dopant ions through non-radiative transitions. The ground-state dopant ions receive the energy and transition to the sub-excited state. They further absorb near-infrared photons and transition to the excited state.

[0042] When all the doped ions of the upconversion quantum dots are in an excited state, they emit photons through energy level transitions to generate fluorescence signals, which are consistent with the activation light L A Mixed to form signal light L S .

[0043] Specifically, the acousto-optic tunable filter generates radio frequency driving signals of different frequencies and converts them into corresponding ultrasonic signals, and selects the first signal light of a specific wavelength based on the acousto-optic effect. The generated electrical signal is stored in the integrating capacitor based on the photomultiplier tube. After the integration time, the charge distribution of the integrating capacitor is converted into a voltage signal distribution and processed into a fluorescence intensity distribution of a specific wavelength. A total of C wavelengths of fluorescence intensity distribution are obtained, and a three-dimensional coordinate system is established. The three dimensions include the pixel two-dimensional plane and the channel dimension. According to the fluorescence spectrum F SThe imaging resolution requires determining the pixel two-dimensional coordinates of each spatial two-dimensional coordinate in the pixel two-dimensional plane and the channel where the C wavelengths are located. The imaging color corresponding to each fluorescence intensity is determined according to the fluorescence intensity range. The fluorescence intensity distribution of the C wavelengths is placed in the three-dimensional coordinate system to obtain the fluorescence intensity sub-maps of the C wavelengths, and the corresponding channels are spliced ​​into the fluorescence spectrum F. S , C is the total number of wavelengths.

[0044] Furthermore, for the reference light L R Each sampling position is phase modulated M times and respectively Merging, the infinitesimal accumulation of the sampled interference light intensities at N sampling positions after each phase modulation generates the interference light intensity of each phase modulation and forms the interference light intensity vector I, including the following specific steps:

[0045] Randomly generate M different random phase sequences, the mth random phase sequence For reference light L R Perform the mth phase modulation at N sampling positions, where m=1,…,M, M is the total number of phase modulations, and N is the total number of sampling positions;

[0046] According to the mth random phase sequence The random phase of each sampling position in the reference light L R Phase modulation is performed on the field intensity at the corresponding sampling position to generate the modulated field intensity corresponding to the mth phase modulation at each sampling position and combine them into the mth modulated field intensity vector

[0047] Acquire the second signal light The field strength at N sampling locations constructs the second signal field strength vector Superimpose the mth modulation field strength vector and the second signal field strength vector And perform interference measurement to obtain the interference light intensity I of the mth phase modulation m , the interference light intensity of the mth phase modulation I m It is the superposition of the product of the sampling interference light intensity of N sampling positions and the sampling interval Δl during the mth phase modulation;

[0048] The interference light intensities of M phase modulations are obtained in sequence and combined to construct an interference light intensity vector I.

[0049] Specifically, the interferometric measurement can be regarded as a linear system, and the interference light intensity I of the mth phase modulation m It can be expressed as the corresponding response vector h m The transposed fluorescence intensity plot F I The product of the response vector h m With the mth random phase sequence Regarding the response vector h m The nth sampling position p in n The response h m,n By taking the partial derivative of the sampling interference light intensity with respect to the sampling position and substituting it into the nth sampling position p n The random phase corresponding to the mth phase modulation Obtain, arrange the transpose of the response vector of the M-time phase modulation in rows, generate and determine the measurement matrix H, the interference light intensity vector I is equal to the measurement matrix H and the fluorescence intensity map F I The product of , m = 1, ..., M, n = 1, ..., N, M is the total number of phase modulation times, N is the total number of sampling positions.

[0050] Furthermore, the compressed sensing technology is introduced to convert the fluorescence intensity map F I The reconstruction problem is transformed into an optimization problem, and the fluorescence intensity map F is iteratively generated by the orthogonal matching algorithm. I , including the following specific steps:

[0051] Define the fluorescence intensity map F I After wavelet transform matrix The mapping result is a sparse vector x. Based on the finite isometric property of the measurement matrix H and the idea of ​​compressed sensing, the fluorescence intensity map F is transformed into I The reconstruction problem is transformed into an optimization problem. The optimization problem is to minimize the L1 norm of the sparse vector x, and set the constraint condition as the mapping result of the sparse vector x through the sensor matrix A is the interference light intensity vector I, and the sensor matrix A is the wavelet transform matrix The product of the inverse matrix of and the measurement matrix H;

[0052] The constraint condition is introduced into the L1 norm of the sparse vector x and rewritten as an equivalent optimization problem in the form of the L2 norm. The equivalent optimization problem is to minimize the error between the estimated interference light intensity vector generated by the mapping of the sparse vector x through the sensor matrix A and the interference light intensity vector I;

[0053] The orthogonal matching algorithm is used to solve the equivalent optimization problem. The residual γ0 of the 0th round is set to the interference light intensity vector I, the index set Λ0 of the 0th round is set to the empty set, the maximum number of rounds is set to K, and the residual of each round is the error between the estimated interference light intensity vector and the interference light intensity vector I in the round.

[0054] In the kth round, calculate the residual γ of the k-1th round k-1 The inner product of each column of the sensor matrix A is used, and the column number of the column with the largest absolute value of the inner product is used as the index of the kth round, and the index set Λ of the k-1th round is used as the index of the kth round. k-1 Take the union to generate the index set Λ of the kth round k ;

[0055] Record the sensor matrix A in the index set Λk The columns of the kth round are retained and the rest of the columns are set to 0 to generate the iterative sensing matrix A(Λ k ) and replace the sensor matrix A in the equivalent optimization problem, and use the least squares method to obtain the estimated sparse vector of the kth round

[0056] The estimated sparse vector of the kth round Multiply it with the sensing matrix A to obtain the estimated interference light intensity vector I of the kth round k , calculate the residual γ of the kth round k And judge whether it is less than or equal to the residual threshold. If it is less than or equal to the residual threshold, stop the iteration and convert the estimated sparse vector of the kth round into Multiply by the wavelet transform matrix The inverse matrix of generates the fluorescence intensity map F I ;

[0057] If it is greater than the residual threshold, it is further determined whether the current round number k is less than the maximum round number K. If it is less than the maximum round number K, it continues to the k+1th round. If it is equal to the maximum round number K, the estimated sparse vector of the kth round is Multiply by the wavelet transform matrix The inverse matrix of generates the fluorescence intensity map F I .

[0058] Compared with the prior art, the present invention adopts a non-local mean filtering algorithm to generate a denoised fluorescence intensity map based on the self-similarity noise reduction of the local area of ​​the fluorescence intensity map, adopts a multi-scale retinal enhancement algorithm to simulate human vision, obtains the reflection sub-map of each scale in the logarithmic form of the denoised fluorescence intensity map based on Gaussian kernels of different scales, and performs weighted summation and exponential restoration to generate an enhanced fluorescence intensity map; the fluorescence spectrum map and the enhanced fluorescence intensity map are channel-joined and normalized and input into the attention segmentation enhancement network, and parallel convolution and residual blocks are used to extract shallow feature maps and deep feature maps, and all channels of the shallow feature map and the deep feature map are fused respectively by point-by-point convolution and void space pyramid pooling to obtain the gate input feature map and the selection feature map, and the attention gate uses the Hadamard product of the attention coefficient map converted from the selection feature map and the gate input feature map as the screening feature map, and the gate input feature map is combined with the gate input feature map. Figure 1 And by upsampling, channel stitching, convolution and upsampling, a corrected fluorescence intensity map is generated, which realizes the coordinated image denoising and enhancement of the map and spectrum, so that the corrected fluorescence intensity map can provide both structural intensity information and functional chemical information with higher accuracy and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of the fluorescence image processing method of the present invention;

[0060] Figure 2This is the attention segmentation enhancement network model diagram in the present invention;

[0061] Figure 3 This is a model diagram of the endoscope system in the present invention;

[0062] Figure 4 This is a flow chart of the orthogonal matching algorithm used in the present invention to solve the equivalent optimization problem. DETAILED DESCRIPTION

[0063] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0064] Example 1

[0065] like Figure 1 As shown, a specific embodiment of the present invention discloses a fluorescence image processing method, including the following specific steps:

[0066] Obtain fluorescence spectrum F S and fluorescence intensity map F I , where the fluorescence spectrum F S It includes C wavelength fluorescence intensity sub-graphs, each wavelength fluorescence intensity sub-graph records the spatial distribution of fluorescence intensity at each wavelength, and the fluorescence intensity graph F I The fluorescence intensity at different spatial positions is recorded, and C is the total number of wavelengths;

[0067] Using the non-local mean filtering algorithm, based on the fluorescence intensity map F I The fluorescence intensity map F I The pixel value of each target point in the image is updated to the product of the pixel values ​​of the remaining comparison points and the similarity weight to generate a denoised fluorescence intensity map. Among them, the comparison point is a point other than the target point, and the similarity weight of the comparison point is related to the similarity measure of the comparison local area centered on the comparison point and the target local area centered on the target point;

[0068] A multi-scale retinal enhancement algorithm is used to denoise the fluorescence intensity map. Logarithmic transformation is performed to generate a logarithmic fluorescence intensity map LF that conforms to the visual characteristics of the human eye. I , capturing the logarithmic fluorescence intensity map LF by Gaussian kernels of different scales I The illumination sub-map at each scale and the logarithmic fluorescence intensity map LF I Eliminate the reflection sub-maps at each scale and generate logarithmic reflection maps by weighted summation Logarithmic reflection graph Generate enhanced fluorescence intensity map F by exponential reduction I ';

[0069] Considering the honeycomb structure and image distortion caused by the endoscope, the fluorescence spectrum FS Figure 5 and enhanced fluorescence intensity I 'Perform channel splicing and channel normalization and input into the attention segmentation enhancement network, and use parallel convolution to extract the shallow feature map X sup And further mine the deep feature map X through the residual block deep , point-by-point convolution and void space pyramid pooling are used to fuse the shallow feature map and deep feature map of each channel to obtain the gate input feature map X in and gating feature map X se , the attention gate will select the feature map X se Converted into attention coefficient map W att And the gate input feature map X in Perform Hadamard product to generate screening feature graph X sift , the feature map X will be filtered sift AND gate input feature map X in Generate the corrected fluorescence intensity map through upsampling, channel splicing, convolution and upsampling in sequence

[0070] Furthermore, the non-local mean filtering algorithm is used to filter the fluorescence intensity map F I Perform noise reduction to generate a denoised fluorescence intensity map The specific steps include:

[0071] For the fluorescence intensity map F I , a target local area of ​​a specified size is selected with the target point as the center. To adapt to the input of the attention segmentation enhancement network, the specified size is set to the same as the input dimension of the attention segmentation enhancement network;

[0072] In the fluorescence intensity graph F I Traverse all comparison points and select comparison local areas with the same size as the target local area with each comparison point as the center;

[0073] Calculate the sum of squares of the differences between the pixel values ​​of each compared local area and the target local area as the similarity measure of each compared local area;

[0074] The inverse of the ratio of the similarity measure of each compared local area to the screening control parameter is used as the exponent of the natural exponential function to obtain the exponential similarity measure of each compared local area, wherein the screening control parameter is a predefined constant;

[0075] Normalize the exponential similarity metrics of all compared local regions to generate similarity weights for all compared local regions. Then, the pixel value of the target point is updated to the product of the pixel values ​​of all compared points and the similarity weights of the corresponding compared local regions.

[0076] The fluorescence intensity map F IEach point in the image is updated as a target point to generate a denoised fluorescence intensity map.

[0077] Furthermore, a multi-scale retinal enhancement algorithm is used to denoise the fluorescence intensity map. Perform logarithmic transformation, multi-scale illumination sub-map capture and elimination, logarithmic reflection sub-map Generate and exponential transformation to generate the enhanced fluorescence intensity map F I ', comprising the following specific steps:

[0078] Denoised fluorescence intensity map Logarithmic transformation was performed to generate logarithmic fluorescence intensity plots The purpose is to reduce the noise of the fluorescence intensity map The light intensity range is more in line with the visual characteristics of the human retina;

[0079] Set Q Gaussian kernels G1,…,G of different scales q ,…,G Q , where G q is the qth scale σ q The corresponding Gaussian kernel is used to transform the logarithmic fluorescence intensity map LF I With the qth scale σ q The corresponding Gaussian kernel G q Perform convolution operations to capture the scale σ q Corresponding lighting sub-map LF I,q =LF I *G q , Lighting Fractal LF I,q Reflects the qth scale σ q Lower logarithmic fluorescence intensity graph LF I The illumination trend of q When it is small, the illumination fraction LF I,q It reflects the change of local detail illumination trend. When the qth scale σ q When it is very large, the illumination sub-map LF I,q It reflects the overall change in illumination trend, where * represents the convolution operation, q = 1,…,Q, and Q is the maximum scale;

[0080] The logarithmic fluorescence intensity graph LF I Subtract the qth scale σ q Lighting fraction LF I,q Generate the qth scale σ q Logarithmic reflection plot of Logarithmic reflection plot Reflects the qth scale σ qThe surface characteristics of the upconversion quantum dots and the target biological tissue, including color, texture and reflectivity, are obtained. Since the image acquisition of the human retina mainly relies on the reflection caused by the light source irradiating the surface of the object rather than the direct light source, at the same scale, the logarithmic reflection sub-map is used to generate the enhanced fluorescence intensity map F compared to the illumination sub-map. I 'The core information required;

[0081] Count the variance of the logarithmic reflection sub-graphs of Q scales and construct the scale variance vector δ σ , use the Softmax function for normalization to generate the scale contribution vector w σ =Softmax(δ σ ), scale contribution vector w σ =[w σ (1),…,w σ (q),…,w σ (Q)], where w σ (q) is the qth scale σ q The corresponding contribution weight, the qth scale σ q Logarithmic reflection plot of The larger the variance, the greater the qth scale σ q The more obvious the contrast is, the qth scale σ q The corresponding contribution weight w σ (q) is larger;

[0082] The logarithmic reflection map is generated by summing the product of the logarithmic reflection sub-map of each scale and the corresponding contribution weight Specifically

[0083] Logarithmic reflection graph Generate enhanced fluorescence intensity graph by exponential reduction using the natural index Compared with the denoised fluorescence intensity map Enhanced fluorescence intensity map F I ′ has higher contrast and clearer details while maintaining the denoised fluorescence intensity map color constancy.

[0084] Specifically, the fluorescence spectrum F S It includes C wavelength fluorescence intensity sub-graphs, each wavelength fluorescence intensity sub-graph occupies one channel, and the fluorescence spectrum F S and enhanced fluorescence intensity map F I ' Perform channel splicing to generate a multi-channel fluorescence intensity map F S+I , multi-channel fluorescence intensity map F S+I There are C+1 channels in total. The first C channels correspond to the fluorescence intensity maps of C wavelengths respectively, and the C+1th channel corresponds to the enhanced fluorescence intensity map FI ′, the multi-channel intensity map F S+I The two-dimensional image in each channel is normalized separately to achieve channel normalization and generate a multi-channel normalized intensity map Multi-channel normalized intensity map Augmenting the input of the attention segmentation network.

[0085] like Figure 2 As shown, further, the fluorescence spectrum F S Figure 5 and enhanced fluorescence intensity I 'Generate a multi-channel normalized intensity map after channel splicing and channel normalization Attention Segmentation Enhanced Network Processing Multi-channel Normalized Intensity Maps Generate corrected fluorescence intensity map The specific steps include:

[0086] A 3D convolution kernel of size 3×3×(C+1) is used to normalize the multi-channel intensity map. Perform parallel convolution, which is to normalize the multi-channel intensity map The two-dimensional image of each channel in the 3D convolution kernel is respectively subjected to a 3×3 two-dimensional convolution with the two-dimensional convolution kernel of the corresponding channel to generate a shallow feature map of C+1 channels and generate a shallow feature map X by channel splicing. sup , where the 3×3×(C+1) three-dimensional convolution kernel includes C+1 channels of 3×3 two-dimensional convolution kernels, and the shallow feature map X sup Used to describe the shallow features of C+1 channels, which include the color and texture related features of the target biological tissue;

[0087] The shallow feature map X sup Input residual block, which includes convolution, batch normalization and ReLU function. The residual block further mines the shallow feature map X sup The potential semantic information feature map in the image reflects the category of the target biological tissue, the outline of the target biological tissue and the whole image context information. The whole image context information refers to the multi-channel normalized intensity map. The relative position of each local area in the two-dimensional image and the relationship between different local areas are compared with the shallow feature map X sup Superposition generates deep feature map X deep , deep feature map X deep A deep feature map consisting of C+1 channels;

[0088] The shallow feature map X sup Perform point-by-point convolution with a 1×1 convolution kernel, and adjust the shallow feature map X without affecting the dimension of the two-dimensional image. supThe number of channels is , and the interaction between the shallow feature sub-maps of C+1 channels is used to aggregate the shallow feature sub-maps of C+1 channels into a single-channel gate input feature map X in ;

[0089] The deep feature map X deep The feature extraction is enhanced by dilated spatial pyramid pooling, which includes point-by-point convolution based on 1×1 convolution kernel, dilated convolution with 3 different dilation rates, global average pooling and second point-by-point convolution. Point-by-point convolution, dilated convolution and global average pooling are parallel network structures. Point-by-point convolution aggregates the deep feature maps of C+1 channels into the first deep gating feature map of a single channel. The dilated convolution with 3 different dilation rates captures C through dilated convolution kernels of 3 different sizes. The semantic information features of the deep feature maps of C+1 channels under different receptive fields are aggregated to generate the second deep gating feature map, the third deep gating feature map and the fourth deep gating feature map of a single channel. The global average pooling averages the deep feature maps of C+1 channels and re-splices them to generate the pooled deep feature map of C+1 channels. The output results of point-by-point convolution, dilated convolution and global average pooling are spliced ​​and aggregated into a single channel gating feature map X through the second point-by-point convolution. se , the void space pyramid pooling learns to mine deep feature maps from different perspectives in 5 ways deep , so that the second point-by-point convolution generates the gated feature map X se Compared with the deep feature map X deep Has richer semantic feature information;

[0090] The attention gate selects the feature map X through the Sigmoid function se Converted into attention coefficient map W att , attention coefficient map W att Reflects the gate input feature map X in The importance of each region in the task of removing honeycomb structure and image distortion is input into the feature map X in and attention coefficient graph W att Perform Hadamard product to screen important areas and generate screening feature map X sift ;

[0091] The feature map X will be filtered sift AND gate input feature map X in The upsampling is performed by bilinear interpolation and further channel splicing, and the 3×3 convolution kernel is aggregated into a single channel and upsampled again by bilinear interpolation to generate the enhanced fluorescence intensity map F I Corrected fluorescence intensity map of the same dimension Corrected fluorescence intensity map Removed enhanced fluorescence intensity map F I ′ and the image distortion, achieving enhanced fluorescence intensity map F I ′’s secondary enhancement.

[0092] Example 2

[0093] like Figure 3 As shown, the present invention also discloses an endoscope system, including an imaging module and an image processing module;

[0094] The imaging module splits the emitted near-infrared light into reference light L R and activation light L A , activate light L A Excite upconversion quantum dots to form energy level transitions and generate signal light L carrying fluorescence signals S And further split into the first signal light and the second signal light The frequency of the RF driving signal is changed by an acousto-optic tunable filter to capture the first signal light The fluorescence intensity at C wavelengths is divided into a graph to generate a fluorescence spectrum F S , using quantum correlation imaging, select N sampling positions, and measure the reference light L R Perform M different phase modulations at each sampling position and respectively combine with the second signal light Merge, the sampling interference light intensity of N sampling positions after each phase modulation is accumulated to generate the interference light intensity of each phase modulation and further form the interference light intensity vector I. The compressed sensing technology is introduced to utilize the sparsity of the fluorescence signal in the wavelet transform domain to convert the fluorescence intensity map F I The reconstruction problem is transformed into an optimization problem, and the fluorescence intensity map F is iteratively generated by the orthogonal matching algorithm. I , C is the total number of wavelengths, M is the total number of phase modulations, and N is the total number of sampling positions;

[0095] The image processing module is used to execute the fluorescence image processing method, and includes a filtering unit, an enhancement unit and a joint enhancement unit;

[0096] The filter unit obtains the fluorescence intensity map F I , the fluorescence intensity map F is transformed into I The pixel value of each target point in the image is updated to the product of the pixel values ​​of the remaining comparison points and the similarity weight to generate a denoised fluorescence intensity map. Among them, the similarity weight of the comparison point is related to the similarity measure of the comparison local area centered on the comparison point and the target local area centered on the target point;

[0097] The enhancement unit uses a multi-scale retinal enhancement algorithm to denoise the fluorescence intensity image Perform logarithmic transformation and capture and separate the illumination sub-map of each scale through Gaussian kernels of different scales, obtain the reflection sub-map of each scale and generate the logarithmic reflection map by weighted summation Logarithmic reflection graph The fluorescence intensity map F was converted to an exponential value. I ';

[0098] Combined enhancement unit to obtain fluorescence spectrum F S , and the enhanced fluorescence intensity F I 'Perform channel splicing and channel normalization and input into the attention segmentation enhancement network, and use parallel convolution to extract the shallow feature map X sup And mine the deep feature map X through the residual block deep , using point-by-point convolution and void space pyramid pooling to transform the shallow feature map X sup and deep feature map X deep All channels of are aggregated to obtain the gate input feature map X in and gating feature map X se , the attention gate will select the feature map X se Converted into attention coefficient map W att And the gate input feature map X in Perform Hadamard product to generate screening feature graph X sift , the feature map X will be filtered sift AND gate input feature map X in Generate the corrected fluorescence intensity map through upsampling, channel splicing, convolution and upsampling in sequence

[0099] Furthermore, the imaging module is used to generate a fluorescence spectrum F S and fluorescence intensity map F I , including an optical path splitting unit, a spectral imaging unit and a correlation imaging unit;

[0100] The optical path splitting unit is equipped with a first beam splitter at the front end of the near-infrared light emitter to split the emitted near-infrared light into reference light L R and activation light L A , activate light L A It is used to illuminate and excite up-conversion quantum dots, and generate signal light L carrying fluorescence signals based on the principle of quantum dot energy level transition. S , and again through the second beam splitter to convert the signal light L S Further split into the first signal light and the second signal light

[0101] The spectral imaging unit is realized based on an acousto-optic tunable filter. By adjusting the generation frequency of the RF driving signal of the acousto-optic tunable filter, the acousto-optic effect and the first signal light are combined to form a spectrum. Diffraction occurs at C wavelengths to realize the first signal light The wavelength of the first signal light is screened by a photodetector. The fluorescence intensity sub-graphs of the fluorescence signal in C wavelengths are combined to generate the fluorescence spectrum F S , C is the total number of wavelengths;

[0102] The correlation imaging unit adopts quantum correlation imaging, selects N sampling positions, and R Perform M different phase modulations at each sampling position and respectively combine with the second signal light Merge, the sampling interference light intensity of N sampling positions after each phase modulation is accumulated by the microelement to generate the interference light intensity of each phase modulation and further form the interference light intensity vector I. The measurement matrix H is determined based on the phase modulation and sampling method. Considering the sparsity of the fluorescence signal in the wavelet transform domain, the compressed sensing technology is introduced to convert the fluorescence intensity map F I The reconstruction problem is transformed into a low-complexity optimization problem, and the optimization problem is iteratively solved by the orthogonal matching algorithm to generate the fluorescence intensity map F I , M is the total number of phase modulations, and N is the total number of sampling positions.

[0103] Specifically, the upconversion quantum dots are nanomaterials, including a matrix material and dopant ions. The phonon energy of the matrix material is low, which can effectively reduce energy loss and assist in upconversion luminescence. The concentration of the dopant ions affects the luminescence characteristics of the upconversion quantum dots. The upconversion quantum dots are marked on the target biological tissue. In this embodiment, the matrix material and the dopant ions are selected as sodium yttrium fluoride NaYF4 and erbium ions Er in the lanthanide series, respectively. 3+ .

[0104] Furthermore, the upconversion quantum dots are activated by light L A After irradiation and excitation, a signal light L carrying a fluorescence signal is generated based on the principle of quantum dot energy level transition. S include:

[0105] The upconversion quantum dots are activated by light L A After irradiation, the doped ions absorb the activation light L A Near-infrared photons directly transition from the ground state to the excited state;

[0106] Since the dopant ions in the excited state are at a high energy level, they will spontaneously transfer energy to the adjacent dopant ions in the ground state through non-radiative transition. The dopant ions in the ground state receive the transferred energy and transition to the sub-excited state.

[0107] If the doping ions in the sub-excited state continue to absorb near-infrared photons, they will further transition to the excited state;

[0108] When all the doped ions of the upconversion quantum dots are in an excited state, they emit photons through energy level transitions to generate fluorescence signals, which are consistent with the activation light L A Mixed to form signal light L S .

[0109] Specifically, the acousto-optic tunable filter generates RF drive signals of different frequencies and loads them to the piezoelectric transducer. The piezoelectric transducer converts the RF drive signals of different frequencies into corresponding ultrasonic signals. Based on the acousto-optic effect, the first signal light of a specific wavelength is combined with the first signal light of a specific wavelength in the acousto-optic crystal. Diffraction occurs, and the first signal light of a specific wavelength is diffracted Input photodetector, the photodetector converts the first signal light into The fluorescence signal in the image is converted into an electrical signal and accumulated and stored on the integrating capacitor. After the integration time, the charge distribution on the integrating capacitor is converted into a voltage signal distribution. The voltage signal distribution is proportional to the fluorescence intensity distribution of the fluorescence signal. The voltage signal distribution is further amplified and noise-reduced and converted into a fluorescence intensity distribution of a specific wavelength. A total of C wavelengths of fluorescence intensity distribution are obtained, and a three-dimensional coordinate system is established. The three dimensions include the pixel two-dimensional plane and channel dimension composed of the pixel horizontal coordinate and the pixel vertical coordinate. According to the fluorescence spectrum F S The imaging resolution requirement is met. The pixel 2D coordinates corresponding to each spatial 2D coordinate in the pixel 2D plane and the channels where the C wavelengths are located are determined. The imaging color corresponding to each fluorescence intensity is determined based on the fluorescence intensity range. The imaging color corresponding to the fluorescence intensity at each spatial 2D coordinate in the fluorescence intensity distribution of the cth wavelength is placed into the pixel 2D coordinate corresponding to the pixel 2D plane to generate the fluorescence intensity sub-map F of the cth wavelength. S,c , the fluorescence intensity graph F of the cth wavelength S,c It reflects the fluorescence intensity distribution at the cth wavelength. The fluorescence intensity sub-graphs of C wavelengths are placed into the corresponding channels to form the fluorescence spectrum graph F S , where the fluorescence spectrum F S Including C wavelength fluorescence intensity sub-graphs, since the fluorescence spectrum F S The subsequent attention segmentation enhancement network needs to be input, so the fluorescence spectrum F S The imaging resolution requirement is specifically the fluorescence spectrum F S The dimension of the fluorescence intensity sub-map of each channel in is consistent with the input dimension of the attention segmentation enhancement network, c = 1,…,C, where C is the total number of wavelengths.

[0110] Furthermore, for the reference light L R Each sampling position is phase modulated M times and respectively Merging, the infinitesimal accumulation of the sampled interference light intensities at N sampling positions after each phase modulation generates the interference light intensity of each phase modulation and forms the interference light intensity vector I, including the following specific steps:

[0111] Randomly generate M different random phase sequences, the mth random phase sequence For reference light L R Perform the mth phase modulation at N sampling positions, where is the nth sampling position p n Perform the random phase corresponding to the mth phase modulation, m = 1, ..., M, n = 1, ..., N, M is the total number of phase modulations, N is the total number of sampling positions;

[0112] Extract the mth random phase sequence The nth sampling position p in n Random phase For reference light L R At the nth sampling position p n The field strength L R (p n ) performs phase modulation to generate the nth sampling position p n The modulation field intensity corresponding to the mth phase modulation The specific formula is as follows:

[0113]

[0114] Among them, i is an imaginary number, i 2 =-1;

[0115] Generate the modulation field intensity corresponding to the mth phase modulation of N sampling positions in sequence to form the mth modulation field intensity vector in, and The first sampling position p1 and the Nth sampling position p N The modulation field intensity corresponding to the mth phase modulation;

[0116] Acquire the second signal light The field strength at N sampling locations constructs the second signal field strength vector in, and The second signal light At the 1st sampling position p1 and the Nth sampling position p N Field strength;

[0117] The mth modulation field strength vector and the second signal field strength vector The interference light intensity I of the mth phase modulation is obtained by superposition of the coupler and interference measurement.m , the interference light intensity of the mth phase modulation I m It is the superposition of the product of the sampling interference light intensity of N sampling positions and the sampling interval Δl during the mth phase modulation, which is as follows:

[0118]

[0119] Among them, I m,n is the nth sampling position p during the mth phase modulation n The sampling interference light intensity is as follows:

[0120]

[0121] in, is the reference light L R and the second signal light Phase difference;

[0122] The interference light intensity of M phase modulations is obtained in sequence and combined to construct the interference light intensity vector I=[I1,…,I M ], where I1 and I M are the interference light intensities of the 1st phase modulation and the Mth phase modulation respectively.

[0123] Specifically, the interferometric measurement can be regarded as a linear system, and the interference light intensity I of the mth phase modulation m It can be expressed as the corresponding response vector h m The transposed fluorescence intensity plot F I The product of Response vector h m With the mth random phase sequence Related, h m =[h m，1 ,…,h m,n ,…,h m,N ], where h m,n is the nth sampling position p in the mth phase modulation n The response is obtained by taking the partial derivative of the sampling interference light intensity with respect to the sampling position and substituting it into the nth sampling position p n The random phase corresponding to the mth phase modulation Obtain, arrange the transpose of the response vector of M-times phase modulation in rows to generate a certain measurement matrix Therefore, the interference light intensity vector I is equal to the measurement matrix H and the fluorescence intensity map F I The product of the interferometric measurement process can be expressed as I = HF I , where h1 and h Mare the response vectors of the 1st phase modulation and the Mth phase modulation respectively, m=1,…,M, n=1,…,N, M is the total number of phase modulations, and N is the total number of sampling positions.

[0124] Furthermore, based on the Nyquist sampling theorem, the fluorescence intensity map F is realized by sampling the interference light intensity at different positions. I The reconstruction needs to meet the sampling frequency greater than or equal to the second signal light The maximum frequency is twice that of the original frequency. At this time, the amount of interference light intensity data to be measured is too large and the complexity of subsequent reconstruction calculation is too high. In this embodiment, the measurement is only performed at N sampling positions. The total number of sampling positions N cannot meet the requirements of the Nyquist sampling theorem. If the interference measurement process is directly based on I = HF I Inverse reconstruction of fluorescence intensity map F I Affected by noise, compressed sensing technology is introduced to convert the fluorescence intensity map F I The reconstruction problem is transformed into an optimization problem, and the fluorescence intensity map F is iteratively generated by the orthogonal matching algorithm. I , including the following specific steps:

[0125] Considering the second signal light In the wavelet transform domain, the fluorescence intensity map F is defined as I After wavelet transform matrix After mapping, a sparse vector x is generated, that is, Wavelet transform matrix For existing theories;

[0126] Since the measurement matrix H satisfies the finite isometric property, based on the compressed sensing idea, the fluorescence intensity map F can be converted to the image without satisfying the Nyquist sampling theorem. I The reconstruction problem is transformed into an optimization problem. The optimization problem is specifically Among them, based on the wavelet transform matrix The reversibility of That is the fluorescence intensity map F I The goal of the optimization problem is to minimize the L1 norm of the sparse vector x to suppress the noise as much as possible to restore the fluorescence intensity map F I In order to facilitate the subsequent expression, the sensor matrix is ​​defined Among them, the finite isometric property is an existing theory;

[0127] The constraints in the optimization problem are introduced into the L1 norm of the sparse vector x, and further rewritten as an equivalent optimization problem in the form of the L2 norm. The equivalent optimization problem is specifically: That is, find a sparse vector x such that the error between the vector transformed by the sensor matrix A and the interference light intensity vector I is as small as possible;

[0128] like Figure 4 As shown, the orthogonal matching algorithm is used to solve the equivalent optimization problem, the initial residual γ0 is set to the interference light intensity vector I, the initial index set Λ0 is set to the empty set, and the maximum number of rounds K is set, wherein the initial residual γ0 and the initial index set Λ0 are the residual and index set of the 0th round respectively, and the residual of each round is used to measure the error between the estimated interference light intensity vector and the interference light intensity vector I in the round. The estimated interference light intensity vector of the round is equal to the estimated sparse vector of the round multiplied by the sensor matrix A. The initial index set Λ0 records the index in the 0th round and is therefore an empty set;

[0129] In the kth round, calculate the residual γ of the k-1th round k-1 The inner product of each column of the sensing matrix A reflects the residual γ k-1 The correlation with each column of the sensor matrix A, find the column with the largest absolute value of the inner product and use the corresponding column number as the index of the kth round, and add it to the index set Λ of the k-1th round k-1 Update the index set Λ of the kth round k ;

[0130] Solve the equivalent optimization problem in the kth round by the least squares method Get the estimated sparse vector of round k A(Λ k ) is the iterative sensing matrix in the kth round, and the index set Λ recorded in the kth round in the sensing matrix A is obtained. k The columns corresponding to all indices in the index set Λ are retained and not recorded in the kth round k After all the column elements in are set to 0, the iterative sensing matrix A(Λ k );

[0131] The estimated sparse vector of the kth round Multiply it with the sensing matrix A to obtain the estimated interference light intensity vector I of the kth round k , calculate the residual γ of the kth round k , the residual γ of the kth round k is the estimated interference light intensity vector I of the kth round k The error with the interference light intensity vector I is specifically γ k =II k ;

[0132] Determine the residual γ of the kth round k Is it less than or equal to the residual threshold? If the residual γ of the kth round k If the error is less than or equal to the residual threshold, the iteration of the orthogonal matching algorithm is stopped directly, and the estimated sparse vector of the kth round is Multiply by the wavelet transform matrix The inverse matrix of generates the fluorescence intensity map F I ;

[0133] If the residual γ of round k k If the error is greater than the residual threshold, the current round number k is determined to be less than the maximum round number K. If the current round number k is less than the maximum round number K, the k+1 round is continued. If the current iteration round number k is equal to the maximum round number K, the estimated sparse vector of the kth round is Multiply by the wavelet transform matrix The inverse matrix of generates the fluorescence intensity map F I , fluorescence intensity map F I is a two-dimensional image, the fluorescence intensity map F I Each pixel coordinate in the image corresponds uniquely to a spatial position, and the fluorescence intensity map F I The dimension of is consistent with the input dimension of the attention segmentation enhancement network.

[0134] The present invention discloses a fluorescence image processing method and an endoscope system, which relate to the field of image processing. A non-local mean filtering algorithm is used to reduce noise by utilizing the self-similarity of a fluorescence intensity map to generate a denoised fluorescence intensity map. A multi-scale retinal enhancement algorithm is used to perform exponential reduction on the weighted sum of reflection sub-maps of different scales in the logarithmic form of the denoised fluorescence intensity map to generate an enhanced fluorescence intensity map. A fluorescence spectrum map and an enhanced fluorescence intensity map are preprocessed and input into an attention segmentation enhancement network. A shallow feature map and a deep feature map are extracted using convolution and residual blocks and aggregated into a gate input feature map and a gating feature map, respectively. An attention gate is used to transform the gating feature map and the Hadamard product is performed with the gate input feature map to generate a screening feature map. A corrected fluorescence intensity map that provides both structural strength information and functional chemical information is generated by sampling, splicing, convolution and sampling with the gate input feature map, thereby achieving collaborative image denoising and enhancement with unified maps.

[0135] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A fluorescence image processing method, characterized in that: The specific steps include: The non-local mean filtering algorithm is used to update the pixel value of each target point in the acquired fluorescence intensity map to the product of the pixel values ​​of the remaining comparison points and the similarity weight, thereby generating a denoised fluorescence intensity map. The similarity weight is related to the similarity measure between the local area of ​​the comparison point and the local area of ​​the target point. A multi-scale retinal enhancement algorithm is used to capture and separate the illumination component of each scale in the logarithmic form of the denoised fluorescence intensity image using Gaussian kernels of different scales to obtain the reflectance component of each scale. The weighted sum of the reflectance component of each scale is then indexed and restored to the enhanced fluorescence intensity image. The acquired fluorescence spectrum map and the enhanced fluorescence intensity map are spliced ​​and normalized on the channel to generate a multi-channel normalized intensity map and input into the attention segmentation enhancement network. Parallel convolution and residual blocks are used to extract shallow feature maps and deep feature maps. All channels of the shallow feature map and the deep feature map are fused respectively through point-by-point convolution and void spatial pyramid pooling to obtain the gate input feature map and the selection feature map. The attention gate uses the Hadamard product of the attention coefficient map converted from the selection feature map and the gate input feature map as the screening feature map, and generates the corrected fluorescence intensity map together with the gate input feature map through upsampling, channel splicing, convolution and upsampling.

2. The fluorescence image processing method according to claim 1, wherein: The attention segmentation enhancement network includes: Parallel convolution convolves the two-dimensional image of each channel in the multi-channel normalized intensity map with the two-dimensional convolution kernel of the corresponding channel in the three-dimensional convolution kernel, generates a shallow feature sub-map of each channel and splices it into a shallow feature map; The residual block mines the semantic information feature map in the shallow feature map through convolution, batch normalization and ReLU function and superimposes it with the shallow feature map to generate a deep feature map; Point-by-point convolution aggregates the shallow feature sub-maps of each channel of the shallow feature map to generate the gate input feature map; Dilated spatial pyramid pooling enhances the features of the aggregated deep feature map through point-by-point convolution, three different dilated convolution kernels and global average pooling in parallel, and uses channel splicing and a second point-by-point convolution to aggregate the features into a gated feature map. The attention gate converts the gate feature map into an attention coefficient map through the Sigmoid function, and performs a Hadamard product between the gate input feature map and the attention coefficient map to obtain the screening feature map; The upsampling results of the screening feature map and the gate input feature map are channel-joined and the corrected fluorescence intensity map is generated through convolution and a second upsampling.

3. The fluorescence image processing method according to claim 1, wherein: The multi-scale retinal enhancement algorithm includes: Performing logarithmic transformation on the denoised fluorescence intensity map to generate a logarithmic fluorescence intensity map; Set Q Gaussian kernels of different scales and convolve the logarithmic fluorescence intensity map with the Gaussian kernel of the qth scale to capture the illumination component corresponding to the qth scale, q = 1,…,Q, where Q is the maximum scale; The logarithmic fluorescence intensity map is subtracted from the illumination component map at the qth scale to generate the logarithmic reflectance component map at the qth scale; The variances of the logarithmic reflectance sub-images of Q scales are combined to generate a scale variance vector and normalized into a scale contribution vector using the Softmax function. The product of the logarithmic reflectance sub-image of each scale and the contribution weight of the corresponding scale in the scale contribution vector is accumulated and reduced to the enhanced fluorescence intensity image through exponential reduction.

4. The fluorescence image processing method according to claim 1, wherein: The non-local means filtering algorithm includes: For a target point in the fluorescence intensity map, a target local area centered on the target point is determined; Traverse all alignment points except the target point in the fluorescence intensity map and determine the alignment local area centered on each alignment point; Calculate the sum of squares of the differences between the pixel values ​​of each compared local area and the target local area and use it as the similarity measure of each compared local area; The inverse of the ratio of the similarity measure of each compared local area to the predefined screening control parameter is used as the exponent of the natural exponential function to obtain the exponential similarity measure of each compared local area; Normalize the exponential similarity measure of each compared local area and multiply it by the pixel value of the corresponding comparison point to update the pixel value of the target point; Each point in the fluorescence intensity map is treated as a target point and updated once to generate a denoised fluorescence intensity map.

5. An endoscope system, characterized in that: including an image processing module for performing a fluorescence image processing method; The image processing module uses a non-local mean filtering algorithm to update the pixel value of each target point in the acquired fluorescence intensity map to the product of the pixel values ​​of the remaining comparison points and the similarity weight to generate a denoised fluorescence intensity map, wherein the similarity weight of the comparison point is associated with the similarity measure between the local area of ​​the comparison point and the local area of ​​the target point. The multi-scale retinal enhancement algorithm uses Gaussian kernels of different scales to capture and separate the illumination sub-maps of each scale in the logarithmic form of the denoised fluorescence intensity map to obtain the reflection sub-maps of each scale, and weighted sums the reflection sub-maps of each scale and exponentially restores them to the enhanced fluorescence intensity map. The enhanced fluorescence intensity map and the acquired fluorescence spectrum map are channel-spliced ​​and channel-normalized and input into the attention segmentation enhancement network. Parallel convolution and residual blocks are used to extract shallow feature maps and deep feature maps. Point-by-point convolution and dilated spatial pyramid pooling are used to aggregate all channels of the shallow feature map and the deep feature map to generate the gate input feature map and the gating feature map respectively. The attention gate uses the Hadamard product of the attention coefficient map converted from the gating feature map and the gate input feature map as the screening feature map, and generates the corrected fluorescence intensity map through upsampling, channel splicing, convolution and upsampling with the gate input feature map.

6. The endoscope system according to claim 5, wherein: The endoscope system further includes an imaging module for generating a fluorescence spectrum map and a fluorescence intensity map; The imaging module splits near-infrared light into reference light and activation light. The activation light excites upconversion quantum dots to generate signal light and splits it into a first signal light and a second signal light. Based on an acousto-optic tunable filter, the fluorescence intensity sub-maps of the first signal light at C wavelengths are captured and a fluorescence spectrum is generated. Quantum correlation imaging is adopted, N sampling positions are selected, the reference light is phase-modulated M times at each sampling position and is respectively merged with the second signal light. The sampled interference light intensities of the N sampling positions after each phase modulation are accumulated by microelement to generate the interference light intensity of each phase modulation and form an interference light intensity vector. Compressed sensing technology is introduced to transform the reconstruction problem of the fluorescence intensity map into an optimization problem. The fluorescence intensity map is iteratively generated through an orthogonal matching algorithm. C is the total number of wavelengths, M is the total number of phase modulations, and N is the total number of sampling positions.

7. The endoscope system according to claim 6, wherein: The imaging module includes a light path splitting unit, a spectral imaging unit and a correlation imaging unit; The optical path splitting unit splits the near-infrared light into reference light and activation light through a first beam splitter. The activation light is used to excite the up-conversion quantum dots to generate signal light carrying a fluorescent signal and is then split into a first signal light and a second signal light through a second beam splitter. The spectral imaging unit adjusts the frequency of the radio frequency driving signal of the acousto-optic tunable filter, diffracts the first signal light at C wavelengths based on the acousto-optic effect, measures the fluorescence intensity components of the fluorescence signal in the first signal light at C wavelengths, and combines them to generate a fluorescence spectrum, where C is the total number of wavelengths; The correlation imaging unit adopts quantum correlation imaging, selects N sampling positions, performs M different phase modulations on the reference light at each sampling position and merges them with the second signal light respectively. The sampled interference light intensity of N sampling positions after each phase modulation is accumulated by micro-element to generate the interference light intensity of each phase modulation and form an interference light intensity vector. The measurement matrix is ​​determined based on the phase modulation and sampling method, and the compressed sensing technology is introduced to transform the reconstruction problem of the fluorescence intensity map into an optimization problem. The fluorescence intensity map is generated by iterative solution through the orthogonal matching algorithm. M is the total number of phase modulations and N is the total number of sampling positions.

8. The endoscope system according to claim 6 or 7, wherein: The method of performing phase modulation M times on N sampling positions of the reference light and combining them with the second signal light respectively, and accumulating the sampled interference light intensities of the N sampling positions after each phase modulation to generate the interference light intensity of each phase modulation and form an interference light intensity vector, includes the following specific steps: M different random phase sequences are randomly generated, and the mth random phase sequence is used to perform the mth phase modulation of the reference light at N sampling positions, where m = 1, ..., M, where M is the total number of phase modulations and N is the total number of sampling positions; Phase modulate the field intensity of the reference light at each sampling position according to the random phase of the m-th random phase sequence, generate the modulated field intensity corresponding to the m-th phase modulation at each sampling position and combine them into the m-th modulated field intensity vector; Obtaining the field intensity of the second signal light at N sampling positions, constructing a second signal field intensity vector, superimposing the m-th modulation field intensity vector and the second signal field intensity vector and performing interference measurement to obtain the interference light intensity of the m-th phase modulation, where the interference light intensity of the m-th phase modulation is the superposition of the sampled interference light intensities at the N sampling positions during the m-th phase modulation and the product of the sampling interval; The interference light intensities of M phase modulations are obtained in sequence and combined to construct an interference light intensity vector.

9. The endoscope system according to claim 6 or 7, wherein: The compressed sensing technology is introduced to transform the reconstruction problem of the fluorescence intensity map into an optimization problem, and the fluorescence intensity map is iteratively generated through an orthogonal matching algorithm, including the following specific steps: The reconstruction problem of the fluorescence intensity map is transformed into an optimization problem. The optimization problem is to minimize the L1 norm of the sparse vector. The constraint condition is that the product of the sparse vector and the sensing matrix is ​​equal to the interference light intensity vector. The sensing matrix is ​​the product of the inverse matrix of the wavelet transform matrix and the measurement matrix. The product of the wavelet transform matrix and the fluorescence intensity map is a sparse vector. The constraint condition is introduced into the L1 norm of the sparse vector and rewritten as an equivalent optimization problem, which is to minimize the error between the estimated interference light intensity vector obtained by multiplying the sparse vector and the sensor matrix and the interference light intensity vector; The orthogonal matching algorithm is used to solve the equivalent optimization problem. The residual and index set are initialized as the interference light intensity vector and the empty set. The maximum number of rounds is set. The residual of each round is the error between the estimated interference light intensity vector and the interference light intensity vector in the round. In the current round, the inner product of the residual of the previous round and each column of the sensor matrix is ​​calculated, and the column number of the column with the largest absolute value of the inner product is added to the index set of the previous round to generate the index set of the current round; The columns of the sensor matrix recorded in the index set of the current round are retained to generate the iterative sensor matrix of the current round and replace the sensor matrix in the equivalent optimization problem. The estimated sparse vector of the current round is obtained by using the least squares method. The estimated sparse vector of the current round is multiplied by the sensor matrix to obtain the estimated interference light intensity vector of the current round, the residual of the current round is calculated and whether it is less than or equal to the residual threshold is determined. If it is less than or equal to the residual threshold, the iteration is stopped and the estimated sparse vector of the current round is multiplied by the inverse matrix of the wavelet transform matrix to generate a fluorescence intensity map; If it is greater than the residual threshold, it is determined whether the round number is less than the maximum round number. If it is less than the maximum round number, the next round of iteration is continued. If it is equal to the maximum round number, the estimated sparse vector of the round is multiplied by the inverse matrix of the wavelet transform matrix to generate a fluorescence intensity map.

10. The endoscope system according to claim 7, wherein: Determining the measurement matrix based on phase modulation and sampling method includes: The interference light intensity of the mth phase modulation is the product of the transpose of the corresponding response vector and the fluorescence intensity map. The response vector of the mth phase modulation is related to the mth random phase sequence. The response of the nth sampling position in the response vector of the mth phase modulation is obtained by taking the partial derivative of the sampled interference light intensity with respect to the sampling position and substituting it into the random phase corresponding to the mth phase modulation at the nth sampling position. The transpose of the response vector of the Mth phase modulation is arranged in rows to generate a certain measurement matrix. The interference light intensity vector is equal to the product of the measurement matrix and the fluorescence intensity map, m = 1, ..., M, n = 1, ..., N, M is the total number of phase modulations, and N is the total number of sampling positions.

Images