Optical coherence imaging noise reduction method, device, equipment and medium

Abstract

The invention discloses an optical coherence imaging noise reduction method and device, equipment and a medium, and the method comprises the following steps: obtaining a B-scan image, and constructing a training set; training an image noise reduction model based on the deep neural network by adopting the training set; obtaining a to-be-denoised B-scan image, inputting the to-be-denoised B-scan image into the trained image denoising model for denoising, and outputting a reconstructed B-scan image; wherein the image noise reduction model comprises an encoder and a decoder, and the encoder takes a B-scan image with noise as input to obtain a potential representation of the B-scan image with noise; and the decoder takes the potential representation of the B-scan with noise as input, and outputs a reconstructed B-scan noise reduction image. According to the method, the image noise reduction model is constructed through the deep neural network, the B-scan image is subjected to noise reduction, the clear B-scan image is effectively obtained, and the method has the advantages of being excellent in noise reduction effect and high in speed. The method can be widely applied to the technical field of image imaging noise reduction.

Description

Technical Field

[0001] The present invention relates to the technical field of image imaging noise reduction, and in particular to an optical coherence imaging noise reduction method, device, equipment and medium. Background Art

[0002] OCT is a tomographic imaging method that uses the coherence of light. Each tomographic scan (cross-sectional view) is called a B-scan image. Clinical diagnosis is mainly based on B-scan images. Generally speaking, the noise of a single B-scan image is relatively large. The existing technology generally obtains multiple B-scan images by repeatedly scanning the same position, aligning these multiple B-scan images, and then averaging the signals to reduce the noise. However, this will increase the scanning time, and it is difficult for patients to maintain long-term eye-opening examinations. On the other hand, filtering can traditionally be used to reduce the noise of a single B-scan image. Common noise reduction methods generally include Gaussian filtering, mean filtering, median filtering, BM3D and other methods, but the effect and speed are still less than ideal. Summary of the invention

[0003] In order to solve at least one of the technical problems existing in the prior art to a certain extent, an object of the present invention is to provide an optical coherence imaging noise reduction method, device, equipment and medium.

[0004] The first technical solution adopted by the present invention is:

[0005] An optical coherence imaging noise reduction method comprises the following steps:

[0006] Obtain B-scan images and construct a training set;

[0007] The training set is used to train the image denoising model based on deep neural network;

[0008] Obtain the B-scan image to be denoised, input the trained image denoising model to perform denoising, and output the reconstructed B-scan image;

[0009] Among them, the image denoising model includes an encoder and a decoder. The encoder takes the noisy B-scan image as input to obtain the potential representation of the noisy B-scan; the decoder takes the potential representation of the noisy B-scan as input and outputs the reconstructed B-scan denoised image.

[0010] Furthermore, the image denoising model works as follows:

[0011] For the input noisy B-scan image, it first undergoes a 3×3 convolution to obtain the feature map of the B-scan image, and then traverses a symmetrical 4-level encoder-decoder structure, where each level contains multiple MixFormer blocks; finally, the feature map passes through a MixFormer block and a 3×3 convolution to generate a residual image, and the residual image is added to the noisy B-scan image to obtain the reconstructed B-scan denoised image.

[0012] Furthermore, the MixFormer block includes a channel-based self-attention layer, a head-based self-attention layer, and two global nonlinear attention layers;

[0013] When the feature map enters the MixFormer block, it first enters the channel-based self-attention layer. The channel-based self-attention layer projects the feature map through the convolution layer to obtain Q (Query), K (Key) and V (Value), and then performs self-attention calculation on the channel;

[0014] Next, the feature map enters a global nonlinear attention layer, which enables the global receptive field to be obtained in the calculation of the feedforward network part, and its features are obtained using convolution operations; the feature map calculated by the global nonlinear attention layer enters the head-based self-attention layer, and the output feature map is calculated by another global nonlinear attention layer to obtain the final output of the MixFormer block.

[0015] Furthermore, the structure of the channel-based self-attention layer is consistent with that of the head-based self-attention layer. The difference is that the self-attention of the channel-based self-attention layer is calculated in the channel dimension, while the self-attention of the head-based self-attention layer is calculated in the head dimension, so that features of different dimensions can be obtained in one MixFormer block.

[0016] Furthermore, when training the image denoising model, the loss function uses a mixed gradient loss function:

[0017] The reconstructed image is analyzed by Advance Sobel loss function. Convolution operation is performed with the target reference image Y using 4 different first-order differential operators, and then the loss is calculated using the L1 Charbonnier loss function, which is expressed as follows:

[0018]

[0019] In the formula, l CL () represents the L1 Charbonnier loss function, and AdvSobel() represents the Advance Sobel loss function.

[0020] Furthermore, the four different first-order differential operators are:

[0021]

[0022] Furthermore, the structure of the image denoising model is a U-shaped CNN combined with a transformer model, and the input and output image sizes are the same;

[0023] Upsampling and downsampling are implemented using the pixel-shuffle and pixel-unshuffle methods respectively.

[0024] The second technical solution adopted by the present invention is:

[0025] An optical coherence imaging noise reduction device, comprising:

[0026] Data acquisition module, used to acquire B-scan images and build training sets;

[0027] A model training module is used to train an image denoising model based on a deep neural network using a training set;

[0028] An image reconstruction module is used to obtain a B-scan image to be denoised, input the trained image denoising model to perform denoising, and output a reconstructed B-scan image;

[0029] Among them, the image denoising model includes an encoder and a decoder. The encoder takes the noisy B-scan image as input to obtain the potential representation of the noisy B-scan; the decoder takes the potential representation of the noisy B-scan as input and outputs the reconstructed B-scan denoised image.

[0030] The third technical solution adopted by the present invention is:

[0031] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement an optical coherence imaging noise reduction method as described above.

[0032] The fourth technical solution adopted by the present invention is:

[0033] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement an optical coherence imaging noise reduction method as described above.

[0034] The fifth technical solution adopted by the present invention is:

[0035] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned optical coherence imaging noise reduction method.

[0036] The beneficial effects of the present invention are as follows: the present invention constructs an image denoising model through a deep neural network, denoises the OCT B-scan image, effectively obtains a clear B-scan image, and has the advantages of excellent denoising effect and fast speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 is a flowchart of the steps of an optical coherence imaging noise reduction method in an embodiment of the present invention;

[0039] Figure 2 is a structural diagram of an image denoising model in an embodiment of the present invention;

[0040] Figure 3 is a structural diagram of a channel-based self-attention layer and a head-based self-attention layer in an embodiment of the present invention;

[0041] Figure 4 is a structural diagram of a global nonlinear attention layer in an embodiment of the present invention;

[0042] Figure 5 It is a comparison diagram of a noisy B-scan image, a reconstructed B-scan image and a target reference image in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0044] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0045] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0046] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0047] Example 1

[0048] like Figure 1 As shown, this embodiment provides an optical coherence imaging noise reduction method, comprising the following steps:

[0049] S1, obtain B-scan images and build a training set;

[0050] S2, using the training set to train the image denoising model based on the deep neural network;

[0051] S3, obtaining a B-scan image to be denoised, inputting a trained image denoising model to perform denoising, and outputting a reconstructed B-scan image.

[0052] See also Figure 2 , Figure 2The overall structure of the image denoising model is a U-shaped CNN combined with a transformer model, and the input and output image sizes are the same. The image denoising model includes an encoder and a decoder. The encoder takes the noisy B-scan image as input and obtains the potential representation of the noisy B-scan; the decoder takes the potential representation of the noisy B-scan as input and outputs the reconstructed B-scan denoised image. Specifically, given a noisy image, it initially obtains the image embedding through a 3×3 convolution, and then traverses a symmetrical 4-level encoder-decoder structure, each level containing multiple MixFormer blocks. Upsampling and downsampling use the pixel-shuffle and pixel-unshuffle methods, respectively. Finally, the feature map is passed through a MixFormer block and a 3×3 convolution to generate a residual image, and the residual image is added to the noisy image to obtain the predicted denoised image.

[0053] For example, each MixFormer block contains a channel-wise self-attention layer (CWSA), a head-wise self-attention layer (HWSA) and two global non-linear attention layers (GNLA). The structures of CWSA and HWSA are as follows: Figure 3 As shown in , the structure of HWSA is consistent with that of CWSA, except that the self-attention of CWSA is calculated in the channel dimension while the self-attention of HWSA is calculated in the head dimension. Figure 4 shown.

[0054] Specifically, when the feature map enters the MixFormer block, it first enters the CWSA layer. The CWSA layer first projects the feature map through the convolution layer to obtain Q (Query), K (Key) and V (Value), and then performs self-attention calculations on the channel. The feature map then enters a GNLA layer. The GNLA layer enables the global receptive field to be obtained in the calculation of the feedforward network part. Its features are all obtained using convolution operations. Since the convolution operation has the property of parameter sharing, the number of parameters of the feedforward network implemented in this way is less than the fully connected layer used in the general transformer model. The feature map calculated by GNLA enters the HWSA. The difference between HWSA and CWSA is that HWSA performs self-attention calculations on the dimension of the head, so that features of different dimensions can be obtained in one MixFormer block. Finally, the feature map is calculated by another GNLA layer to obtain the output of a MixFormer block.

[0055] In some specific embodiments, the input feature map First, it is layer normalized, and then projected through 1x1 convolution and 3x3 depth convolution to obtain query Q, key K and value V respectively. where w p is a 1x1 convolution, w d is a 3x3 depthwise convolution, and LN is layer normalization. The difference between CWSA and HWSA lies in the reshaping operation of Q, K, and V. For CWSA that requires channel-level dot product attention calculation, Q, K, and V are reshaped. We get the attention map A = K·Q, and h is the number of heads and c is the number of channels. In contrast, for HWSA that requires head-level dot product attention calculation, Q, Get the attention map In summary, the process of CWSA and HWSA is defined as in is the output feature map of CWSA or HWSA, and α is a learnable parameter.

[0056] During the training process, the OCT device is used to collect K-space images of different parts of the eye, and the same position is scanned repeatedly to obtain multiple K-space images, which are superimposed and averaged to obtain a low-noise K-space image. The low-noise K-space image is Fourier transformed to obtain the noise-reduced B-scan image Y as the neural network learning target reference image (GT). Single or multiple B-scans are used as the input of the network. When multiple images are used, the number of input channels of the network is changed accordingly. See Figure 5 , Figure 5 (a) is a noisy B-scan image. Figure 5 (b) is the B-scan image reconstructed by the image denoising model. Figure 5 (c) is the target reference image.

[0057] The input of the image denoising model is a noisy B-scan image, and the output is a predicted denoised B-scan image by The difference between Y and Y is used as the loss function. The loss function uses a hybrid gradient loss function, which is based on the L1 Charbonnier loss function:

[0058]

[0059] Combined with the Advance Sobel loss function based on the first-order differential operator. The details are as follows:

[0060] Advance Sobel loss function Using 4 different first-order differential operators with Y: Perform convolution operations and then use the L1Charbonnier loss function to calculate their errors, which are expressed as follows:

[0061]

[0062] As an optional implementation, the Adam algorithm is used to optimize the model parameters during training to update the learnable weight parameters in the network. The learning rate is set to 1e-4 and trained for 200 epochs. The neural network can take a single or multiple noisy b-scans as input. Generally speaking, the more inputs, the better the reconstruction effect. Use random cropping, horizontal or vertical mirroring to augment the data.

[0063] Example 2

[0064] This embodiment provides an optical coherence imaging noise reduction device, including:

[0065] Data acquisition module, used to acquire B-scan images and build training sets;

[0066] A model training module is used to train an image denoising model based on a deep neural network using a training set;

[0067] An image reconstruction module is used to obtain a B-scan image to be denoised, input the trained image denoising model to perform denoising, and output a reconstructed B-scan image;

[0068] Among them, the image denoising model includes an encoder and a decoder. The encoder takes the noisy B-scan image as input to obtain the potential representation of the noisy B-scan; the decoder takes the potential representation of the noisy B-scan as input and outputs the reconstructed B-scan denoised image.

[0069] Since the device is an optical coherence imaging noise reduction device of an embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0070] Example 3

[0071] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 1 An optical coherence imaging noise reduction method is shown.

[0072] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0073] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0074] Since the electronic device is an electronic device corresponding to an optical coherence imaging noise reduction method in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0075] Example 4

[0076] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 An optical coherence imaging noise reduction method is shown.

[0077] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0078] Since the storage medium is a storage medium corresponding to an optical coherence imaging noise reduction method of an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0079] Example 5

[0080] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of an optical coherence imaging noise reduction method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing various embodiments may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0081] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0082] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0083] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. An optical coherence imaging noise reduction method, characterized in that: The following steps are involved: Obtain B-scan images and construct a training set; The training set is used to train the image denoising model based on deep neural network; Obtain the B-scan image to be denoised, input the trained image denoising model to perform denoising, and output the reconstructed B-scan image; The image denoising model includes an encoder and a decoder. The encoder takes the noisy B-scan image as input and obtains the potential representation of the noisy B-scan. The decoder takes the latent representation of the noisy B-scan as input and outputs the reconstructed B-scan denoised image.

2. The optical coherence imaging noise reduction method according to claim 1, characterized in that: The image denoising model works as follows: For the input noisy B-scan image, it first undergoes a 3×3 convolution to obtain the feature map of the B-scan image, and then traverses a symmetrical 4-level encoder-decoder structure, where each level contains multiple MixFormer blocks; finally, the feature map passes through a MixFormer block and a 3×3 convolution to generate a residual image, and the residual image is added to the noisy B-scan image to obtain the reconstructed B-scan denoised image.

3. The optical coherence imaging noise reduction method according to claim 2, characterized in that: The MixFormer block includes a channel-based self-attention layer, a head-based self-attention layer and two global nonlinear attention layers; when the feature map enters the MixFormer block, it first enters the channel-based self-attention layer, which projects the feature map through the convolution layer to obtain Q, K and V, and then performs self-attention calculation on the channel; then, the feature map enters a global nonlinear attention layer, which enables the global receptive field to be obtained in the calculation of the feedforward network part, and its features are all obtained using convolution operations; the feature map calculated by the global nonlinear attention layer enters the head-based self-attention layer, and the output feature map is calculated by another global nonlinear attention layer to obtain the final output of the MixFormer block.

4. The optical coherence imaging noise reduction method according to claim 3, characterized in that: The structure of the channel-based self-attention layer is consistent with that of the head-based self-attention layer. The difference is that the self-attention of the channel-based self-attention layer is calculated in the channel dimension, while the self-attention of the head-based self-attention layer is calculated in the head dimension. In this way, features of different dimensions can be obtained in one MixFormer block.

5. The optical coherence imaging noise reduction method according to claim 1, characterized in that: When training the image denoising model, the loss function uses a mixed gradient loss function: The reconstructed image is analyzed by Advance Sobel loss function. Convolution operation is performed with the target reference image Y using 4 different first-order differential operators, and then the loss is calculated using the L1 Charbonnier loss function, which is expressed as follows: In the formula, l CL () represents the L1 Charbonnier loss function, and AdvSobel() represents the Advance Sobel loss function.

6. The optical coherence imaging noise reduction method according to claim 5, characterized in that: The four different first-order differential operators are:

7. The optical coherence imaging noise reduction method according to claim 1, characterized in that: The structure of the image denoising model is a U-shaped CNN combined with a transformer model, and the input and output image sizes are the same; Upsampling and downsampling are implemented using the pixel-shuffle and pixel-unshuffle methods respectively.

8. An optical coherence imaging noise reduction device, characterized in that: include: Data acquisition module, used to acquire B-scan images and build training sets; A model training module is used to train an image denoising model based on a deep neural network using a training set; An image reconstruction module is used to obtain a B-scan image to be denoised, input the trained image denoising model to perform denoising, and output a reconstructed B-scan image; The image denoising model includes an encoder and a decoder. The encoder takes the noisy B-scan image as input and obtains the potential representation of the noisy B-scan. The decoder takes the latent representation of the noisy B-scan as input and outputs the reconstructed B-scan denoised image.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images