Neural network-based blood vessel image processing method and apparatus

Abstract

The application provides a blood vessel image processing method and device based on a neural network. The method comprises the following steps: obtaining a blood vessel image of an eye to be examined, dividing the blood vessel image into a plurality of blood vessel sub-images based on the input image size of a denoising model, and the blood vessel image of the eye to be examined is generated by optical coherence tomography blood vessel imaging; inputting the plurality of blood vessel sub-images into the trained denoising model, and outputting a plurality of denoised blood vessel sub-images; and obtaining a fused denoised blood vessel image based on the plurality of denoised blood vessel sub-images. In this way, a first training blood vessel image can be obtained by using a first scanning protocol with a smaller data acquisition range parameter, thereby avoiding the problem that a human eye cannot withstand long-time acquisition, so that a first training image with a higher signal-to-noise ratio can be obtained, and the prediction result of the denoising model trained by using the first training image also has a higher signal-to-noise ratio.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and apparatus for processing blood vessel images based on neural networks. Background Technology

[0002] Currently, blood flow imaging generally involves multiple OCT (Optical Coherence Tomography) image acquisitions at the same site, OCT image alignment, vascular image computation, vascular image enhancement, and denoising. Among these, deep learning is typically used for vascular image enhancement and denoising. However, existing deep learning methods suffer from the following problems:

[0003] (1) Denoising models based on deep learning all adopt the UNet network structure. During the downsampling process of UNet, there is a risk of high-frequency feature loss, which makes the denoised image have a certain blur in the details of blood vessels. At the same time, due to the limitation of the receptive field of convolution operation, it is impossible to capture the horizontal bright line features caused by eye movement noise on a large scale, resulting in unsatisfactory removal effect when removing eye movement noise, or incorrectly removing straight blood vessels as bright lines.

[0004] (2) The lack of diversity in the training data for eye movement noise affects the model’s removal effect.

[0005] (3) During the collection of training data, there is no clear indication of a scheme to use a small field of view high repetition rate scanning protocol to obtain high signal-to-noise ratio images as training labels, which results in poor training results of the model. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a vascular image processing method and apparatus based on neural networks. This method uses a first scanning protocol with a smaller data acquisition range parameter to acquire first training vascular images, avoiding the problem of prolonged acquisition that the human eye cannot sustain. This results in acquiring first training images with a higher signal-to-noise ratio (SNR), and the prediction results of the denoising model trained using these first training images also have a higher SNR. When using the denoising model for prediction, each segmented vascular sub-image can output a denoised vascular sub-image with a high SNR as the prediction result. Therefore, the fused denoised vascular image obtained based on multiple denoised vascular sub-images also has a higher SNR.

[0007] In a first aspect, embodiments of the present invention provide a vascular image processing method based on a neural network, the method comprising: acquiring a vascular image of an eye under examination; segmenting the vascular image into multiple vascular sub-images based on an input image size of a denoising model; the vascular image of the eye under examination being generated by optical coherence tomography (OCTA); inputting the multiple vascular sub-images into a trained denoising model; outputting multiple denoised vascular sub-images; and obtaining a fused denoised vascular image based on the multiple denoised vascular sub-images.

[0008] In an optional embodiment of this application, the above-mentioned segmentation of a blood vessel image into multiple blood vessel sub-images based on the input image size of the denoising model includes: performing pixel expansion on the four edges of the blood vessel image to obtain a blood vessel image to be segmented; segmenting the blood vessel image to be segmented to obtain multiple blood vessel sub-images; the size of the multiple blood vessel sub-images obtained after segmentation is equal to the size of the input image of the denoising model; and the edges of two adjacent blood vessel sub-images in the multiple blood vessel sub-images overlap by a preset width.

[0009] In an optional embodiment of this application, the above-mentioned method of obtaining a fused denoised blood vessel image based on multiple denoised blood vessel sub-images includes: obtaining denoised blood vessel sub-images output by a denoising model; shrinking the four boundaries of each output denoised blood vessel sub-image inward by half a preset width to obtain a target denoised blood vessel sub-image; and fusing multiple target denoised blood vessel sub-images to obtain a fused denoised blood vessel image.

[0010] In an optional embodiment of this application, the method further includes: using a first scanning protocol to acquire a first training vascular image included in the training dataset; processing the data acquired by the first scanning protocol to determine a second training vascular image included in the training dataset; training the denoising model to be trained based on the training dataset to obtain a trained denoising model; wherein the first training vascular image is used as the label for training, and the second training vascular image is used as the input to the denoising model.

[0011] In an optional embodiment of this application, the method further includes: acquiring a vascular image of the eye being examined using a second scanning protocol, wherein the data acquisition range parameter in the second scanning protocol is greater than the data acquisition range parameter in the first scanning protocol, and the number of repeated acquisitions parameter in the second scanning protocol is less than the number of repeated acquisitions parameter in the first scanning protocol.

[0012] In an optional embodiment of this application, the above-mentioned method of obtaining the first training blood vessel image of the training dataset using the first scanning protocol includes: determining a scanning range, wherein the area of ​​the scanning range is greater than the data acquisition range parameter in the first scanning protocol; segmenting the scanning range to obtain multiple target data acquisition ranges, wherein the area of ​​the target data acquisition range is the same as the data acquisition range parameter in the first scanning protocol; and scanning the multiple target data acquisition ranges using the first scanning protocol to obtain multiple first training blood vessel images.

[0013] In an optional embodiment of this application, the above-mentioned processing of the data obtained by the first scanning protocol to determine the second training vascular images included in the training dataset includes: performing at least one of downsampling, rotation, and translation processing on the data obtained by the first scanning protocol to determine the second training vascular images included in the training dataset.

[0014] In an optional embodiment of this application, the denoising model includes: an initial feature extraction unit, a deep feature extraction unit, and a feature restoration unit; wherein, the feature restoration unit includes a convolution module and a residual module; the deep feature extraction unit includes: multiple dilated convolution modules, each with a different dilation coefficient.

[0015] In an optional embodiment of this application, the aforementioned depth feature extraction unit further includes: a spatial attention module, which is a coordinate attention module; the coordinate attention module is used to perform pooling in the horizontal and vertical directions on the depth feature map to obtain aggregated features in the horizontal and vertical directions, and the depth feature map is a primary prediction image obtained by the convolution module and residual module in the depth feature extraction unit. In a second aspect, embodiments of the present invention also provide a vascular image processing device based on a neural network, the device including: a vascular image segmentation module, used to acquire a vascular image of an eye under examination, and segment the vascular image into multiple vascular sub-images based on the input image size of a denoising model, the vascular image of the eye under examination being generated by optical coherence tomography (OCT) vascular imaging; a denoising model processing module, used to input the multiple vascular sub-images into a trained denoising model, and output multiple denoised vascular sub-images; and a denoised vascular sub-image fusion module, used to obtain a fused denoised vascular image based on the multiple denoised vascular sub-images.

[0016] The embodiments of the present invention bring the following beneficial effects:

[0017] This invention provides a method and apparatus for vascular image processing based on neural networks. The method acquires vascular images of the examined eye generated by optical coherence tomography (OCT) vascular imaging, segments the vascular image into multiple vascular sub-images based on the input image size of a denoising model, inputs the multiple vascular sub-images into a trained denoising model, outputs multiple denoised vascular sub-images, and obtains a fused denoised vascular image based on the multiple denoised vascular sub-images.

[0018] In this method, for some patients with eye diseases or who cannot cooperate with the doctor to maintain stable fixation, it can lead to blurred or distorted images. OCT examinations are also time-consuming, and some patients may find it difficult to tolerate the requirement of prolonged fixation, especially when experiencing eye discomfort or pain, which can also result in blurred or distorted images. For these patients, only protocols with fewer repeated acquisitions can be used, resulting in low image signal-to-noise ratios and frequent noise.

[0019] In this method, a first scanning protocol with a smaller data acquisition range parameter can be used to acquire the first training blood vessel image, avoiding the problem that the human eye cannot endure long-term acquisition. This allows the acquisition of a first training image with a higher signal-to-noise ratio, and the prediction result of the denoising model trained using the first training image also has a higher signal-to-noise ratio.

[0020] When using a denoising model for prediction, each segmented blood vessel sub-image can output a denoised blood vessel sub-image with a high signal-to-noise ratio as the prediction result. Therefore, the fused denoised blood vessel image obtained based on multiple denoised blood vessel sub-images also has a higher signal-to-noise ratio.

[0021] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0022] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a neural network-based vascular image processing method provided in an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram illustrating pixel expansion of a blood vessel image according to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram illustrating image segmentation of a blood vessel image to be segmented, provided as an embodiment of the present invention.

[0027] Figure 4 A schematic diagram of a fused and denoised blood vessel image provided in an embodiment of the present invention;

[0028] Figure 5 A flowchart illustrating a training method for a denoising model provided in an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of the structure of a denoising model provided in an embodiment of the present invention;

[0030] Figure 7 A schematic diagram of a depth feature extraction unit provided in an embodiment of the present invention;

[0031] Figure 8 A schematic diagram of an original image and a predicted image provided in an embodiment of the present invention;

[0032] Figure 9 A schematic diagram of a vascular image processing device based on a neural network provided in an embodiment of the present invention;

[0033] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Currently, blood flow imaging generally involves multiple OCT image acquisitions of the same site, OCT image alignment, vascular image calculation, vascular image enhancement, and denoising. Among these, deep learning is typically used for vascular image enhancement and denoising. However, existing deep learning methods suffer from the following problems:

[0036] (1) Denoising models based on deep learning all adopt the UNet network structure. During the downsampling process of UNet, there is a risk of high-frequency feature loss, which makes the denoised image have a certain blur in the details of blood vessels. At the same time, due to the limitation of the receptive field of convolution operation, it is impossible to capture the horizontal bright line features caused by eye movement noise on a large scale, resulting in unsatisfactory removal effect when removing eye movement noise, or incorrectly removing straight blood vessels as bright lines.

[0037] (2) The lack of diversity in the training data for eye movement noise affects the model’s removal effect.

[0038] (3) During the collection of training data, there is no clear indication of a scheme to use a small field of view high repetition rate scanning protocol to obtain high signal-to-noise ratio images as training labels, which results in poor training results of the model.

[0039] Based on this, the present invention provides a neural network-based vascular image processing method and apparatus that can acquire first training vascular images using a first scanning protocol with a smaller data acquisition range parameter. This avoids the problem of prolonged acquisition that the human eye cannot sustain, thus enabling the acquisition of first training images with higher signal-to-noise ratios. The prediction results of the denoising model trained using these first training images also have higher signal-to-noise ratios. When using the denoising model for prediction, each segmented vascular sub-image can output a denoised vascular sub-image with a high signal-to-noise ratio as the prediction result. Therefore, the fused denoised vascular image obtained based on multiple denoised vascular sub-images also has a higher signal-to-noise ratio.

[0040] To facilitate understanding of this embodiment, a detailed description of a neural network-based vascular image processing method disclosed in this embodiment of the invention will be provided first.

[0041] Example 1:

[0042] This invention provides a method for processing blood vessel images based on neural networks. See [link to relevant documentation]. Figure 1 The flowchart shown illustrates a neural network-based vascular image processing method, which includes the following steps:

[0043] Step S102: Obtain the vascular image of the eye being examined, and divide the vascular image into multiple vascular sub-images based on the input image size of the denoising model. The vascular image of the eye being examined is generated by optical coherence tomography vascular imaging.

[0044] In this embodiment, the vascular image of the examined eye can be generated by optical coherence tomography (OCT). OCT utilizes the basic principle of weak coherence interferometry to detect the back reflection or multiple scattering signals of incident weak coherent light at different depths of biological tissue. Through scanning, a two-dimensional or three-dimensional structural image of the biological tissue can be obtained. After acquiring the vascular image, this embodiment can use image segmentation technology to divide the large field-of-view vascular image into multiple smaller field-of-view vascular sub-images.

[0045] In some embodiments, the four edges of the blood vessel image can be pixel-expanded to obtain the blood vessel image to be segmented; the blood vessel image to be segmented is segmented to obtain multiple blood vessel sub-images; the size of the multiple blood vessel sub-images obtained after segmentation is equal to the size of the input image of the denoising model; the edges of two adjacent blood vessel sub-images in the multiple blood vessel sub-images overlap by a preset width.

[0046] See Figure 2 The diagram illustrates a pixel-expanding process for a blood vessel image. First, the four edges of the blood vessel image are expanded pixel by pixel. The expanded blood vessel image can be referred to as the blood vessel image to be segmented. The pixel values ​​of the expanded portion of the image can be filled with 0. For example... Figure 2 As shown, the size of the blood vessel image can be 128×128, and the size of the blood vessel image to be segmented can be 178×178.

[0047] After pixel-expanding the blood vessel image to obtain the blood vessel image to be segmented, the blood vessel image to be segmented can be further segmented to obtain multiple blood vessel sub-images. The size of each blood vessel sub-image is equal to the size of the input image for the denoising model; the edges of adjacent blood vessel sub-images overlap by a preset width.

[0048] See Figure 3 The diagram illustrates an image segmentation process for a blood vessel image to be segmented. The blood vessel image to be segmented (i.e., Figure 3 The solid-line box in the image can be 178×178 pixels, and the input image size for the denoising model is 128×128 pixels. Figure 3 The image shows two adjacent sub-images of blood vessels (i.e., Figure 3 (The dashed box in the middle).

[0049] Step S104: Input multiple blood vessel sub-images into the trained denoising model and output multiple denoised blood vessel sub-images.

[0050] In this embodiment, multiple small-field-of-view vascular sub-images can be input into the trained denoising model to output a small-field-of-view predicted image with a high signal-to-noise ratio, which is called a denoised vascular sub-image.

[0051] Step S106: Obtain the fused denoised blood vessel image based on multiple denoised blood vessel sub-images.

[0052] After the denoising model outputs multiple denoised blood vessel sub-images, this embodiment can fuse the above-mentioned denoised blood vessel sub-images using image fusion technology to obtain the final large-field-of-view image, which is called the denoised blood vessel image.

[0053] This invention provides a neural network-based vascular image processing method, which acquires a vascular image of the examined eye generated by optical coherence tomography (OCT) vascular imaging, segments the vascular image into multiple vascular sub-images based on the input image size of a denoising model, inputs the multiple vascular sub-images into a trained denoising model, outputs multiple denoised vascular sub-images, and obtains a fused denoised vascular image based on the multiple denoised vascular sub-images.

[0054] In this method, for some patients with eye diseases or who cannot cooperate with the doctor to maintain stable fixation, it can lead to blurred or distorted images. OCT examinations are also time-consuming, and some patients may find it difficult to tolerate the requirement of prolonged fixation, especially when experiencing eye discomfort or pain, which can also result in blurred or distorted images. For these patients, only protocols with fewer repeated acquisitions can be used, resulting in low image signal-to-noise ratios and frequent noise.

[0055] In this method, a first scanning protocol with a smaller data acquisition range parameter can be used to acquire the first training blood vessel image, avoiding the problem that the human eye cannot endure long-term acquisition. This allows the acquisition of a first training image with a higher signal-to-noise ratio, and the prediction result of the denoising model trained using the first training image also has a higher signal-to-noise ratio.

[0056] When using a denoising model for prediction, each segmented blood vessel sub-image can output a denoised blood vessel sub-image with a high signal-to-noise ratio as the prediction result. Therefore, the fused denoised blood vessel image obtained based on multiple denoised blood vessel sub-images also has a higher signal-to-noise ratio.

[0057] In some embodiments, a denoised blood vessel sub-image output by a denoising model can be obtained; a target denoised blood vessel sub-image can be obtained by shrinking the four boundaries of each output denoised blood vessel sub-image inward by half a preset width; and multiple target denoised blood vessel sub-images can be fused to obtain a fused denoised blood vessel image.

[0058] See Figure 4 The diagram shown is a schematic of a fused denoised blood vessel image, and the denoised blood vessel sub-image (i.e. Figure 4 The solid-line box in the image can be 128×128 pixels in size, with a preset width of 16 pixels. The four boundaries of the denoised blood vessel sub-image are shrunk inward by half the preset width (i.e., 8 pixels) to obtain the target denoised blood vessel sub-image (i.e., Figure 4 The size of the dashed box in the image can be 112×112. Multiple target denoised blood vessel sub-images can then be fused to obtain a fused denoised blood vessel image.

[0059] In the output of the denoising model optimization, edge pixels are prone to artifacts or edge effects. In this embodiment, image fusion is performed using only the target denoised blood vessel sub-image with pixels in the central region, which can reduce the influence of these edges. Figure 4 The target denoised vessel sub-image shown is 128×128 in size. The target denoised vessel sub-image is located in the central region, which can avoid the checkerboard effect (i.e., unnatural artificial artifacts at the seams of each denoised vessel sub-image) from being produced in the fused denoised vessel sub-image.

[0060] Furthermore, during data acquisition, this embodiment can use fundus images for image alignment to avoid horizontal bright lines. This embodiment can also further utilize a denoising model to remove horizontal bright lines, but this will not be elaborated upon further.

[0061] Example 2:

[0062] This embodiment provides a training method for a denoising model, which is implemented based on the above embodiment, such as... Figure 5 The flowchart shown illustrates a training method for a denoising model. The training method for the denoising model in this embodiment includes the following steps:

[0063] Step S502: Use the first scanning protocol to obtain the first training blood vessel images included in the training dataset.

[0064] The training dataset in this embodiment may include a first training vascular image and a second training vascular image. The first training vascular image can be a gold standard training image obtained by repeatedly acquiring images over a small area to achieve a high signal-to-noise ratio. For example, the first training vascular image may use a first scanning protocol of 3mm × 3mm R8. R8 means scanning the same location 8 times, with a total of 8 acquisitions.

[0065] For a single vascular image acquisition, several b-scan images are acquired at the same location along the lateral direction for blood flow calculation. The entire blood flow map is obtained by scanning the entire area from top to bottom. For blood flow signal calculation at the same location, at least two b-scan images are required; therefore, this location needs to be scanned twice, meaning the number of acquisitions is 2. A higher number of acquisitions will increase the signal-to-noise ratio of the acquired blood flow signal, but the scanning time will also increase accordingly, thus increasing the risk of eye movement noise.

[0066] Eye movement noise is noise generated during the data acquisition process of blood flow imaging due to eye movements. A common manifestation is the appearance of "bright horizontal lines" in the image, which is caused by the principle of blood flow imaging scanning.

[0067] The principle of blood flow imaging is based on the fact that blood flow will cause the imaging results of the same part at different times to be different, so the blood flow signal can be calculated. However, the signal of parts without blood is the same, so the blood flow signal is 0.

[0068] When the human eye moves, the images of the same part that are actually captured multiple times to calculate the difference in blood flow signals actually come from different parts, so the difference in calculation will increase, which leads to eye movement noise.

[0069] Prolonged scanning can cause eye fatigue, making eye movements more likely. Therefore, the longer the scan time, the greater the eye movement noise. In this embodiment, by reducing the range of each acquisition, the single scan time is controlled to a very short time, ensuring that no eye movement noise is introduced when using a high number of repetitions, thereby obtaining a clean first training vascular image.

[0070] In some embodiments, a second scanning protocol can be used to acquire vascular images of the eye being examined, wherein the data acquisition range parameter in the second scanning protocol is greater than the data acquisition range parameter in the first scanning protocol, and the number of repeated acquisitions parameter in the second scanning protocol is less than the number of repeated acquisitions parameter in the first scanning protocol.

[0071] The second scanning protocol has a larger data acquisition range and fewer repetitions than the first scanning protocol. For example, the first scanning protocol can be 3mm × 3mm R8, and the second scanning protocol can be 12mm × 12mm R2 or R4. That is, the first scanning protocol has a data acquisition range of 3mm × 3mm and 8 repetitions; the second scanning protocol has a data acquisition range of 12mm × 12mm and 2 or 4 repetitions. Furthermore, the second scanning protocol can also be 26mm × 21mm R2, meaning a data acquisition range of 26mm × 21mm and 2 repetitions.

[0072] Additionally, it should be noted that in this embodiment, the data obtained by the first scanning protocol (e.g., a 178×178 image) can be segmented to obtain multiple smaller images (e.g., a 128×128 image) as the first training blood vessel images in the training set, thereby enriching the amount of data in the training set.

[0073] In some embodiments, a scanning range can be determined, the area of ​​which is larger than the data acquisition range parameter in the first scanning protocol; the scanning range is segmented to obtain multiple target data acquisition ranges, the area of ​​which is the same as the data acquisition range parameter in the first scanning protocol; the first scanning protocol is used to scan the multiple target data acquisition ranges respectively to obtain multiple first training blood vessel images.

[0074] In this embodiment, multiple first training vascular images can correspond to multiple different scanning positions of the same examined eye. Therefore, when using the denoising model, it has a better denoising effect on vascular sub-images.

[0075] Step S504: Process the data acquired by the first scanning protocol to determine the second training blood vessel images included in the training dataset.

[0076] In some embodiments, the acquired data obtained through the first scanning protocol can be processed to determine the second training vascular images included in the training dataset. Specifically, at least one of downsampling, rotation, and translation processing can be performed on the data acquired through the first scanning protocol to determine the second training vascular images included in the training dataset.

[0077] In other words, the data acquired by the first scanning protocol can be processed by downsampling, rotation, translation, or one or more of these methods to increase artificial misalignment noise, thereby improving the data acquisition range of the second scanning protocol and reducing the number of repeated acquisitions by the second scanning protocol.

[0078] In this embodiment, positional shifts caused by eye movements can be simulated through artificial means such as downsampling, rotation, and translation processing, thereby adding artificial misalignment noise to the data obtained by the first scanning protocol to obtain the second training vascular image.

[0079] In some embodiments, vascular images of the examined eyes at multiple age stages can be collected as training set data; and / or, vascular images of the examined eyes with multiple eye diseases can be collected as training set data.

[0080] In this embodiment, the age ranges can be: 0 (newborn)-6 years old as infants; 7-12 years old as children; 13-17 years old as teenagers; 18-45 years old as young adults; 46-69 years old as middle-aged; and 69 years old and above as the elderly.

[0081] The ocular diseases mentioned in this embodiment may include: macular degeneration, central serous chorioretinopathy (CSC), choroidal neovascularization (CNV), polypoidal choroidal vasculopathy (PCV), macular telangiectasia, etc.

[0082] Therefore, this embodiment can collect vascular images of the examined eyes of different age groups and different eye diseases, thus improving the richness of the training set data.

[0083] Step S506: Train the denoising model to be trained based on the training dataset to obtain the trained denoising model; wherein, the first training blood vessel image is used as the training label, and the second training blood vessel image is used as the input of the denoising model.

[0084] When training the denoising model, the second training image is used as input. The second training blood vessel image can be input into the denoising model to be trained, and the output is a predicted image. When training the denoising model, the first training blood vessel image is used as the training label. The parameters of the denoising model can be adjusted according to the output predicted image and the first training blood vessel image until training is completed, and the trained denoising model is obtained.

[0085] The method provided in this embodiment of the invention can control the single scan time to a very short time by narrowing the range of each acquisition, ensuring that no eye movement noise is introduced when using a high number of repeated acquisitions, thereby obtaining a clean first training vascular image; it can also process the data acquired by the first scanning protocol by downsampling, rotation, translation processing or one or more of these methods, increasing artificial misalignment noise, thereby improving the data acquisition range of the second scanning protocol and reducing the number of repeated acquisitions of the second scanning protocol; it can also acquire vascular images of the examined eyes of different age groups and different eye diseases, improving the richness of the training set data.

[0086] Example 3:

[0087] This embodiment provides a denoising model, which is implemented based on the above embodiments, such as... Figure 6 The diagram shows a structural schematic of a denoising model. In this embodiment, the denoising model includes an initial feature extraction unit, a deep feature extraction unit, and a feature restoration unit; wherein, the feature restoration unit includes a convolution module and a residual module.

[0088] The initial feature extraction unit is used to extract shallow features of the original image, the deep feature extraction unit is used to extract deep features of the original image, and the feature restoration unit is used to map the extracted shallow features, deep features, and the original image to obtain the final predicted image.

[0089] like Figure 6 As shown, the initial feature extraction unit includes a standard 3×3 convolution and a Linear Rectification (ReLU) activation function (i.e., block a). The ReLU activation function usually refers to a nonlinear function represented by the ramp function and its variants.

[0090] like Figure 6 As shown, the deep feature extraction unit can include multiple repeatable convolutional blocks (i.e., boxes 1-n), and the feature reconstruction unit can include a standard 3×3 convolution and a ReLU activation function (i.e., block b) as well as a residual connection. The input and output parts of the denoising model learn the residual through long-distance skip connections. This process maintains the spatial scale invariant, abandoning the traditional UNet's downsampling-upsampling structure. While maintaining the spatial resolution, dilated convolution is further used to expand the model's receptive field, thereby improving the blood vessel restoration effect.

[0091] In some embodiments, the deep feature extraction unit includes: multiple dilated convolutional modules, each with a different dilation coefficient.

[0092] See Figure 7 The diagram shown is a schematic of a deep feature extraction unit. Figure 7 Blocks 1 through 4 in the diagram are all dilated convolutional modules. These four dilated convolutional modules can have different dilation coefficients. For example, the dilation coefficient of block 4 is greater than that of block 3, which is greater than that of block 2, which is greater than that of block 1.

[0093] like Figure 7 As shown, Figure 7 In the diagram, squares 5-9 can represent a 3×3 convolution, and square 10 can represent a 1×1 convolution. Figure 7 The algorithm also incorporates three local skip connections for residual learning. It should be noted here that... Figure 7 This embodiment only shows one form of the deep feature extraction unit. The specific form of the deep feature extraction unit is not limited in this embodiment and will not be described in detail hereafter.

[0094] In some embodiments, the deep feature extraction unit further includes a spatial attention module. For example... Figure 7 As shown, Figure 7The model can include a spatial attention module, which, as a spatial attention mechanism, can make full use of location information, enabling the region of interest to be accurately captured and significantly improving the denoising model's ability to remove eye-tracking noise.

[0095] In some embodiments, the spatial attention module is a coordinate attention module; the coordinate attention module is used to perform pooling in the horizontal and vertical directions on the depth feature map to obtain aggregated features in the horizontal and vertical directions, and the depth feature map is a primary prediction image obtained by the convolution module and residual module in the depth feature extraction unit.

[0096] Coordinate attention modules are a type of spatial attention module. Common spatial attention modules transform the input into a single feature vector using two-dimensional global pooling, assigning different weights to different channels. However, this global pooling process loses spatial information. Coordinate attention modules, on the other hand, perform pooling along both the X and Y directions, decomposing the input into one-dimensional feature codes that aggregate features in two different directions. This captures long-range dependencies along one spatial direction while preserving precise location information along the other. This is particularly effective for capturing eye-movement noise in vascular images, especially horizontal bright line noise, and can further improve the denoising model's ability to remove eye-movement noise.

[0097] See Figure 8 The diagram shown illustrates an original image and a predicted image, as follows: Figure 8 As shown, the denoised predicted image shows a significant improvement in eye movement noise removal compared to the original image, and the image signal-to-noise ratio is significantly improved. This helps to improve the efficiency of doctors in interpreting images and makes it easier to detect changes in some physiological structures.

[0098] Example 4:

[0099] Corresponding to the above method embodiments, this invention provides a vascular image processing device based on a neural network, see [link to relevant documentation]. Figure 9 The diagram shows a structural schematic of a neural network-based vascular image processing device, which includes:

[0100] The vascular image segmentation module 91 is used to acquire the vascular image of the eye under examination and segment the vascular image into multiple vascular sub-images based on the input image size of the denoising model. The vascular image of the eye under examination is generated by optical coherence tomography vascular imaging.

[0101] The denoising model processing module 92 is used to input multiple blood vessel sub-images into the trained denoising model and output multiple denoised blood vessel sub-images.

[0102] The denoised blood vessel sub-image fusion module 93 is used to obtain a fused denoised blood vessel image based on multiple denoised blood vessel sub-images.

[0103] This invention provides a neural network-based vascular image processing device that acquires vascular images of an eye being examined generated by optical coherence tomography (OCT) vascular imaging, segments the vascular image into multiple vascular sub-images based on the input image size of a denoising model, inputs the multiple vascular sub-images into a trained denoising model, outputs multiple denoised vascular sub-images, and obtains a fused denoised vascular image based on the multiple denoised vascular sub-images.

[0104] In this method, for some patients with eye diseases or who cannot cooperate with the doctor to maintain stable fixation, it can lead to blurred or distorted images. OCT examinations are also time-consuming, and some patients may find it difficult to tolerate the requirement of prolonged fixation, especially when experiencing eye discomfort or pain, which can also result in blurred or distorted images. For these patients, only protocols with fewer repeated acquisitions can be used, resulting in low image signal-to-noise ratios and frequent noise.

[0105] In this method, a first scanning protocol with a smaller data acquisition range parameter can be used to acquire the first training blood vessel image, avoiding the problem that the human eye cannot endure long-term acquisition. This allows the acquisition of a first training image with a higher signal-to-noise ratio, and the prediction result of the denoising model trained using the first training image also has a higher signal-to-noise ratio.

[0106] When using a denoising model for prediction, each segmented blood vessel sub-image can output a denoised blood vessel sub-image with a high signal-to-noise ratio as the prediction result. Therefore, the fused denoised blood vessel image obtained based on multiple denoised blood vessel sub-images also has a higher signal-to-noise ratio.

[0107] The aforementioned blood vessel image segmentation module is used to expand the four edges of a blood vessel image by pixels to obtain a blood vessel image to be segmented; to segment the blood vessel image to be segmented into multiple blood vessel sub-images; the size of the multiple blood vessel sub-images obtained after segmentation is equal to the size of the input image of the denoising model; the edges of two adjacent blood vessel sub-images overlap by a preset width. The aforementioned denoised blood vessel sub-image fusion module is used to obtain denoised blood vessel sub-images output by the denoising model; to shrink the four boundaries of each output denoised blood vessel sub-image inward by half a preset width to obtain a target denoised blood vessel sub-image; and to fuse multiple target denoised blood vessel sub-images to obtain a fused denoised blood vessel image.

[0108] The aforementioned apparatus further includes: a denoising model training module, configured to acquire a first training vascular image included in the training dataset using a first scanning protocol; process the data acquired by the first scanning protocol to determine a second training vascular image included in the training dataset; train the denoising model to be trained based on the training dataset to obtain a trained denoising model; wherein the first training vascular image serves as the label for training, and the second training vascular image serves as the input to the denoising model.

[0109] The aforementioned denoising model training module is also used to acquire vascular images of the examined eye using a second scanning protocol, wherein the data acquisition range parameter in the second scanning protocol is greater than the data acquisition range parameter in the first scanning protocol, and the number of repeated acquisitions parameter in the second scanning protocol is less than the number of repeated acquisitions parameter in the first scanning protocol.

[0110] The aforementioned denoising model training module is used to determine the scanning range, the area of ​​which is larger than the data acquisition range parameter in the first scanning protocol; to segment the scanning range to obtain multiple target data acquisition ranges, the area of ​​which is the same as the data acquisition range parameter in the first scanning protocol; and to scan the multiple target data acquisition ranges using the first scanning protocol to obtain multiple first training blood vessel images.

[0111] The aforementioned denoising model training module is used to perform at least one of downsampling, rotation, and translation processing on the data acquired by the first scanning protocol to determine the second training blood vessel images included in the training dataset.

[0112] The denoising model described above includes an initial feature extraction unit, a deep feature extraction unit, and a feature restoration unit. The feature restoration unit includes a convolution module and a residual module. The deep feature extraction unit includes multiple dilated convolution modules, each with a different dilation coefficient.

[0113] The aforementioned deep feature extraction unit further includes: a spatial attention module, which is a coordinate attention module; the coordinate attention module is used to perform pooling in the horizontal and vertical directions on the deep feature map to obtain aggregated features in the horizontal and vertical directions, and the deep feature map is a primary prediction image obtained by the convolution module and residual module in the deep feature extraction unit.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the neural network-based vascular image processing device described above can be referred to the corresponding process in the aforementioned embodiments of the neural network-based vascular image processing method, and will not be repeated here.

[0115] Example 5:

[0116] This invention also provides an electronic device for running the above-described neural network-based vascular image processing method; see also Figure 10 The diagram shows the structure of an electronic device, which includes a memory 100 and a processor 101. The memory 100 is used to store one or more computer instructions, which are executed by the processor 101 to implement the aforementioned neural network-based vascular image processing method.

[0117] Furthermore, Figure 10 The electronic device shown also includes a bus 102 and a communication interface 103, with the processor 101, the communication interface 103 and the memory 100 connected via the bus 102.

[0118] The memory 100 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 103 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network. The bus 102 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0119] Processor 101 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 101 or by instructions in software form. Processor 101 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 100, and processor 101 reads information from memory 100 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0120] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the aforementioned neural network-based vascular image processing method. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0121] The vascular image processing method and apparatus based on neural networks provided in this invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0122] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and / or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0123] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0124] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0126] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A neural network-based blood vessel image processing method, characterized by, The method comprises: Obtaining a blood vessel image of an eye to be examined, and dividing the blood vessel image into a plurality of blood vessel sub-images based on an input image size of a denoising model, wherein the blood vessel image of the eye to be examined is generated by optical coherence tomography blood vessel imaging; Inputting the plurality of blood vessel sub-images into the trained denoising model, and outputting a plurality of denoised blood vessel sub-images; Obtaining a fused denoised blood vessel image based on the plurality of denoised blood vessel sub-images; The method further comprises: obtaining a first training blood vessel image included in a training data set using a first scanning protocol; determining a second training blood vessel image included in the training data set by processing data obtained by the first scanning protocol; training the denoising model to be trained based on the training data set to obtain the trained denoising model; wherein the first training blood vessel image is used as a label for training, the second training blood vessel image is used as an input of the denoising model, and the number of scans when obtaining the first training blood vessel image is greater than the number of scans when obtaining the second training blood vessel image.

2. The method of claim 1, wherein, Dividing the blood vessel image into a plurality of blood vessel sub-images based on an input image size of a denoising model comprises: Pixel expansion is performed on the four edges of the blood vessel image to obtain a blood vessel image to be divided; The blood vessel image to be divided is divided to obtain a plurality of blood vessel sub-images; the size of the plurality of blood vessel sub-images obtained after division is equal to the size of the input image of the denoising model; and the edges of two adjacent blood vessel sub-images in the plurality of blood vessel sub-images overlap by a preset width.

3. The method of claim 2, wherein, Obtaining a fused denoised blood vessel image based on the plurality of denoised blood vessel sub-images comprises: Obtaining a denoised blood vessel sub-image output by the denoising model; Contraction of the half of the preset width from the four boundaries in each of the output denoised blood vessel sub-images is performed to obtain a target denoised blood vessel sub-image; The plurality of target denoised blood vessel sub-images are fused to obtain a fused denoised blood vessel image.

4. The method of claim 1, wherein, The method further comprises: Obtaining the blood vessel image of the eye to be examined using a second scanning protocol, wherein the data acquisition range parameter in the second scanning protocol is greater than the data acquisition range parameter in the first scanning protocol, and the number of repeated acquisitions in the second scanning protocol is less than the number of repeated acquisitions in the first scanning protocol.

5. The method of claim 4, wherein, Obtaining a first training blood vessel image included in a training data set using a first scanning protocol comprises: Determining a to-be-scanned range, wherein the area of the to-be-scanned range is greater than the data acquisition range parameter in the first scanning protocol; Dividing the to-be-scanned range to obtain a target data acquisition range at a plurality of positions, wherein the area of the target data acquisition range is the same as the data acquisition range parameter in the first scanning protocol; Scanning the plurality of target data acquisition ranges using the first scanning protocol to obtain a plurality of first training blood vessel images.

6. The method of claim 1, wherein, Determining a second training blood vessel image included in the training data set by processing data obtained by the first scanning protocol comprises: At least one of downsampling, rotation, and translation processing is performed on the data obtained by the first scanning protocol to determine the second training blood vessel image included in the training data set.

7. The method according to any one of claims 1 to 6, characterized in that, The denoising model comprises an initial feature extraction unit, a deep feature extraction unit and a feature restoration unit; the feature restoration unit comprises a convolution module and a residual module; the deep feature extraction unit comprises a plurality of dilated convolution modules, each dilated convolution module having a different dilated coefficient.

8. The method of claim 7, wherein, The deep feature extraction unit further comprises a spatial attention module, which is a coordinate attention module; the coordinate attention module is configured to perform pooling on a deep feature map in a horizontal coordinate direction and a vertical coordinate direction to obtain aggregated features in the horizontal coordinate direction and the vertical coordinate direction, the deep feature map being a preliminary prediction image obtained according to the convolution module and the residual module in the deep feature extraction unit.

9. A neural network-based blood vessel image processing apparatus, characterized by comprising: The device comprises: a blood vessel image segmentation module configured to obtain a blood vessel image of an eye under examination, segment the blood vessel image into a plurality of blood vessel sub-images based on an input image size of a denoising model, and generate the blood vessel image of the eye under examination by optical coherence tomography blood vessel imaging; a denoising model processing module configured to input the plurality of blood vessel sub-images into the trained denoising model and output a plurality of denoised blood vessel sub-images; a denoised blood vessel sub-image fusion module configured to obtain a fused denoised blood vessel image based on the plurality of denoised blood vessel sub-images; The device further comprises a denoising model training module configured to obtain a first training blood vessel image included in a training data set using a first scanning protocol, process data obtained by the first scanning protocol to determine a second training blood vessel image included in the training data set, train the denoising model to be trained based on the training data set, and obtain the trained denoising model; wherein the first training blood vessel image is used as a label for training, the second training blood vessel image is used as an input of the denoising model, the number of scans for obtaining the first training blood vessel image is greater than the number of scans for obtaining the second training blood vessel image.

Images