Shadow recovery method, model training method, system, medium, product and equipment

Abstract

The invention belongs to the technical field of image processing. The invention provides a shadow recovery method, a model training method, a system, a medium, a product and equipment, and the method comprises the steps: carrying out the shadow recovery of a preprocessed OCT image through a pre-trained dual-stage auxiliary gradient model, and obtaining a shadow-recovered OCT image; the training process comprises the following steps: acquiring a plurality of OCT images as training images, preprocessing the training images, selecting training images with different shadow proportions, and expanding the training images with different shadow proportions to construct a training set; according to the training set, in a first-stage training process, training loss minimization is carried out in a mode of predicting a whole picture as a background; natural prior knowledge is used for guiding, and second-stage training is carried out in combination with an artificially synthesized training image shadow; according to the method, high-precision recovery of the shadow region is realized through a binary mask mechanism, and generation of wrong structure information is avoided.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a shadow restoration method, a model training method, a system, a medium, a product and a device. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] Optical coherence tomography is a technology that detects the interference intensity of the backscattered light signal of the sample to obtain the internal structural information of the sample. This technology has an axial resolution of micrometers or even submicrometers, can measure the thickness and structural changes of the retina, and is considered the gold standard for diagnosing eye diseases. However, due to the presence of blood vessels in the fundus, the incident light beam is absorbed by the blood flow in the blood vessels, resulting in a shadow area in the retinal B-Scan image, resulting in loss of structural information, inaccurate retinal stratification, and increased difficulty in observing tiny lesions at the bottom of the blood vessels, missing the opportunity to discover early eye diseases.

[0004] At present, there are two methods to remove blood vessel shadows and improve image quality: one is to remove or reduce shadow pixels in the shadow area by optimizing the hardware optical path. For example, by designing the hardware optical path system, changing the second channel, incorporating it into a typical ICI system, removing the self-coherent image in coherent imaging, and improving image quality; the other method is to restore the shadow area by processing the image, such as directly processing the image using image pixel statistical features or deep learning neural networks, using automatic numerical interpolation methods to compensate for retinal shadows, and denoising and restoring the shadow area based on the GAN model. All of the above methods fail to achieve the requirement of unchanged image structure in the retinal shadow restoration task, and will generate erroneous structural information or erase existing structural information. Summary of the invention

[0005] In order to address the deficiencies in the prior art, the present invention provides a shadow recovery method, a model training method, a system, a medium, a product and an apparatus, and designs a dual-stage OCT image shadow recovery model, which achieves high-precision recovery of shadow areas through a binary mask mechanism, thereby avoiding the generation of erroneous structural information.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides a shadow restoration method.

[0007] A shadow restoration method includes the following steps: Acquire an OCT image to be processed, and preprocess the OCT image; The pre-processed OCT image is subjected to shadow restoration using a pre-trained two-stage auxiliary gradient model to obtain an OCT image after shadow restoration; wherein the training of the two-stage auxiliary gradient model includes: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

[0008] As a further limitation of the first aspect of the present invention, selecting training images with different shadow ratios includes: The mean value of the matching area to be matched and the matching area of ​​the template area are tested, and the pixels of the template area and the pixels of the matching area are normalized. T The test determines the significant difference between the two at the data mean level; when T When the test confidence is greater than the set threshold, the difference between the pixels in the template area and the pixels in the area to be matched is calculated. ， If the difference value is less than a set threshold, it is determined that there is a shadow area in the to-be-matched area; when T When the test confidence is less than or equal to the set threshold, the cosine similarity between the template area and the area to be matched is taken as the adaptive adjustment factor, and the adaptive adjustment factor is used to adaptively correct the matching result.

[0009] As a further limitation of the first aspect of the present invention, expanding the training images according to different shadow ratios includes: expanding the images by affine transformation, perspective transformation, contrast adjustment and random occlusion to obtain a training set.

[0010] As a further limitation of the first aspect of the present invention, during the first stage of training, the loss function for: ,in, Indicates Focal-loss of auxiliary training head, Indicates Dice-loss of the auxiliary training head, For the Focal-loss of an auxiliary training head.

[0011] As a further limitation of the first aspect of the present invention, the first stage training process is used to train the OSSNet model, and the OSSNet model is used to perform multi-level feature joint segmentation on the training image.

[0012] As a further limitation of the first aspect of the present invention, during the second stage of training, the natural image with shadows and the training image with artificially synthesized shadows are used as input, and the loss function for: , ,in, N is the total batch size of each iteration, and Indicates Model output values ​​and label values ​​for batches, Indicates the number of auxiliary training heads, Represents the model loss of the auxiliary training branch corresponding to the auxiliary training recovery head.

[0013] As a further limitation of the first aspect of the present invention, the second stage training process is used to train the OSRNet model, and the OSRNet model is a shadow recovery model of the Encode-Decode architecture.

[0014] As a further limitation of the first aspect of the present invention, artificially synthesizing shadows to simulate the contour shape of the segmentation model mask: , , , ,in, To generate a set of contour points, To generate contour points, A is a random number to control the contour amplitude, f To control the random number of contour change speed, is the number of points, is a random initial center point, is the fluctuation radius; Different brightness is given to the generated area, and Gaussian blur is performed on the filled area to simulate the uneven brightness, including: , , where Fill areas for uneven shadows, is the Gaussian kernel, for The uniform binary mask image after the point set is filled, is the original image data, represents the image input to the model, Represents convolution.

[0015] In a second aspect, the present invention provides a shadow restoration system.

[0016] A shadow restoration system, comprising: An OCT image acquisition unit is configured to: acquire an OCT image to be processed and pre-process the OCT image; The shadow recovery unit is configured to: perform shadow recovery on the preprocessed OCT image using a pre-trained two-stage auxiliary gradient model to obtain an OCT image after shadow recovery; wherein the training of the two-stage auxiliary gradient model includes: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

[0017] In a third aspect, the present invention provides a two-stage auxiliary gradient model training method.

[0018] A dual-stage auxiliary gradient model training method, wherein the dual-stage auxiliary gradient model is used for performing OCT image shadow restoration, comprises the following process: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

[0019] In a fourth aspect, the present invention provides a dual-stage auxiliary gradient model training system.

[0020] A dual-stage auxiliary gradient model training system, wherein the dual-stage auxiliary gradient model is used for performing OCT image shadow restoration, comprising: The training set construction unit is configured to: obtain a plurality of OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct the training set according to the training images with different shadow ratios; The two-stage training unit is configured as follows: according to the training set, in the first stage training process, the training loss is minimized by predicting the entire image as the background; and the second stage training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

[0021] In a fifth aspect, the present invention provides a computer device, comprising: a processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the shadow recovery method described in the first aspect of the present invention or the dual-stage auxiliary gradient model training method described in the third aspect of the present invention is implemented.

[0022] In a sixth aspect, the present invention provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the shadow recovery method described in the first aspect of the present invention, or the two-stage auxiliary gradient model training method described in the third aspect of the present invention.

[0023] In the seventh aspect, the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the shadow recovery method described in the first aspect of the present invention, or the two-stage auxiliary gradient model training method described in the third aspect of the present invention.

[0024] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention innovatively designs a dual-stage eye B-Scan image shadow restoration model architecture, which achieves high-precision restoration of shadow areas through a binary mask mechanism, avoids the generation of erroneous structural information, and improves the accuracy of shadow restoration.

[0025] 2. This invention innovatively proposes a solution to the difficulty of model training in the process of small target area segmentation. By introducing the combined effect of Focal-loss and Dice-loss, the model's learning ability for positive samples is improved.

[0026] 3. The present invention innovatively proposes a training strategy for restoring eye B-Scan images. By adding prior knowledge of natural images, the model's shadow perception of B-Scan images is improved.

[0027] 4. Adaptive template matching is achieved through a few template areas in the numerous OCT images collected, so as to select images containing shadow areas.

[0028] 5. The present invention proposes an artificial synthesis method for simulating eye shadow areas, which constructs label pairs of OCT images by simulating eye shadows, thereby solving the problem of difficulty in obtaining label pairs of shadow areas of OCT images.

[0029] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0031] Figure 1 A flowchart of the training process of the dual-stage auxiliary gradient model provided in Example 1 of the present invention; Figure 2 The grayscale histogram of the shadow area provided in Example 1 of the present invention; wherein, Figure 2 (a) is the original image. Figure 2 (b) in the figure is the grayscale histogram at the first position. Figure 2 (c) in the figure is the grayscale histogram at the second position. Figure 2 (d) in the figure is the grayscale histogram at the third position; Figure 3 A schematic diagram of the structure of a dual-stage auxiliary gradient model provided in Example 1 of the present invention; Figure 4 A schematic diagram of the second stage training process provided by Embodiment 1 of the present invention; Figure 5 A schematic diagram of a shadow restoration system provided in Embodiment 2 of the present invention; Figure 6 A schematic diagram of a dual-stage auxiliary gradient model training system provided in Example 4 of the present invention; Figure 7 A schematic diagram of a computer device provided in Example 5 of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0033] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0034] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0035] Embodiment 1: This implementation proposes a shadow restoration method. The invention takes the positioning and restoration of retinal shadow areas as an example. The image is a B-Scan image of the retina. By using strategies such as segmentation masks, natural prior knowledge, artificial synthetic shadows, and programmable gradient information, high-precision restoration is achieved while keeping the original structure unchanged. It includes the following processes: Acquire an OCT image to be processed, and preprocess the OCT image; The pre-processed OCT image is subjected to shadow restoration using a pre-trained two-stage auxiliary gradient model to obtain an OCT image after shadow restoration; wherein the training of the two-stage auxiliary gradient model includes: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

[0036] More specifically, the two-stage auxiliary gradient model training method, such as Figure 1 As shown, K B-Scan images of the eye are collected, i=0; data set selection is performed, when there is a shadow area, the current image is saved, and it is determined whether i is greater than or equal to k, when there is no shadow area, it is determined whether i is greater than or equal to k; when i is less than k, i=i+1 is set, and the data set selection is returned to continue; when i is greater than or equal to k, the data set standard is segmented, a two-stage restoration network model is established, and a loss function is set, artificial synthetic shadows and natural image shadow data sets are introduced, model training is performed, and shadow restoration of eye B-Scan images is performed.

[0037] More specifically, the process includes the following: (1) B-Scan image preprocessing.

[0038] There is speckle noise in the B-Scan image of the eye, so a Gaussian filter is first used for smoothing to reduce the impact of speckle noise in the image on edge information. Secondly, in order to improve the robustness of the shadow recovery method, it is necessary to select B-Scan images with different shadow ratios. Due to the uneven thickness of blood vessels in the B-Scan image, the absorption degree of incident light is different, resulting in different shadow area sizes and pixel values.

[0039] In order to more efficiently find B-Scan images containing vascular shadows, this implementation proposes an adaptive threshold template matching method to find images of shadow areas with different proportions and different brightness levels. The calculation method is as follows: (1); (2); (3); Where: P for T Test confidence, w and h are the width and height of the image, is the mean value of pixel samples in the template area, is the mean value of pixel samples in the area to be matched, is the sample standard deviation of the template area, is the sample standard deviation of the area to be matched, is the number of pixels in the template area, is the number of pixels in the area to be matched, For the template area The pixel value of the point, is the pixel value of the area to be matched.

[0040] Specifically, we first perform a mean test on the matching area, and the pixel distribution patterns of different shadow areas have common patterns (such as Figure 2 The template pixels and the pixels in the area to be matched are normalized. T The test can determine the significance of the difference between the two at the data mean level. When the threshold is set, it is considered that the difference between the pixels in the template area and the pixels in the area to be matched is small. At this time, the difference between the pixels in the template and the area to be matched is calculated. V ,when V When it is less than the threshold, it is determined that there is a shadow in the area to be matched; when , take the template area T To match the area I Negative cosine similarity As an adaptive adjustment factor, when the brightness is inconsistent with the template, Adaptively correct the matching results to improve matching accuracy.

[0041] More specifically, the brightness difference between the template pixel and the area to be matched is Positively correlated, at this time It is also positively correlated with the brightness difference. When the fluctuation range of the pixel values ​​of the two is small, The first half of the value ( ) is smaller, combined The scaling effect V A smaller value improves the stability of small fluctuation matching; for the area to be matched with a slightly larger fluctuation range, Reducing by nonlinear mapping The influence of the first half of the value improves the robustness of the matching; for images with large differences Smaller scaling capacity. The value is mainly affected by the first half. A larger value can remove irrelevant images in time.

[0042] (2) Dataset preparation.

[0043] After obtaining B-Scan images with different shadow pixel area sizes, the original images need to be augmented to expand the data set, thereby increasing the generalization of the model. The diversity of training data is increased by affine transformation, perspective transformation, contrast adjustment, and random occlusion. The enhanced data set is annotated for the first-stage shadow area segmentation task, and the B-Scan shadow area is annotated using the auxiliary segmentation and annotation tool. The annotated data is used for B-Scan shadow area segmentation, and the segmented mask image can be used to locate the restored area in the second stage, which is used to restore the model constraint restoration area.

[0044] In the second stage of restoration, there is a problem that the original image label is difficult to obtain, that is, the shadow-free image of the bottom of the blood vessel in the B-Scan image cannot be directly obtained. The present invention proposes a random artificial shadow generation method by simulating the contour shape of the segmentation model mask, and the calculation formula is as follows: (4); (5); (6); (7); Where: To generate a set of contour points, To generate contour points, is a random number to control the contour amplitude, To control the random number of contour change speed, is the number of points, is a random initial center point, is the fluctuation radius. Different brightness is given to the generated area. In order to simulate the unevenness of brightness, Gaussian blur is performed on the filled area, as shown in the following formula: (8); (9); Where: Fill areas for uneven shadows, is the Gaussian kernel, for Uniform binary mask image after point set filling, is the original image data, represents the image input to the model, Represents convolution, and restores the eye B-Scan image based on the above artificial synthetic shadow training model.

[0045] (3) Model design.

[0046] Figure 3 Resnet_block represents the feature extraction module, which consists of 1*1 convolution (Conv ), Convolution ), batch normalization (Batchnormal) and activation function (Relu), using multiple auxiliary heads and multiple auxiliary decoding heads; SW-Transformer block represents a feature extraction module containing self-attention and cross-attention, which is composed of channel normalization (Layernormal), window attention mechanism (W-MSA), sliding window attention mechanism (SW-MSA) and multi-layer perceptron (MLP). Among them, Downsample is a downsampling operation, Upsample is an upsampling operation, contact is an overlay operation along the channel dimension, interpolate is an interpolation operation, flatten is a flattening operation, sum is a summation operation along the channel dimension, skip is a skip connection operation, x, x0~xn represent two-dimensional input feature map information, v, v0~vn represent the feature map data input during the encoding-decoding process, and are converted into feature vectors through the flatten operation for attention calculation.

[0047] Figure 3 In the first stage, a segmentation model with all levels of feature joints is constructed with Resnet50 as the backbone feature extraction network. During the forward propagation of the network, the original image is downsampled and channel expanded to pass the feature map into Resnet_block. Each Block contains BatchNorm normalization, feature extraction, data dimension reduction, residual connection and Relu activation. The entire backbone network undergoes three downsamplings in total during the forward propagation process to generate feature maps of three different scales. Each downsampling is to halve the width and height of the image, but the number of features along the channel direction is doubled, that is, the original input dimension (B, C, H, W) becomes (B, 2C, H / 2, W / 2), where B is the number of images passed into the network at one time, C is the number of feature map channels, and H and W represent the height and width of the image. The features from the three-level feature layer are fused through the feature fusion module, and the fused features are assigned to the auxiliary training head. The auxiliary heads of different scales are superimposed on the corresponding scale feature detection heads along the feature channels, thereby improving the propagation efficiency of the image reverse gradient during the training stage and increasing the model accuracy.

[0048] Through the segmentation task in the first stage, a binary mask area is obtained, and the mask is superimposed with the original image along the feature map channel as the input image of the second stage.

[0049] In the second stage, the model of the encoder-decode architecture is used to fuse global features by constructing multi-head self-attention and local self-attention to achieve interaction between pixels in the shadow area and pixels in the background area. In the encoding stage, the attention between pixels is calculated by stretching the multi-channel image into blocks into feature vectors, and the self-attention weight matrix and the cross-attention weight matrix are introduced to realize the parameter calculation of each level of features and different regions of a single image, and then feature vectors of different scales are generated through multiple downsampling. In the decoding stage, the model further realizes feature mining by combining the decoded feature vectors, and gradually decodes the attention information of the model in the self-attention weight matrix and the cross-attention weight to realize functional decoding by combining feature information of different scales. Similarly, the vector feature fusion module realizes information fusion between feature vectors by designing the tensor stretching dimension, and distributes the fused features to the auxiliary decoder structure. Auxiliary decoding heads of different scales and the corresponding scale feature decoder detection heads are superimposed on each other along the feature channel to realize bidirectional propagation of training gradients.

[0050] In the two-stage shadow restoration task, the first-stage model provides a high-precision binary mask for the second-stage restoration task by segmentation. Based on the mask mechanism, the ability of the restoration model is constrained, and the global pixel information is no longer changed. The pixels in the background area can be highly preserved, thereby maximizing the invariance of structural information in the retinal shadow restoration process.

[0051] (4) Training strategy.

[0052] In the first stage of training, due to the problem of small area segmentation, the model gradient disappears. The model minimizes the training loss by predicting the entire image as the background. In order to balance the huge imbalance between positive and negative samples (the number of positive samples accounts for about 5%, sometimes less than 1%), the combination of Focal-loss and Dice-loss is introduced to enhance the model's ability to learn positive sample areas. The loss function is as follows: (10); (11); (12); (13); Where: is the output value, is the label value, for , To prevent the denominator from being zero, , represents the cross entropy loss function, represents Focal-loss, represents Dice-loss, represents the Focal-loss of the i-th auxiliary training head, represents the Dice-loss of the i-th auxiliary training head, is the Focal-loss of the i-th auxiliary training head.

[0053] In the second stage of training, natural prior knowledge is used to guide the model, by learning the shadow distribution law in natural images ( Figure 4 As shown in Figure 2, the shadows in the B-Scan image are enhanced. Since the label file in the B-Scan image cannot be obtained, the artificial synthesis method in step 2 is needed to add shadows. The added shadow image is used as the model input, and the model training loss is As shown below: (14); (15); In the formula, is the total batch size of each iteration, and Indicates n Model output values ​​and label values ​​for batches, Indicates the number of auxiliary training heads, here it is 1, Represents the model loss of the auxiliary training branch corresponding to the auxiliary training recovery head.

[0054] Embodiment 2: like Figure 5 As shown, this implementation provides a shadow recovery system, including: An OCT image acquisition unit is configured to: acquire an OCT image to be processed and pre-process the OCT image; The shadow recovery unit is configured to: perform shadow recovery on the preprocessed OCT image using a pre-trained two-stage auxiliary gradient model to obtain an OCT image after shadow recovery; wherein the training of the two-stage auxiliary gradient model includes: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

[0055] It is understandable that the above-mentioned various units can be separately or all combined into one or several other units to constitute, or some (some) units therein can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units can be realized by one unit. In other embodiments of the present application, the system can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0056] According to another embodiment of the present application, the system described in this embodiment can be constructed, and the method of Example 1 of the present application can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0057] Embodiment 3: This implementation provides a dual-stage auxiliary gradient model training method, wherein the dual-stage auxiliary gradient model is used for OCT image shadow restoration, including the following process: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

[0058] For detailed details, please refer to the introduction of the training process in Example 1, which will not be repeated here.

[0059] Embodiment 4: like Figure 6As shown, this implementation provides a dual-stage auxiliary gradient model training system, wherein the dual-stage auxiliary gradient model is used for performing OCT image shadow restoration, including: The training set construction unit is configured to: obtain a plurality of OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct the training set according to the training images with different shadow ratios; The two-stage training unit is configured as follows: according to the training set, in the first stage training process, the training loss is minimized by predicting the entire image as the background; and the second stage training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

[0060] It is understandable that the above-mentioned various units can be separately or all combined into one or several other units to constitute, or some (some) units therein can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units can be realized by one unit. In other embodiments of the present application, the system can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0061] According to another embodiment of the present application, the system described in this embodiment can be constructed, and the method of Example 1 of the present application can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0062] Embodiment 5: like Figure 7 As shown, this implementation provides an electronic device, which includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 may be connected via a bus or other means.

[0063] Among them, the communication interface 1002 is used to receive and send data, the computer-readable storage medium 1003 can be stored in the memory of the electronic device, the computer-readable storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0064] Processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.

[0065] The processor 1001 is configured to perform the following shadow recovery process: Acquire an OCT image to be processed, and preprocess the OCT image; The pre-processed OCT image is subjected to shadow restoration using a pre-trained two-stage auxiliary gradient model to obtain an OCT image after shadow restoration; wherein the training of the two-stage auxiliary gradient model includes: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows; Alternatively, perform the following two-stage auxiliary gradient model training process: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

[0066] Embodiment 6: This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.

[0067] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.

[0068] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following shadow recovery process: Acquire an OCT image to be processed, and preprocess the OCT image; The pre-processed OCT image is subjected to shadow restoration using a pre-trained two-stage auxiliary gradient model to obtain an OCT image after shadow restoration; wherein the training of the two-stage auxiliary gradient model includes: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows; Alternatively, implement the following two-stage auxiliary gradient model training process: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

[0069] Embodiment 7: The present implementation provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the following shadow recovery process: Acquire an OCT image to be processed, and preprocess the OCT image; The pre-processed OCT image is subjected to shadow restoration using a pre-trained two-stage auxiliary gradient model to obtain an OCT image after shadow restoration; wherein the training of the two-stage auxiliary gradient model includes: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows; Alternatively, perform the following two-stage auxiliary gradient model training process: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

[0070] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0071] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer instruction can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (for example, coaxial cable, optical fiber, digital line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server, a data center, etc. that contains one or more available media integrated. Available media can be magnetic media (for example, floppy disk, hard disk, tape), optical media (for example, DVD), or semiconductor media (for example, solid-state drive (Solid State Disk, SSD)) and the like.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A shadow restoration method, characterized in that: The process includes: Acquire an OCT image to be processed, and preprocess the OCT image; The pre-processed OCT image is subjected to shadow restoration using a pre-trained two-stage auxiliary gradient model to obtain an OCT image after shadow restoration; wherein the training of the two-stage auxiliary gradient model includes: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

2. The shadow restoration method according to claim 1, characterized in that: Select training images with different shadow ratios, including: The mean value of the matching area to be matched and the matching area of ​​the template area are tested, and the pixels of the template area and the pixels of the matching area are normalized. T The test determines the significant difference between the two at the data mean level; when T When the test confidence is greater than the set threshold, the difference between the pixels in the template area and the pixels in the area to be matched is calculated. ， If the difference value is less than a set threshold, it is determined that there is a shadow area in the to-be-matched area; when T When the test confidence is less than or equal to the set threshold, the cosine similarity between the template area and the area to be matched is taken as the adaptive adjustment factor, and the adaptive adjustment factor is used to adaptively correct the matching result.

3. The shadow restoration method according to claim 1, characterized in that: The training images with different shadow ratios are expanded, including: expanding the images by affine transformation, perspective transformation, contrast adjustment and random occlusion to obtain a training set.

4. The shadow restoration method according to claim 1, characterized in that: During the first stage of training, the loss function for: ,in, Indicates Focal-loss of auxiliary training head, Indicates Dice-loss of the auxiliary training head, For the Focal-loss of an auxiliary training head.

5. The shadow restoration method according to claim 4, characterized in that: The first stage of the training process is used to train the OSSNet model, which is used to perform multi-level feature joint segmentation on the training images.

6. The shadow restoration method according to claim 1, characterized in that: In the second stage of training, natural images with shadows and training images with artificial shadows are used as input, and the loss function for: , ,in, is the total batch size of each iteration, and Indicates Model output values ​​and label values ​​for batches, Indicates the number of auxiliary training heads, represents the model loss of the auxiliary training branch corresponding to the auxiliary training recovery head, To prevent the denominator from being zero, .

7. The shadow restoration method according to claim 6, characterized in that: The second stage training process is used to train the OSRNet model, which is a shadow restoration model of the Encode-Decode architecture.

8. The shadow restoration method according to claim 6, characterized in that: Artificially synthesize shadows to mimic the contour shape of the segmentation model mask: , , , ,in, To generate a set of contour points, To generate contour points, is a random number to control the contour amplitude, To control the random number of contour change speed, is the number of points, is a random initial center point, is the fluctuation radius; Different brightness is given to the generated area, and Gaussian blur is performed on the filled area to simulate the uneven brightness, including: , , where Fill areas for uneven shadows, is the Gaussian kernel, for The uniform binary mask image after the point set is filled, is the original image data, represents the image input to the model, Represents convolution.

9. A shadow restoration system, characterized in that: include: An OCT image acquisition unit is configured to: acquire an OCT image to be processed and pre-process the OCT image; The shadow recovery unit is configured to: perform shadow recovery on the preprocessed OCT image using a pre-trained two-stage auxiliary gradient model to obtain an OCT image after shadow recovery; wherein the training of the two-stage auxiliary gradient model includes: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

10. A two-stage auxiliary gradient model training method, characterized in that: The dual-stage auxiliary gradient model is used to perform OCT image shadow restoration, including the following process: Acquire multiple OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct a training set based on the training images with different shadow ratios; According to the training set, in the first stage of training, the training loss is minimized by predicting the entire image as the background; the second stage of training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

11. The dual-stage auxiliary gradient model training method according to claim 10, characterized in that: Select training images with different shadow ratios, including: The mean value of the matching area to be matched and the matching area of ​​the template area are tested, and the pixels of the template area and the pixels of the matching area are normalized. T The test determines the significant difference between the two at the data mean level; when T When the test confidence is greater than the set threshold, the difference between the pixels in the template area and the pixels in the area to be matched is calculated. ， If the difference value is less than a set threshold, it is determined that there is a shadow area in the to-be-matched area; when T When the test confidence is less than or equal to the set threshold, the cosine similarity between the template area and the area to be matched is taken as the adaptive adjustment factor, and the adaptive adjustment factor is used to adaptively correct the matching result.

12. The dual-stage auxiliary gradient model training method according to claim 10, characterized in that: During the first stage of training, the loss function for: ,in, Indicates Focal-loss of auxiliary training head, Indicates Dice-loss of the auxiliary training head, For the Focal-loss of an auxiliary training head.

13. The dual-stage auxiliary gradient model training method according to claim 10, characterized in that: In the second stage of training, natural images with shadows and training images with artificial shadows are used as input, and the loss function for: , ,in, N is the total batch size of each iteration, and Indicates n Model output values ​​and label values ​​for batches, K Indicates the number of auxiliary training heads, represents the model loss of the auxiliary training branch corresponding to the auxiliary training recovery head, To prevent the denominator from being zero, .

14. The dual-stage auxiliary gradient model training method according to claim 13, characterized in that: Artificially synthesize shadows to mimic the contour shape of the segmentation model mask: , , , ,in, To generate a set of contour points, To generate contour points, A is a random number to control the contour amplitude, f To control the random number of contour change speed, is the number of points, is a random initial center point, is the fluctuation radius; Different brightness is given to the generated area, and Gaussian blur is performed on the filled area to simulate the uneven brightness, including: , , where Fill areas for uneven shadows, is the Gaussian kernel, , is the original image data, represents the image input to the model, Represents convolution.

15. A dual-stage auxiliary gradient model training system, characterized in that: The dual-stage auxiliary gradient model is used to perform OCT image shadow restoration, including: The training set construction unit is configured to: obtain a plurality of OCT images as training images, pre-process the training images, select training images with different shadow ratios, and expand and construct the training set according to the training images with different shadow ratios; The two-stage training unit is configured as follows: according to the training set, in the first stage training process, the training loss is minimized by predicting the entire image as the background; and the second stage training is conducted by using natural prior knowledge for guidance and combining artificially synthesized training image shadows.

16. A computer device, characterized in that: include: a processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the shadow recovery method according to any one of claims 1 to 8 is implemented; or the dual-stage auxiliary gradient model training method according to claims 10-14 is implemented.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the shadow restoration method as described in any one of claims 1 to 8; or, executing the dual-stage auxiliary gradient model training method as described in claims 10-14.

18. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the shadow restoration method as described in any one of claims 1 to 8; or, it implements the dual-stage auxiliary gradient model training method as described in claims 10-14.

Images