A method and apparatus for generating OCTA volumetric data based on hierarchical information

Abstract

This invention discloses a method and apparatus for generating OCTA volumetric data based on hierarchical information. The method acquires OCT volumetric data to be generated and decomposes the OCT volumetric data into a sequence containing multiple OCT images. This method and apparatus inputs the current OCT image and its left and right adjacent OCT images into a generation model to obtain an OCTA image corresponding to the current OCT image. Specifically, it fuses features from adjacent elements in the current OCT image to enhance the continuity and consistency of blood flow signals, thereby improving the detection capability of weak blood flow regions. Simultaneously, through an enhancement module, hierarchical information is embedded as a semantic prior into the generation process, enabling the deep learning network to perceive the context of different anatomical layers. This generates images that better reflect the actual vascular distribution characteristics, avoiding ineffective synthesis and effectively improving the quality of the synthesized images. Ultimately, this improves the quality of the OCTA volumetric data, allowing it to contain more blood flow information.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning networks and fundus angiography technology, specifically relating to a method and apparatus for generating OCTA volume data based on hierarchical information. Background Technology

[0002] Optical coherence tomography (OCTA) is a non-invasive, high-resolution vascular imaging method based on OCT (Optical Coherence Tomography) technology. This technology detects the movement of red blood cells within the blood vessel lumen by detecting changes in the backscattered signal of incident light by moving red blood cells, thus achieving three-dimensional imaging of the blood vessels. Compared to traditional fluorescein angiography, OCTA does not require the injection of exogenous contrast agents, avoiding risks such as patient allergies. In clinical applications, physicians typically use retinal layer contour information to segment OCTA volume data to visualize retinal and choroidal blood flow information, thereby effectively identifying fundus vascular abnormalities. This provides crucial auxiliary diagnostic evidence for various ophthalmic diseases such as diabetic retinopathy and age-related macular degeneration.

[0003] Despite the high clinical application potential of OCTA images, their practical application still has certain limitations. This is because OCTA imaging requires multiple scans of the same location to acquire blood flow information. This process significantly prolongs acquisition time, leading to motion artifacts and affecting image quality. To address these limitations, OCTA generation methods based on deep learning networks have been proposed. These methods require only a small number of OCT images to directly generate OCTA images, greatly improving imaging efficiency. Furthermore, many studies have shown that OCTA images generated using deep learning networks perform better. However, deep learning-based OCTA generation methods still face the following problems: First, extracting blood flow information from OCT images is difficult. Due to speckle noise and motion artifacts, the weak blood flow signals of low-velocity capillaries are easily submerged by noise, making it difficult for deep learning networks to accurately distinguish them. Second, deep learning networks cannot differentiate between areas with and without blood flow. Since the blood flow region typically occupies less than half of an OCT image, the lack of effective semantic guidance during training makes it difficult for the network model to accurately locate detailed blood flow information, further increasing the difficulty of blood flow extraction. These issues severely affect the reliability and structural accuracy of OCT images generated by deep learning networks. Summary of the Invention

[0004] The purpose of this invention is to address the problems raised in the background art by proposing an OCTA volume data generation method and apparatus based on hierarchical information.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] This invention proposes a method for generating OCTA volumetric data based on hierarchical information, comprising:

[0007] Obtain the OCT volume data to be generated as OCT volume data, and split the OCT volume data into a sequence containing multiple OCT images;

[0008] For each OCT image in the sequence, the current OCT image and its left and right adjacent OCT images are input into the trained generative model to obtain the OCT image corresponding to the current OCT image. For the first and last OCT images in the sequence, the current OCT image is directly copied as the missing adjacent OCT image.

[0009] All the obtained OCTA images are stitched together to obtain OCTA volume data;

[0010] The generative model includes an encoder, a decoder, and five fusion modules. The current OCT image and its left and right adjacent OCT images are used as inputs to the encoder. The encoder includes five stages connected in sequence. The first stage is a convolutional layer, and the second to fifth stages are downsampling stages. The decoder includes four upsampling enhancement fusion modules connected in sequence. The five stages in the encoder correspond one-to-one with the five fusion modules. The outputs of each stage in the encoder are input to the corresponding fusion modules. The output of the fusion module corresponding to the fifth stage of the encoder is used as the input to the decoder. The outputs of the fusion modules corresponding to the first four stages of the encoder are fused with the fourth, third, second, and first upsampling enhancement fusion modules in the decoder in sequence. The output of the fourth upsampling enhancement fusion module is then convolved to obtain the OCT image.

[0011] Preferably, each downsampling stage includes a convolutional layer, a batch normalization layer, an activation layer, and a max pooling layer connected in sequence.

[0012] Preferably, the output of each stage in the encoder consists of three features. The three features are concatenated to obtain a concatenated feature, which is then input into the corresponding fusion module. In the fusion module corresponding to the first four stages of the encoder, the concatenated feature is sequentially passed through the time transfer module and the convolutional layer to obtain the first feature. The first feature is then added to the current OCT image to obtain the output of the fusion module.

[0013] In the fusion module corresponding to the fifth stage of the encoder, the spliced ​​features are sequentially passed through the time transfer module and the self-attention mechanism module to obtain the second features. The second features are then added to the current OCT image to obtain the output of the fusion module.

[0014] In the self-attention mechanism module, the input of the self-attention mechanism module is passed through a convolutional layer to obtain the Q vector, K vector and V vector. The transpose of the K vector is multiplied by the Q vector and then passed through the Softmax function to obtain the third feature. The third feature is then multiplied by the V vector and then passed through a convolutional layer to obtain the output of the self-attention mechanism module.

[0015] Preferably, each of the upsampling enhancement fusion modules includes a parallel first upsampling module and a second upsampling module, as well as an enhancement module. The outputs of the first upsampling module and the second upsampling module serve as the two inputs of the enhancement module. For the first upsampling enhancement fusion module, the output of the fusion module corresponding to the fifth stage of the encoder serves as the input of the first upsampling module and the second upsampling module, respectively.

[0016] For the remaining upsampling enhancement fusion modules, the output of the enhancement module in the previous upsampling enhancement fusion module is used as the input of the first upsampling module in the current upsampling enhancement fusion module, and the output of the second upsampling module in the previous upsampling enhancement fusion module is used as the input of the second upsampling module in the current upsampling enhancement fusion module;

[0017] For the fourth upsampling enhancement fusion module, the output of the enhancement module is passed through a convolutional layer to obtain the OCTA image, and the output of the second upsampling module is passed through a convolutional layer to obtain the layered map, which is used to calculate the loss of the generation model.

[0018] Preferably, when the outputs of the fusion modules corresponding to the first four stages of the encoder are fused sequentially with the fourth, third, second, and first upsampling enhancement fusion modules in the decoder, they are fused with the first and second upsampling modules in each upsampling enhancement fusion module, respectively. In each upsampling module, the input of the upsampling module is bilinearly interpolated, concatenated with the output of the corresponding fusion module, and then sequentially passed through two convolutional layers to obtain the output of the upsampling module.

[0019] Preferably, each enhancement module includes a layer attention component and a high-frequency feature component connected in sequence. In the layer attention component, the two inputs of the enhancement module are respectively passed through a convolutional layer to obtain a fourth feature and a fifth feature. The fourth feature and the fifth feature are added together and then passed through a convolutional layer and an activation layer in sequence to obtain a sixth feature. The sixth feature is multiplied by the output of the corresponding second upsampling module to obtain a seventh feature. The seventh feature is then added by the output of the corresponding first upsampling module to obtain the output of the layer attention component.

[0020] The high-frequency feature component includes parallel first-scale pooling operations, second-scale pooling operations, and third-scale pooling operations, as well as first-scale convolutional layers, second-scale convolutional layers, and third-scale convolutional layers that correspond one-to-one with the scales of the first-scale pooling operations, second-scale pooling operations, and third-scale pooling operations, respectively. The output of the layer attention component is processed by the first-scale pooling operations, second-scale pooling operations, and third-scale pooling operations to obtain three high-frequency feature maps at different scales. The three high-frequency feature maps at different scales are added to the output of the layer attention component to obtain three features at different scales. These three features at different scales are then sequentially input into the first-scale convolutional layer, second-scale convolutional layer, and third-scale convolutional layer to obtain three output results at different scales. The three output results at different scales are then concatenated and processed by the convolutional layer to obtain the high-frequency feature output.

[0021] Preferably, each of the scale pooling operations includes a parallel max pooling layer and an average pooling layer, and the outputs of the max pooling layer and the average pooling layer are subtracted to obtain the output of the scale pooling operation.

[0022] Preferably, the loss function of the generative model is calculated using the following formula:

[0023] ;

[0024] in,

[0025] ;

[0026] ;

[0027] ;

[0028] in, The loss function for generating the model, This represents the loss used for OCTA images. This represents the loss used for the hierarchical graph. This represents the OCTA image predicted by the generative model. Represents the true label of the OCTA image. This represents a hierarchical graph representing the predictions of the generative model. Represents the true labels of the layered diagram. and These are the weights used for OCTA image loss and the weights used for hierarchical map loss, respectively. For intermediate parameters, For L1 loss of OCTA images, , and KL loss in the retina, choroid, and other parts of the OCTA image, respectively. For cross-entropy loss, For Dice's loss, , , , , , and All parameters are preset.

[0029] Preferably, during the training process of the generative model, in the first preset training rounds, In the remaining training rounds, .

[0030] An OCTA volumetric data generation device based on hierarchical information, the OCTA volumetric data generation device based on hierarchical information includes:

[0031] The first module is used to acquire the OCT volume data to be generated as OCT volume data, and to split the OCT volume data into a sequence containing multiple OCT images.

[0032] The second module is used to input the current OCT image and its left and right adjacent OCT images into the trained generative model for each OCT image in the sequence to obtain the OCTA image corresponding to the current OCT image. It is also used to directly copy the current OCT image as the missing adjacent OCT image for the first and last OCT images in the sequence.

[0033] The third module is used to stitch together all the obtained OCTA images to obtain OCTA volume data;

[0034] The generative model includes an encoder, a decoder, and five fusion modules. The current OCT image and its left and right adjacent OCT images are used as inputs to the encoder. The encoder includes five stages connected in sequence. The first stage is a convolutional layer, and the second to fifth stages are downsampling stages. The decoder includes four upsampling enhancement fusion modules connected in sequence. The five stages in the encoder correspond one-to-one with the five fusion modules. The outputs of each stage in the encoder are input to the corresponding fusion modules. The output of the fusion module corresponding to the fifth stage of the encoder is used as the input to the decoder. The outputs of the fusion modules corresponding to the first four stages of the encoder are fused with the fourth, third, second, and first upsampling enhancement fusion modules in the decoder in sequence. The output of the fourth upsampling enhancement fusion module is then convolved to obtain the OCT image.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] This method and apparatus for generating OCTA volumetric data based on hierarchical information inputs the current OCT image and its left and right adjacent OCT images into the generation model to obtain an OCTA image corresponding to the current OCT image. That is, it fuses the features of the left and right adjacent features in the current OCT image to enhance the continuity and consistency of blood flow signals, thereby improving the detection capability of weak blood flow areas. At the same time, through the enhancement module, hierarchical information is embedded as a semantic prior into the generation process, enabling the deep learning network to perceive the context of different anatomical layers, thereby generating images that are more consistent with the actual blood vessel distribution characteristics, avoiding invalid synthesis, effectively improving the quality of synthesized images, and thus improving the quality of OCTA volumetric data, enabling it to contain more blood flow information. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the OCTA volume data generation method and apparatus based on hierarchical information according to the present invention.

[0038] Figure 2 This is a structural block diagram of the model generated by the present invention;

[0039] Figure 3 This is a structural block diagram of the layer attention component of the present invention;

[0040] Figure 4 This is a structural block diagram of the high-frequency feature component of the present invention;

[0041] Figure 5 These are OCT B scan images of the fundus of the present invention;

[0042] Figure 6 This is the actual OCTA image corresponding to the fundus OCT B scan image of this invention;

[0043] Figure 7 The OCTA image is generated from the fundus OCT B scan image using the generation model of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0046] In one embodiment, such as Figures 1-7 As shown, a method for generating OCTA volumetric data based on hierarchical information is provided, including:

[0047] Step 1: Obtain the OCT volume data to be generated as OCT volume data, and split the OCT volume data into a sequence containing multiple OCT images;

[0048] It should be noted that the fundus OCT volume data to be generated is obtained by acquiring fundus OCT volume data, which consists of multiple fundus OCT images (B scan images), and the fundus OCT volume data is split into a sequence containing multiple fundus OCT images; Figure 5 This image shows a fundus OCT B scan. Figure 6 It is the actual OCTA image corresponding to a fundus OCT B scan image. Figure 7 It is an OCTA image generated from a fundus OCT B scan image using a generative model.

[0049] Step 2: For each OCT image in the sequence, input the current OCT image and its two adjacent OCT images to the trained generative model (that is, the process of generating an OCTA image for each OCT image requires the current OCT image and its two adjacent OCT images to be used as input to the generative model) to obtain the OCTA image corresponding to the current OCT image. For the first and last OCT images in the sequence, directly copy the current OCT image as the missing adjacent OCT image.

[0050] Step 2.1, as follows Figure 2 As shown, the generative model includes an encoder, a decoder, and five fusion modules. The current OCT image and its left and right adjacent OCT images are used as the input to the encoder. The encoder includes five stages connected in sequence (from data input to output direction). The first stage is a convolutional layer (3×3), and the second to fifth stages are downsampling stages. Each downsampling stage includes a convolutional layer, a batch normalization layer, an activation layer (ReLU activation function), and a max pooling layer connected in sequence (from data input to output direction). The encoder is used to extract and compress input features layer by layer, and the five stages in the encoder correspond one-to-one with the five fusion modules. The output of each stage in the encoder is input to the corresponding fusion module.

[0051] The output of each stage in the encoder consists of three features (all three features are of the same size). The three features are concatenated to obtain the concatenated feature, which is then input into the corresponding fusion module. In the fusion modules corresponding to the first four stages of the encoder, the concatenated feature is sequentially passed through a time transfer module (which performs a time transfer operation with a shift width of one-third of the channel, which is existing technology) and a convolutional layer (1×1) to obtain the first feature. The first feature is then added to the current OCT image to obtain the output of the fusion module (that is, the output of the fusion modules corresponding to the first four stages of the encoder (i.e., the first to the fourth stages) is the same).

[0052] In the fusion module corresponding to the fifth stage of the encoder, the spliced ​​features are sequentially processed through the time transfer module and the self-attention mechanism module to obtain the second features. The second features are then added to the current OCT image to obtain the output of the fusion module (the output of the fusion module corresponding to the fifth stage of the encoder).

[0053] In the self-attention mechanism module, the input of the self-attention mechanism module is passed through a convolutional layer (1×1) to obtain the Q vector, K vector and V vector. The transpose of the K vector is multiplied by the Q vector and then passed through the Softmax function to obtain the third feature. The third feature is then multiplied by the V vector and then passed through a convolutional layer (1×1) to obtain the output of the self-attention mechanism module.

[0054] Step 2.2: The decoder includes four upsampling enhancement fusion modules connected sequentially (from data input to output direction) (namely, the first upsampling enhancement fusion module, the second upsampling enhancement fusion module, the third upsampling enhancement fusion module, and the fourth upsampling enhancement fusion module). The output of the fusion module corresponding to the fifth stage of the encoder is used as the input of the decoder. The outputs of the fusion modules corresponding to the first four stages of the encoder (i.e., the first stage, the second stage, the third stage, and the fourth stage) are fused sequentially with the fourth upsampling enhancement fusion module, the third upsampling enhancement fusion module, the second upsampling enhancement fusion module, and the first upsampling enhancement fusion module in the decoder. The output of the fourth upsampling enhancement fusion module is then convolved to obtain the OCTA image.

[0055] Step 2.3: Each upsampling enhancement fusion module includes a parallel first upsampling module and a second upsampling module, as well as an enhancement module. The outputs of the first upsampling module and the second upsampling module serve as the two inputs of the enhancement module. For the first upsampling enhancement fusion module, the output of the fusion module corresponding to the fifth stage of the encoder serves as the input of the first upsampling module and the second upsampling module, respectively.

[0056] For the remaining upsampling enhancement fusion modules, the output of the enhancement module in the previous upsampling enhancement fusion module is used as the input of the first upsampling module in the current upsampling enhancement fusion module, and the output of the second upsampling module in the previous upsampling enhancement fusion module is used as the input of the second upsampling module in the current upsampling enhancement fusion module; the output of the enhancement module in the first upsampling enhancement fusion module is used as the input of the first upsampling module in the second upsampling enhancement fusion module, and the output of the second upsampling module in the first upsampling enhancement fusion module is used as the input of the second upsampling module in the second upsampling enhancement fusion module.

[0057] For the fourth upsampling enhancement fusion module, the output of the enhancement module is passed through a convolutional layer (3×3) to obtain the OCT image (i.e., the OCT image corresponding to the current OCT image). The output of the second upsampling module is passed through a convolutional layer (3×3) to obtain the layered map, and the layered map is used for loss calculation of the generation model.

[0058] Step 2.4: When the outputs of the fusion modules corresponding to the first four stages of the encoder are fused with the fourth, third, second, and first upsampling enhancement fusion modules in the decoder, they are fused with the first and second upsampling modules in each upsampling enhancement fusion module respectively. In each upsampling module, the input of the upsampling module is bilinearly interpolated and then concatenated with the output of the corresponding fusion module (i.e., the fusion modules corresponding to the first four stages of the encoder are sequentially fused with the fourth, third, second, and first upsampling enhancement fusion modules). Then, the output is passed through two convolutional layers to obtain the output of the upsampling module.

[0059] Step 2.5: Each enhancement module includes a layer attention component and a high-frequency feature component connected sequentially (from data input to output direction), such as... Figure 3 As shown, in the layer attention component, the two inputs of the enhancement module are passed through a convolutional layer (1×1) to obtain the fourth and fifth features. The fourth and fifth features are added together and then passed through a convolutional layer (1×1) and an activation layer (sigmoid activation function) to obtain the sixth feature. The sixth feature is multiplied by the output of the corresponding second upsampling module (i.e., the second upsampling module in the upsampling enhancement fusion module where the current enhancement module is located) to obtain the seventh feature. The seventh feature is then added by the output of the corresponding first upsampling module (i.e., the first upsampling module in the upsampling enhancement fusion module where the current enhancement module is located) to obtain the output of the layer attention component (which serves as the input of the high-frequency feature component).

[0060] like Figure 4As shown, the high-frequency feature component includes parallel first-scale pooling operations, second-scale pooling operations, and third-scale pooling operations (the scales of the first-scale pooling operations, second-scale pooling operations, and third-scale pooling operations are 3×3, 5×5, and 7×7, respectively), and first-scale convolutional layers, second-scale convolutional layers, and third-scale convolutional layers (the scales of the first-scale pooling operations, second-scale pooling operations, and third-scale pooling operations are 3×3, 5×5, and 7×7, respectively), which correspond one-to-one with the scales of the first-scale pooling operations, second-scale pooling operations, and third-scale pooling operations. The output of the layer attention component is processed through the first-scale pooling operation, second-scale pooling operation, and third-scale pooling operation to obtain three different scales. The high-frequency feature maps at three different scales are added to the output of the layer attention component to obtain three features at different scales. These three features (the eighth, ninth, and tenth features corresponding to the first, second, and third scale pooling operations, respectively) are then sequentially input into the first-scale convolutional layer, the second-scale convolutional layer, and the third-scale convolutional layer to obtain three output results at different scales. The three output results at different scales are then concatenated and passed through a convolutional layer (1×1) to obtain the output of the high-frequency features (i.e., the output of the enhancement module).

[0061] Each scale pooling operation includes parallel max pooling and average pooling layers (the size of the max pooling and average pooling layers is 3×3 in the first scale pooling operation, 5×5 in the second scale pooling operation, and 7×7 in the third scale pooling operation). The outputs of the max pooling and average pooling layers are subtracted to obtain the output of the scale pooling operation.

[0062] The formula for calculating the loss function of the generative model is as follows:

[0063] ;

[0064] in,

[0065] ;

[0066] ;

[0067] ;

[0068] in,

[0069] ;

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] ;

[0075] in, The loss function for generating the model, This represents the loss used for OCTA images. This represents the loss used for the hierarchical graph. This represents the OCTA image predicted by the generative model. Represents the true label of the OCTA image. This represents a hierarchical graph representing the predictions of the generative model. Represents the true labels of the layered diagram. and These are the weights used for OCTA image loss and the weights used for hierarchical map loss, respectively. For intermediate parameters, For L1 loss of OCTA images, , and KL loss in the retina, choroid, and other parts of the OCTA image, respectively. For cross-entropy loss, For Dice's loss, , , , , , and All are preset parameters (e.g.) , , , , , , ), , and These are represented, in order, the total number of pixels in the retina, the total number of pixels in the choroid, and the total number of pixels in other parts of the OCTA image. , and The following are the retinal images of the OCTA image, in sequence: The pixel, the first in the choroid The pixel and other parts 1 pixel, The retina in OCTA images The actual value of each pixel. The first retinal image in OCTA The predicted value of each pixel. The choroid in OCTA images The true value of each pixel The first choroid in the OCTA image The predicted value of each pixel. For other parts of the OCTA image The actual value of each pixel. For other parts of the OCTA image The predicted value of each pixel. Represented as the total number of pixels in the OCTA image. For the OCTA image of the first 1 pixel, For the OCTA image, the first The actual value of each pixel. For the OCTA image, the first The predicted value of each pixel. This represents the total number of pixels in the layered image. This refers to the number of retinal layers. No. Layer The real label of each pixel The Middle The pixel belongs to the first The probability of the layer, where .

[0076] During the training process of the generative model , The value is dynamic; in the initial preset training rounds, In the remaining training rounds, (like ,and During the early training rounds, the information in the layered images may be incorrect, misleading the OCTA image. Therefore, [the following settings are implemented]. This ensures that the model is trained only on the hierarchical graph, and after a preset number of training rounds, the information in the hierarchical graph will not be erroneous, allowing for further adjustments. This strategy aims to maintain the training effectiveness of the hierarchical map while shifting the optimization focus of the model to the OCTA image generation task, thereby achieving OCTA image generation by incorporating hierarchical information.

[0077] Step 3: Stitch together all the obtained OCTA images (that is, stitch together the OCTA images corresponding to each OCT image in the sequence) to obtain OCTA volume data; for example, the obtained... open OCTA images stitched together OCTA body data.

[0078] In another embodiment, based on a hierarchical information-based OCTA volumetric data generation method, an OCTA volumetric data generation apparatus based on hierarchical information is also proposed, comprising:

[0079] The first module is used to acquire the OCT volume data to be generated as OCT volume data, and to split the OCT volume data into a sequence containing multiple OCT images.

[0080] The second module is used to input the current OCT image and its left and right adjacent OCT images into the trained generative model for each OCT image in the sequence to obtain the OCTA image corresponding to the current OCT image. It is also used to directly copy the current OCT image as the missing adjacent OCT image for the first and last OCT images in the sequence.

[0081] The third module is used to stitch together all the obtained OCTA images to obtain OCTA volume data;

[0082] The generative model includes an encoder, a decoder, and five fusion modules. The current OCT image and its left and right adjacent OCT images are used as inputs to the encoder. The encoder consists of five stages connected in sequence. The first stage is a convolutional layer, and the second to fifth stages are downsampling stages. The decoder consists of four upsampling enhancement fusion modules connected in sequence. The five stages in the encoder correspond one-to-one with the five fusion modules. The outputs of each stage in the encoder are input to the corresponding fusion modules. The output of the fusion module corresponding to the fifth stage of the encoder is used as the input to the decoder. The outputs of the fusion modules corresponding to the first four stages of the encoder are fused with the fourth, third, second, and first upsampling enhancement fusion modules in the decoder in sequence. The output of the fourth upsampling enhancement fusion module is then convolved to obtain the OCT image.

[0083] For specific limitations on the OCTA volume data generation device based on hierarchical information, please refer to the limitations on the OCTA volume data generation method based on hierarchical information above, which will not be repeated here.

[0084] This method and apparatus for generating OCTA volumetric data based on hierarchical information inputs the current OCT image and its left and right adjacent OCT images into the generation model to obtain an OCTA image corresponding to the current OCT image. That is, it fuses the features of the left and right adjacent features in the current OCT image to enhance the continuity and consistency of blood flow signals, thereby improving the detection capability of weak blood flow areas. At the same time, through the enhancement module, hierarchical information is embedded as a semantic prior into the generation process, enabling the deep learning network to perceive the context of different anatomical layers, thereby generating images that are more consistent with the actual blood vessel distribution characteristics, avoiding invalid synthesis, effectively improving the quality of synthesized images, and thus improving the quality of OCTA volumetric data, enabling it to contain more blood flow information.

[0085] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0086] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for OCTA volume data generation based on layered information, characterized in that: The OCTA volume data generation method based on hierarchical information comprises: obtaining OCT volume data to be generated OCTA volume data, and splitting the OCT volume data into a sequence comprising multiple OCT images; for each OCT image in the sequence, inputting the current OCT image and its left and right adjacent OCT images into the trained generation model to obtain an OCTA image corresponding to the current OCT image, and for the first and last two OCT images in the sequence, directly copying the current OCT image as the missing adjacent OCT image; stitching all the obtained OCTA images to obtain the OCTA volume data; wherein the generation model comprises an encoder, a decoder and five fusion modules, the current OCT image and its left and right adjacent OCT images are respectively taken as inputs of the encoder, the encoder comprises five stages connected in turn, the first stage is a convolution layer, the second stage to the fifth stage are down-sampling stages, the decoder comprises four up-sampling enhancement fusion modules connected in turn, and the five stages in the encoder correspond to the five fusion modules one by one, the outputs of each stage in the encoder are input into the corresponding fusion module, the output of the fusion module corresponding to the fifth stage of the encoder is taken as the input of the decoder, the outputs of the fusion modules corresponding to the first four stages of the encoder are sequentially fused with the fourth, third, second and first up-sampling enhancement fusion modules in the decoder, and the output of the fourth up-sampling enhancement fusion module is obtained by convolution operation to obtain the OCTA image; wherein the output of each stage in the encoder is three features, after the stitching feature is obtained by stitching the three features, it is input into the corresponding fusion module, for the fusion module corresponding to the first four stages of the encoder, the stitching feature sequentially passes through a time transfer module and a convolution layer to obtain a first feature, and then the first feature is added to the current OCT image to obtain the output of the fusion module; for the fusion module corresponding to the fifth stage of the encoder, the stitching feature sequentially passes through a time transfer module and a self-attention mechanism module to obtain a second feature, and then the second feature is added to the current OCT image to obtain the output of the fusion module; wherein in the self-attention mechanism module, the input of the self-attention mechanism module passes through a convolution layer to obtain a Q vector, a K vector and a V vector, the transpose of the K vector is multiplied by the Q vector, and then the third feature is obtained by passing through a Softmax function, and then the third feature is multiplied by the V vector, and then the output of the self-attention mechanism module is obtained by passing through a convolution layer.

2. The method of claim 1, wherein: Each down-sampling stage comprises a convolution layer, a batch normalization layer, an activation layer and a maximum pooling layer connected in turn.

3. The method of claim 1, wherein: Each up-sampling enhancement fusion module comprises a first up-sampling module and a second up-sampling module connected in parallel, and an enhancement module, and the outputs of the first up-sampling module and the second up-sampling module are taken as two inputs of the enhancement module, for the first up-sampling enhancement fusion module, the output of the fusion module corresponding to the fifth stage of the encoder is taken as the input of the first up-sampling module and the second up-sampling module. For the rest of the up-sampling enhancement fusion module, the output of the enhancement module in the previous up-sampling enhancement fusion module is taken as the input of the first up-sampling module in the current up-sampling enhancement fusion module, and the output of the second up-sampling module in the previous up-sampling enhancement fusion module is taken as the input of the second up-sampling module in the current up-sampling enhancement fusion module; For the fourth up-sampling enhancement fusion module, the output of the enhancement module is obtained by a convolution layer to obtain the OCTA image, and the output of the second up-sampling module is obtained by a convolution layer to obtain the layered image, and the layered image is used for loss calculation of the generation model.

4. The method of claim 3, wherein: When the outputs of the fusion modules corresponding to the first four stages of the encoder are sequentially fused with the fourth up-sampling enhancement fusion module, the third up-sampling enhancement fusion module, the second up-sampling enhancement fusion module and the first up-sampling enhancement module in the decoder, they are respectively fused with the first up-sampling module and the second up-sampling module in each up-sampling enhancement fusion module, and in each up-sampling module, the input of the up-sampling module is subjected to bilinear interpolation, then spliced with the output of the corresponding fusion module, and then sequentially subjected to two convolution layers to obtain the output of the up-sampling module.

5. The method of claim 4, wherein: Each enhancement module includes a layer attention component and a high-frequency feature component connected in sequence, wherein in the layer attention component, two inputs of the enhancement module are subjected to convolution layers to obtain fourth and fifth features, the fourth and fifth features are added and then sequentially subjected to a convolution layer and an activation layer to obtain a sixth feature, the sixth feature is multiplied by the output of the corresponding second up-sampling module to obtain a seventh feature, and the seventh feature is added with the output of the corresponding first up-sampling module to obtain the output of the layer attention component; The high-frequency feature component includes a first scale pooling operation, a second scale pooling operation and a third scale pooling operation in parallel, and a first scale convolution layer, a second scale convolution layer and a third scale convolution layer corresponding to the scales of the first scale pooling operation, the second scale pooling operation and the third scale pooling operation respectively, the output of the layer attention component is subjected to the first scale pooling operation, the second scale pooling operation and the third scale pooling operation to obtain three high-frequency feature maps of different scales, the three high-frequency feature maps of different scales are added with the output of the layer attention component to obtain three features of different scales, and the three features of different scales are sequentially input into the first scale convolution layer, the second scale convolution layer and the third scale convolution layer to obtain three output results of different scales, and then the three output results of different scales are spliced and subjected to a convolution layer to obtain the output of the high-frequency feature.

6. The method of claim 5, wherein: Each scale pooling operation includes a maximum pooling layer and an average pooling layer in parallel, and the output results of the maximum pooling layer and the average pooling layer are subtracted to obtain the output of the scale pooling operation.

7. The method of claim 3, wherein: The calculation formula of the loss function of the generation model is as follows: ； wherein, ； ； ； wherein, is a loss function for the generative model, denotes a loss for the OCTA image, denotes a loss for the layering map, denotes the OCTA image predicted by the generative model, denotes the true label of the OCTA image, denotes the layering map predicted by the generative model, denotes the true label of the layering map, and are a weight for the OCTA image loss and a weight for the layering map loss, respectively, is an intermediate parameter, is an LI loss for the OCTA image, , and are KL losses for the retina, choroid and other parts in the OCTA image, respectively, is a cross-entropy loss, is a Dice loss, , , , , , and are all preset parameters.

8. The method of claim 7, wherein: In the generation model training process, in the preset training rounds, in the remaining training rounds, .

9. An OCTA volume data generation device based on layered information, characterized by, The OCTA volume data generation device based on layered information comprises: A first module for obtaining OCT volume data to be generated into OCTA volume data, and splitting the OCT volume data into a sequence containing multiple OCT images; A second module for obtaining layered information of the OCT volume data, and generating a layered image corresponding to each OCT image in the sequence according to the layered information; The second module is configured to input, for each OCT image in the sequence, the current OCT image and its left and right adjacent OCT images into the trained generation model to obtain an OCTA image corresponding to the current OCT image, and to directly copy the current OCT image as the missing adjacent OCT image for the first and last two OCT images in the sequence; The third module is configured to splice all the obtained OCTA images to obtain an OCTA volume data; The generation model comprises an encoder, a decoder and five fusion modules, the current OCT image and its left and right adjacent OCT images are respectively taken as inputs of the encoder, the encoder comprises five stages connected in sequence, the first stage is a convolution layer, the second stage to the fifth stage are down-sampling stages, the decoder comprises four up-sampling enhancement fusion modules connected in sequence, the five stages in the encoder correspond to the five fusion modules one by one, the outputs of the stages in the encoder are respectively input into the corresponding fusion modules, the output of the fusion module corresponding to the fifth stage of the encoder is taken as the input of the decoder, the outputs of the fusion modules corresponding to the first four stages of the encoder are sequentially fused with the fourth up-sampling enhancement fusion module, the third up-sampling enhancement fusion module, the second up-sampling enhancement fusion module and the first up-sampling enhancement fusion module in the decoder, and the output of the fourth up-sampling enhancement fusion module is obtained by convolution to obtain an OCTA image; The outputs of the stages in the encoder are all three features, the spliced features are input into the corresponding fusion modules, for the fusion modules corresponding to the first four stages of the encoder, the spliced features sequentially pass through a temporal transfer module and a convolution layer to obtain a first feature, and the first feature is added to the current OCT image to obtain the output of the fusion module; For the fusion module corresponding to the fifth stage of the encoder, the spliced features sequentially pass through a temporal transfer module and a self-attention mechanism module to obtain a second feature, and the second feature is added to the current OCT image to obtain the output of the fusion module; In the self-attention mechanism module, the input of the self-attention mechanism module passes through a convolution layer to obtain a Q vector, a K vector and a V vector, the transpose of the K vector is multiplied by the Q vector, and then the Softmax function is used to obtain a third feature, the third feature is multiplied by the V vector, and then the convolution layer is used to obtain the output of the self-attention mechanism module.

Images