Methods, devices and terminals for detecting retinal optical coherence tomography images

Abstract

This invention discloses a method, device, terminal, and medium for detecting retinal optical coherence tomography images. It obtains multiple image features at different scales through a convolutional neural network, which are used to extract local classification labels and global classification labels through a Transformer layer. In the Transformer layer, a cross-attention module is used to exchange feature information between image features at different scales to compensate for the shortcomings of the Transformer model in extracting local features. Then, the global classification labels and local classification labels are fused, that is, the local information in the image is further combined with the global information to obtain accurate recognition and classification results.

Description

Technical Field

[0001] This invention relates to the field of retinal image detection technology, and in particular to a method, apparatus and terminal for detecting retinal optical coherence tomography images. Background Technology

[0002] The number of people with myopia, especially those with high myopia, is gradually increasing. The harm of high myopia, especially pathological myopia, mainly lies in the visual impairment caused by its complications, such as choroidal retinal atrophy, choroidal neovascularization in the macula, and macular holes.

[0003] Optical coherence tomography (OCT) images of the retina can reveal the microscopic morphology and structure of retinal tissues. However, because the proportion of meaningless background areas in OCT images is relatively large, while some instructive characterization areas are relatively small, direct observation of OCT images can easily lead to misjudgments.

[0004] Currently, there are techniques that use pre-trained convolutional neural networks to predict the equivalent spherical power of horizontally scanned OCT images, which can help clinicians avoid overlooking risks related to ocular refractive errors during OCT evaluation. However, existing network models suffer from low recognition accuracy and inaccurate classification when recognizing OCT images.

[0005] Therefore, existing technologies need to be improved and enhanced. Summary of the Invention

[0006] The main objective of this invention is to provide a method, apparatus, smart terminal, and storage medium for detecting retinal optical coherence tomography (OCT) images, which can accurately identify and classify retinal OCT images.

[0007] To achieve the above objectives, a first aspect of the present invention provides a method for detecting retinal optical coherence tomography images, the method comprising:

[0008] The optical coherence tomography (OCT) images of the retina are input into a convolutional neural network to obtain image features at different scales. The image features are then grouped into a global group containing two of the image features and a local group containing one of the image features. The number of image features is at least three.

[0009] The global group is input into the Transformer layer to obtain the global classification label output by the Transformer layer. The Transformer layer includes a cross attention module, which is used to exchange information between two input image features.

[0010] Feature extraction is performed on the local groups to obtain local classification labels;

[0011] All the global classification labels are fused with the local classification labels to obtain the fused classification labels;

[0012] Based on the fused classification labels, the detection results of the retinal optical coherence tomography image are obtained.

[0013] Optionally, the image features are five, and the image features are grouped to obtain two global groups and one local group. There are two Transformer layers, and each global group is input to one Transformer layer.

[0014] Optionally, the step of inputting the global group into the Transformer layer to obtain the global classification label output by the Transformer layer includes:

[0015] Each image feature in the global group is input into a first Transformer encoder to obtain a first feature sequence corresponding to the image feature. The first feature sequence includes a classification label and a patch label.

[0016] Based on the cross-attention module, the classification label of each first feature sequence in the global group is sequentially fused with the patch label of another first feature sequence and the first feature sequence corresponding to the classification label is updated.

[0017] Each of the first feature sequences in the global group is input into a second Transformer encoder to obtain a second feature sequence;

[0018] By concatenating all the classification labels in the second feature sequence, the global classification label is obtained.

[0019] Optionally, the step of fusing the classification label of each first feature sequence in the global group with the patch label of another first feature sequence based on the cross-attention module and updating the first feature sequence corresponding to the classification label includes:

[0020] The classification label is processed according to the first dimension aligned linear projection function, and the processed classification label is fused with the patch label to obtain the first fused feature sequence;

[0021] The first fused feature sequence is input into the cross-attention encoder to obtain the attention score-weighted feature sequence.

[0022] The feature sequence weighted by the classification label and the attention score is input into the second dimension aligned linear projection function to obtain the processed classification label. The second dimension aligned linear projection function matches the first dimension aligned linear projection function.

[0023] The processed classification label is fused with the patch label to obtain a second fused feature sequence;

[0024] Update the first feature sequence corresponding to the classification label based on the second fused feature sequence.

[0025] Optionally, the cross-attention encoder employs a multi-head attention mechanism, and the expression of the cross-attention encoder is:

[0026]

[0027] in, Let C represent the learnable parameters in the network, and let C and h represent the embedding feature dimension and the number of heads in the Transformer encoder, respectively. The classification label in the fused feature sequence, The fused feature sequence The feature sequence after attention score weighting.

[0028] Optionally, the classification label and the feature sequence weighted by the attention score are input into a second-dimensional aligned linear projection function to obtain the processed classification label, including:

[0029] The attention-score-weighted feature sequence is processed according to the second-dimensional aligned linear projection function to obtain the processing result;

[0030] The processing result is vector-added with the classification label to obtain the processed classification label.

[0031] Optionally, after obtaining the detection results of the retinal optical coherence tomography image based on the fused classification labels, the method further includes:

[0032] Retinal optical coherence tomography images of the target patient in the horizontal and vertical scanning directions, respectively, are acquired in accordance with the detection results.

[0033] The step described above involves inputting the retinal optical coherence tomography image into a convolutional neural network until a first fused classification label corresponding to the horizontal scanning direction and a second fused classification label corresponding to the vertical scanning direction are obtained, respectively.

[0034] The first fused classification label and the second fused classification label are concatenated and classified to obtain the classification result.

[0035] A second aspect of the present invention provides a retinal optical coherence tomography image detection apparatus, wherein the apparatus comprises:

[0036] The image feature acquisition module is used to input retinal optical coherence tomography images into a convolutional neural network to obtain image features at different scales and group the image features into a global group containing two of the image features and a local group containing one of the image features, wherein the image features are at least three.

[0037] The global classification label extraction module inputs the global group into the Transformer layer to obtain the global classification label output by the Transformer layer. The Transformer layer includes a cross attention module, which is used to exchange information between two input image features.

[0038] The local classification label extraction module is used to extract features from the local groups to obtain local classification labels;

[0039] The fusion module is used to fuse all the global classification labels with the local classification labels to obtain fused classification labels;

[0040] The classification module is used to obtain the detection results of the retinal optical coherence tomography image based on the fused classification labels.

[0041] A third aspect of the present invention provides a smart terminal, the smart terminal including a memory, a processor, and a retinal optical coherence tomography (OCT) image detection program stored in the memory and executable on the processor, wherein the retinal OCT image detection program, when executed by the processor, implements any of the steps of the retinal OCT image detection method.

[0042] A fourth aspect of the present invention provides a computer-readable storage medium storing a retinal optical coherence tomography (OCT) image detection program, wherein the OCT image detection program, when executed by a processor, implements any of the steps of the aforementioned retinal optical coherence tomography image detection method.

[0043] As can be seen from the above, compared with the prior art, the present invention first obtains multiple image features at different scales through a convolutional neural network, which are used to extract local classification labels and global classification labels through a Transformer layer. In the Transformer layer, a cross-attention module is used to exchange feature information between image features at different scales to make up for the shortcomings of the Transformer model in extracting local features. Then, the global classification label and the local classification label are fused, that is, the local information in the image is further combined with the global information to obtain accurate recognition and classification results. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic flowchart of the retinal optical coherence tomography image detection method provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the module architecture of an embodiment of the present invention;

[0047] Figure 3 yes Figure 1 A schematic diagram of the Transformer layer architecture in this embodiment;

[0048] Figure 4 yes Figure 1 A detailed flowchart of step S200 in the embodiment is shown below;

[0049] Figure 5 yes Figure 4 A detailed flowchart of step S220 in the embodiment is shown below;

[0050] Figure 6 yes Figure 1 A schematic diagram of the cross-attention module in the embodiment;

[0051] Figure 7 This is a schematic diagram of the structure of the retinal optical coherence tomography image detection device provided in an embodiment of the present invention;

[0052] Figure 8 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0053] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0054] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0055] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0056] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0057] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0060] Automatic identification and accurate classification of retinal optical coherence tomography (OCT) images can help ophthalmologists perform OCT assessments, reducing the risk of misjudgments due to ignoring representational information in the images. However, current network models suffer from low accuracy and inaccurate classification when identifying OCT images, thus necessitating a method that can accurately classify and identify OCT images.

[0061] In recent years, following the significant success of the Transformer network architecture in natural language processing, the Vision Transformer has also emerged for image processing. Compared to Convolutional Neural Networks (CNNs), which focus only on local features, Transformers can capture long-range features, meaning they can easily capture global information. However, due to the lack of some inductive biases inherent in CNNs, such as translation equivalence and locality, existing Transformers do not generalize well on small datasets and lack the ability to extract local information, resulting in a performance gap compared to existing CNNs.

[0062] Therefore, based on the technical concept of combining Transformer with Convolutional Neural Network (CNN) to introduce locality into the traditional Transformer, this invention first obtains image features from the image through a CNN and then inputs them into a Transformer. The encoder in the Transformer is then applied to the image features at different scales extracted from the CNN to enhance the ability to handle long-range dependencies. Furthermore, the local features obtained from the CNN are fused with the global features obtained from the Transformer. This effective fusion of the local features from the CNN and the global features from the Transformer compensates for the shortcomings of these two architectures, resulting in more accurate classification results.

[0063] Exemplary methods

[0064] like Figure 1 As shown, the retinal optical coherence tomography image detection method of this embodiment specifically includes the following steps:

[0065] Step S100: Input the retinal optical coherence tomography image into a convolutional neural network to obtain image features at different scales and group the image features into a global group containing two image features and a local group containing one image feature, wherein the image features are at least three.

[0066] Specifically, the convolutional neural network extracts features from retinal optical coherence tomography images multiple times at different scales to obtain image features at different scales. Here, scale refers to different lengths, widths, and number of channels of the image, such as... Figure 2 As shown, we can assume that scale 1 is 1024*1024*3, scale 2 is 512*512*12, scale 3 is 256*256*24, and so on. The convolutional neural network is preferably the ResNet-50 network (residual network) used in this embodiment. ResNet (residual network) has been widely used in various feature extraction applications. Theoretically, the deeper the deep learning network, the stronger its expressive power; however, further deepening of a CNN network after reaching a certain depth does not improve classification performance, but rather leads to slower network convergence and decreased accuracy. Even if the dataset is increased to address overfitting, classification performance and accuracy will not improve.

[0067] After obtaining image features at different scales, the image features are grouped into one or more global groups containing two image features and at least one local group containing one image feature. In this invention, at least three image features are obtained from the convolutional neural network, which can then be grouped into a global group and a local group for extracting global and local features, respectively.

[0068] In this embodiment, five image features are extracted from the convolutional neural network. After grouping the image features, the following results are obtained: Figure 2 The diagram shows two global groups (F1 and F2 form one global group, F3 and F4 form another) and one local group (F5). Correspondingly, the Transformer layer ( Figure 2 The MST Block in the system also has two, so that each global group can input a Transformer layer.

[0069] It should be noted that the specific number of image features at different scales in this invention can be more than five, and the number of local groups can also be multiple. The specific number and grouping can be configured according to the processing efficiency and accuracy requirements.

[0070] Step S200: Input the global group into the Transformer layer to obtain the global classification label output by the Transformer layer. The Transformer layer includes a cross attention module, which is used to exchange information between two input image features.

[0071] Specifically, because the Transformer lacks the ability to extract local information, while the image features output by the convolutional neural network have already acquired local information from the OCT image, inputting them into the Transformer layer allows the Transformer to extract global information more accurately based on the input local information, thus obtaining the global classification label output by the Transformer layer. Furthermore, as... Figure 3 As shown, in this invention, the input to the Transformer layer is not a single image feature, but two image features of different scales are input together as a global group into the Transformer layer, and a cross-attention model is embedded in the Transformer layer. Through this model, information exchange is realized between image features of different scales, that is, the exchange of abstract information representations of images learned in the convolutional neural network. This can more effectively fuse image features of different scales to improve their feature representation ability and obtain a global classification label that can accurately express image information.

[0072] Step S300: Extract features from local groups to obtain local classification labels;

[0073] Specifically, image features from local groups are input into a pooling layer for pooling operations, and local classification labels are obtained after feature extraction. The pooling operation can be any of the commonly used pooling operations in this field, such as the average pooling operation used in this embodiment and the conventional max pooling operation.

[0074] Step S400: Merge all global classification labels with local classification labels to obtain merged classification labels;

[0075] Specifically, fusing the global and local classification labels output by all Transformer layers enhances the information representation capability of OCT images and yields better-performing fused classification labels. The fusion operation can be performed by simply concatenating the global and local classification labels as shown in this embodiment, or by using projection methods or other techniques employed in this field.

[0076] Step S500: Based on the fused classification labels, obtain the detection results of the retinal optical coherence tomography image.

[0077] Specifically, by inputting the fused classification labels into a fully connected layer, the detection results of the retinal optical coherence tomography image can be obtained. These results represent the probabilities of various classifications, such as normal, non-pathological high myopia, pathological high myopia, and other eye diseases, to help ophthalmologists conduct further analysis.

[0078] In summary, this embodiment obtains multiple image features at different scales through a residual network, which are used to extract local classification labels and global classification labels through a Transformer layer. A cross-attention module is used in the Transformer layer to exchange feature information between image features at different scales to improve the information representation ability of each image feature. Then, the global classification labels and local classification labels are fused, that is, the local information is combined with the global information, so as to effectively focus on meaningful lesion areas in OCT images and obtain accurate identification and classification results.

[0079] In one embodiment, such as Figure 4 As shown, step S200 above, which inputs the global group into the Transformer layer to obtain the global classification label output by the Transformer layer, more specifically includes the following steps:

[0080] Step S210: Input each image feature in the global group into a first Transformer encoder to obtain a first feature sequence corresponding to the image feature, wherein the first feature sequence includes a classification label and a patch label;

[0081] Specifically, the input image features are first linearly projected, and then the input image features are divided into a series of patch markers of a specified size through convolution operation to obtain a patch marker sequence. An additional classification marker is added to the head of the patch marker sequence to form a complete feature sequence. Then, the feature sequence is input into the first Transformer encoder to obtain the first feature sequence.

[0082] Because the visual Transformer inputs image chunks into the Transformer and flattens these chunks into vectors for processing, the internal structural information of the image chunks is destroyed, and the unique properties of the image are ignored. Therefore, this invention differs from the visual Transformer model in that it segments the input image features, while the visual Transformer model directly segments the input image.

[0083] Step S220: Based on the cross-attention module, the classification label of each first feature sequence in the global group is fused with the patch label of another first feature sequence in turn, and the first feature sequence corresponding to the classification label is updated.

[0084] Specifically, the global group includes two image features, and correspondingly, two first feature sequences are obtained in step S210. Then, the classification label of one of the first feature sequences is fused with the patch label of the other first feature sequence using a cross-attention module, and the first feature sequence corresponding to the classification label is updated based on the result. That is, firstly, the classification label of first feature sequence A is fused with the patch label of first feature sequence B to update first feature sequence A; then, the classification label of first feature sequence B is fused with the patch label of first feature sequence A to update first feature sequence B.

[0085] By using class labels in the first feature sequences at different scales as information media, and exchanging the learned image abstract information representations in the first feature sequences through embedded cross-attention modules, the information representation between different patch labels is enriched, thereby improving the feature representation capability of each first feature sequence.

[0086] Step S230: Input each first feature sequence in the global group into a second Transformer encoder to obtain the second feature sequence;

[0087] Step S240: Concatenate all the classification labels in the second feature sequences to obtain the global classification label.

[0088] Specifically, each first feature sequence after information exchange in step S220 is input into a second Transformer encoder to obtain the second feature sequence output by each second Transformer encoder. At this time, the class labels in the feature sequences output by the second Transformer encoder have learned the abstract information representation in the patch features. Then, the classification labels in the two second feature sequences are concatenated to obtain the global classification label.

[0089] As described above, this embodiment not only inputs the image features with acquired local information into the Transformer layer, but also uses a cross-attention module to exchange feature information between image features of different scales to improve the feature representation ability of each image feature, making the output global classification label more accurate.

[0090] In one embodiment, such as Figure 5 As shown, step S220 above specifically includes the following steps:

[0091] Step S221: Process the classification labels according to the first dimension aligned linear projection function and fuse the processed classification labels with the patch labels to obtain the first fused feature sequence;

[0092] Specifically, since the scales of the two first feature sequences in the global group are different, which is equivalent to the dimensionality of feature sequences at different scales being different, it is necessary to unify the dimensions before forming a new feature sequence. For example... Figure 6 As shown, for the high-dimensional first feature sequence (hereinafter referred to as the high-dimensional feature sequence), its classification label needs to be fused with the patch label of the low-dimensional first feature sequence (hereinafter referred to as the low-dimensional feature sequence) to form the first fused feature sequence. Similarly, a similar operation is performed on the low-dimensional feature sequence. The specific expression for the fusion operation is: in, It is a first-dimensional aligned linear projection function. , These are the classification labels for high-dimensional feature sequences and low-dimensional feature sequences, respectively. , These are patch labels for low-dimensional feature sequences and patch labels for high-dimensional feature sequences, respectively. , These are the high-dimensional fused feature sequences and the low-dimensional fused feature sequences, respectively.

[0093] Step S222: Input the first fused feature sequence into the cross-attention encoder to obtain the attention score-weighted feature sequence;

[0094] Specifically, taking a high-dimensional fused feature sequence as an example, the high-dimensional fused feature sequence is input into a cross-attention encoder, and a multi-head attention mechanism is used to obtain the attention score-weighted feature sequence. This means the classification labels in the high-dimensional fused feature sequence are... As a high-dimensional fused feature sequence in multi-head attention mechanisms The only query.

[0095] The expression for the cross-attention encoder is:

[0096]

[0097] in, Let C represent the learnable parameters in the network, and let C and h represent the embedding feature dimension and the number of heads in the Transformer encoder, respectively. The classification label in the fused feature sequence, The fused feature sequence The feature sequence is weighted by the attention score. For high-dimensional feature sequences, Represented as , Represented as For low-dimensional feature sequences, Represented as , Represented as .

[0098] Step S223: Input the feature sequence after weighting the classification label and attention score into the second dimension aligned linear projection function to obtain the processed classification label. The second dimension aligned linear projection function matches the first dimension aligned linear projection function.

[0099] Specifically, firstly, the linear projection function is aligned according to the second dimension. Processing attention-weighted feature sequences The processing result is obtained; then the processing result is vector-added with the classification label to obtain the processed classification label.

[0100] For high-dimensional feature sequences, the specific expression is: ,in The classification label corresponding to the high-dimensional feature sequence. This refers to the processed classification label corresponding to the high-dimensional feature sequence.

[0101] For low-dimensional feature sequences, the specific expression is: ,in The classification label corresponding to the low-dimensional feature sequence. The processed classification label corresponds to the low-dimensional feature sequence.

[0102] in, Is with A matching dimension-aligned linear projection function is used to ensure that the dimension of the output of IA is consistent with the dimension of the original high-dimensional feature sequence; Representation layer normalization.

[0103] Step S224: Fuse the processed classification label with the patch label to obtain the second fused feature sequence;

[0104] Step S225: Update the first feature sequence corresponding to the classification label based on the second fused feature sequence.

[0105] Specifically, for high-dimensional feature sequences, the specific expression for fusion is: ,in The processed classification labels corresponding to the high-dimensional feature sequences The patch markers are the corresponding low-dimensional feature sequences. This is the second fused feature sequence corresponding to the high-dimensional feature sequence. Similarly, for the low-dimensional feature sequence, the specific expression is: ,in The processed classification labels corresponding to the low-dimensional feature sequences For patch markers corresponding to high-dimensional feature sequences, This is the second fused feature sequence corresponding to the low-dimensional feature sequence.

[0106] As described above, the cross-attention model in this embodiment employs multi-head attention and does not use the feedforward network found in conventional Transformer encoders. By using class labels in image feature sequences at different scales as information media, and by exchanging the learned abstract information representations of the images, the information representation between different patch labels can be enriched, thereby improving the feature representation capability of each first feature sequence.

[0107] In one embodiment, progressive classification of OCT images is also implemented. Specific steps include: after obtaining the detection results of the retinal optical coherence tomography (OCT) images, the first stage of screening is completed, obtaining classification probability results for four categories: normal, high myopia without pathological changes, pathological high myopia, and other eye diseases. Then, the second stage further classifies and judges pathological high myopia. For example... Figure 2 As shown, for patients with pathological high myopia identified in the first stage of screening, OCT images of the patients in the horizontal and vertical scanning directions are obtained respectively. Then, steps S100 to S400 are repeated for these two images to obtain the fused classification label corresponding to each image. Finally, all the fused classification labels are stitched together and classified through a fully connected layer to obtain the classification probability result.

[0108] To address the significant differences in the location, shape, and size of lesion structures on images from different scanning orientations, the accuracy of classification is increased by fusing features from dual-view OCT images. This can assist doctors in determining whether patients have pathological changes such as retinal schisis, macular holes, retinal detachment, and PMCNV.

[0109] To better illustrate the classification accuracy and effectiveness of this invention, a total of 9984 qualified OCT images compiled by Zhongshan Ophthalmic Center of Sun Yat-sen University were collected. These images included four categories: normal high myopia, high myopia without pathological changes, pathological high myopia, and other eye diseases. Progressive classification tasks were implemented on this dataset, namely, a high myopia initial screening task, and a classification task targeting the pathological changes in high myopia such as retinal schisis, macular hole, retinal detachment, and PMCNV.

[0110] Experimental results show that the classification accuracy reached 93.91% in the initial screening task for high myopia. Experimental results in the classification task for pathological changes in high myopia indicate that fusing OCT image features from both horizontal and vertical scanning directions can effectively explore pathological changes associated with high myopia, thereby improving the accuracy of automatic classification of high myopia. Experimental results using clinically obtained datasets on two progressively different classification tasks demonstrate the effectiveness and efficiency of the detection method of this invention.

[0111] Exemplary device

[0112] like Figure 7 As shown, corresponding to the above-described retinal optical coherence tomography (OCT) image detection method, this embodiment of the invention also provides a retinal OCT image detection device, which includes:

[0113] The image feature acquisition module 600 is used to input retinal optical coherence tomography images into a convolutional neural network to obtain image features at different scales and group the image features into a global group containing two image features and a local group containing one image feature, wherein the image features are at least three.

[0114] The global classification label extraction module 610 inputs the global group into the Transformer layer to obtain the global classification label output by the Transformer layer. The Transformer layer includes a cross attention module, which is used to exchange information between two input image features.

[0115] The local classification label extraction module 620 is used to extract features from the local group to obtain local classification labels;

[0116] The fusion module 630 is used to fuse all the global classification labels with the local classification labels to obtain fused classification labels;

[0117] The classification module 640 is used to obtain the detection results of the retinal optical coherence tomography image based on the fused classification labels.

[0118] Specifically, in this embodiment, the specific functions of each module of the above-mentioned retinal optical coherence tomography image detection device can be referred to the corresponding description in the above-mentioned retinal optical coherence tomography image detection method, and will not be repeated here.

[0119] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 8As shown. The aforementioned smart terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a retinal optical coherence tomography (OCT) image detection program. The internal memory provides an environment for the operation of the operating system and the OCT image detection program stored in the non-volatile storage medium. The network interface of the smart terminal is used for communication with external terminals via a network connection. When the OCT image detection program is executed by the processor, it implements the steps of any of the aforementioned retinal optical coherence tomography image detection methods. The display screen of the smart terminal can be a liquid crystal display (LCD) or an e-ink display.

[0120] Those skilled in the art will understand that Figure 8 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0121] In one embodiment, a smart terminal is provided, the smart terminal including a memory, a processor, and a retinal optical coherence tomography (OCT) image detection program stored in the memory and executable on the processor, wherein the retinal OCT image detection program, when executed by the processor, performs the following operation instructions:

[0122] The optical coherence tomography (OCT) images of the retina are input into a convolutional neural network to obtain image features at different scales. The image features are then grouped into a global group containing two image features and a local group containing one image feature. The number of image features is at least three.

[0123] The global group is input into the Transformer layer to obtain the global classification label output by the Transformer layer. The Transformer layer includes a cross attention module, which is used to exchange information between two input image features.

[0124] Feature extraction is performed on the local groups to obtain local classification labels;

[0125] All the global classification labels are fused with the local classification labels to obtain the fused classification labels;

[0126] Based on the fused classification labels, the detection results of the retinal optical coherence tomography image are obtained.

[0127] Optionally, the image features are five, and the image features are grouped to obtain two global groups and one local group. There are two Transformer layers, and each global group is input to one Transformer layer.

[0128] Optionally, the step of inputting the global group into the Transformer layer to obtain the global classification label output by the Transformer layer includes:

[0129] Each image feature in the global group is input into a first Transformer encoder to obtain a first feature sequence corresponding to the image feature. The first feature sequence includes a classification label and a patch label.

[0130] Based on the cross-attention module, the classification label of each first feature sequence in the global group is sequentially fused with the patch label of another first feature sequence and the first feature sequence corresponding to the classification label is updated.

[0131] Each of the first feature sequences in the global group is input into a second Transformer encoder to obtain a second feature sequence;

[0132] By concatenating all the classification labels in the second feature sequence, the global classification label is obtained.

[0133] Optionally, the step of fusing the classification label of each first feature sequence in the global group with the patch label of another first feature sequence based on the cross-attention module and updating the first feature sequence corresponding to the classification label includes:

[0134] The classification label is processed according to the first dimension aligned linear projection function, and the processed classification label is fused with the patch label to obtain the first fused feature sequence;

[0135] The first fused feature sequence is input into the cross-attention encoder to obtain the attention score-weighted feature sequence.

[0136] The feature sequence weighted by the classification label and the attention score is input into the second dimension aligned linear projection function to obtain the processed classification label. The second dimension aligned linear projection function matches the first dimension aligned linear projection function.

[0137] The processed classification label is fused with the patch label to obtain a second fused feature sequence;

[0138] Update the first feature sequence corresponding to the classification label based on the second fused feature sequence.

[0139] Optionally, the cross-attention encoder employs a multi-head attention mechanism, and the expression of the cross-attention encoder is:

[0140]

[0141] in, Let C represent the learnable parameters in the network, and let C and h represent the embedding feature dimension and the number of heads in the Transformer encoder, respectively. The classification label in the fused feature sequence, The fused feature sequence The feature sequence after attention score weighting.

[0142] Optionally, the classification label and the feature sequence weighted by the attention score are input into a second-dimensional aligned linear projection function to obtain the processed classification label, including:

[0143] The attention-score-weighted feature sequence is processed according to the second-dimensional aligned linear projection function to obtain the processing result;

[0144] The processing result is vector-added with the classification label to obtain the processed classification label.

[0145] Optionally, after obtaining the detection results of the retinal optical coherence tomography image based on the fused classification labels, the method further includes:

[0146] Retinal optical coherence tomography images of the target patient in the horizontal and vertical scanning directions, respectively, are acquired in accordance with the detection results.

[0147] The step described above involves inputting the retinal optical coherence tomography image into a convolutional neural network until a first fused classification label corresponding to the horizontal scanning direction and a second fused classification label corresponding to the vertical scanning direction are obtained, respectively.

[0148] The first fused classification label and the second fused classification label are concatenated and classified to obtain the classification result.

[0149] This invention also provides a computer-readable storage medium storing a retinal optical coherence tomography (OCT) image detection program. When the OCT image detection program is executed by a processor, it implements the steps of any of the retinal optical coherence tomography (OCT) image detection methods provided in this invention.

[0150] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0151] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0152] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0153] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0154] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of the above modules or units is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0155] If the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0156] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not mean that the essence of the corresponding technical solutions deviates from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting retinal optical coherence tomography images, characterized in that, The method includes: The optical coherence tomography (OCT) images of the retina are input into a convolutional neural network to obtain image features at different scales. The image features are then grouped into a global group containing two image features and a local group containing one image feature. The number of image features is at least three. The global group is input into the Transformer layer to obtain the global classification label output by the Transformer layer. The Transformer layer includes a cross attention module, which is used to exchange information between two input image features. Feature extraction is performed on the local groups to obtain local classification labels; All the global classification labels are fused with the local classification labels to obtain the fused classification labels; Based on the fused classification labels, the detection results of the retinal optical coherence tomography image are obtained; The step of inputting the global group into the Transformer layer to obtain the global classification label output by the Transformer layer includes: Each image feature in the global group is input into a first Transformer encoder to obtain a first feature sequence corresponding to the image feature. The first feature sequence includes a classification label and a patch label. Based on the cross-attention module, the classification label of each first feature sequence in the global group is sequentially fused with the patch label of another first feature sequence and the first feature sequence corresponding to the classification label is updated. Each of the first feature sequences in the global group is input into a second Transformer encoder to obtain a second feature sequence; By concatenating all the classification labels in the second feature sequence, the global classification label is obtained; The method based on the cross-attention module sequentially fuses the classification label of each first feature sequence in the global group with the patch label of another first feature sequence and updates the first feature sequence corresponding to the classification label, including: The classification label is processed according to the first dimension aligned linear projection function, and the processed classification label is fused with the patch label to obtain the first fused feature sequence; The first fused feature sequence is input into the cross-attention encoder to obtain the attention score-weighted feature sequence. The feature sequence weighted by the classification label and the attention score is input into the second dimension aligned linear projection function to obtain the processed classification label. The second dimension aligned linear projection function matches the first dimension aligned linear projection function. The processed classification label is fused with the patch label to obtain a second fused feature sequence; Update the first feature sequence corresponding to the classification label according to the second fused feature sequence; The specific expression for the fusion operation is: , in, It is a first-dimensional aligned linear projection function. , These are the classification labels for high-dimensional feature sequences and low-dimensional feature sequences, respectively. , These are patch labels for low-dimensional feature sequences and patch labels for high-dimensional feature sequences, respectively. , These are the high-dimensional fused feature sequences and the low-dimensional fused feature sequences, respectively.

2. The method for detecting retinal optical coherence tomography images as described in claim 1, characterized in that, The image features are five in number. The image features are grouped to obtain two global groups and one local group. There are two Transformer layers. Each global group is input to one Transformer layer.

3. The method for detecting retinal optical coherence tomography images as described in claim 1, characterized in that, The cross-attention encoder employs a multi-head attention mechanism, and its expression is as follows: in, Let C represent the learnable parameters in the network, and let C and h represent the embedding feature dimension and the number of heads in the Transformer encoder, respectively. The classification label in the fused feature sequence, The fused feature sequence The feature sequence after attention score weighting.

4. The method for detecting retinal optical coherence tomography images as described in claim 1, characterized in that, The feature sequence weighted by the classification label and the attention score is input into the second-dimensional aligned linear projection function to obtain the processed classification label, including: The attention-score-weighted feature sequence is processed according to the second-dimensional aligned linear projection function to obtain the processing result; The processing result is vector-added with the classification label to obtain the processed classification label.

5. The method for detecting retinal optical coherence tomography images as described in claim 1, characterized in that, After obtaining the detection results of the retinal optical coherence tomography image based on the fused classification labels, the method further includes: Retinal optical coherence tomography images of the target patient in the horizontal and vertical scanning directions corresponding to the detection results were acquired respectively. The step described above involves inputting the retinal optical coherence tomography image into a convolutional neural network until a first fused classification label corresponding to the horizontal scanning direction and a second fused classification label corresponding to the vertical scanning direction are obtained, respectively. The first fused classification label and the second fused classification label are concatenated and classified to obtain the classification result.

6. A retinal optical coherence tomography image detection device, characterized in that, The device includes: The image feature acquisition module is used to input retinal optical coherence tomography images into a convolutional neural network to obtain image features at different scales and group the image features into a global group containing two image features and a local group containing one image feature, wherein the image features are at least three. The global classification label extraction module inputs the global group into the Transformer layer to obtain the global classification label output by the Transformer layer. The Transformer layer includes a cross attention module, which is used to exchange information between two input image features. The step of inputting the global group into the Transformer layer to obtain the global classification label output by the Transformer layer includes: Each image feature in the global group is input into a first Transformer encoder to obtain a first feature sequence corresponding to the image feature. The first feature sequence includes a classification label and a patch label. Based on the cross-attention module, the classification label of each first feature sequence in the global group is sequentially fused with the patch label of another first feature sequence and the first feature sequence corresponding to the classification label is updated. Each of the first feature sequences in the global group is input into a second Transformer encoder to obtain a second feature sequence; By concatenating all the classification labels in the second feature sequence, the global classification label is obtained; The method based on the cross-attention module sequentially fuses the classification label of each first feature sequence in the global group with the patch label of another first feature sequence and updates the first feature sequence corresponding to the classification label, including: The classification label is processed according to the first dimension aligned linear projection function, and the processed classification label is fused with the patch label to obtain the first fused feature sequence; The first fused feature sequence is input into the cross-attention encoder to obtain the attention score-weighted feature sequence. The feature sequence weighted by the classification label and the attention score is input into the second dimension aligned linear projection function to obtain the processed classification label. The second dimension aligned linear projection function matches the first dimension aligned linear projection function. The processed classification label is fused with the patch label to obtain a second fused feature sequence; Update the first feature sequence corresponding to the classification label according to the second fused feature sequence; The specific expression for the fusion operation is: , in, It is a first-dimensional aligned linear projection function. , These are the classification labels for high-dimensional feature sequences and low-dimensional feature sequences, respectively. , These are patch labels for low-dimensional feature sequences and patch labels for high-dimensional feature sequences, respectively. , These are the high-dimensional fused feature sequences and the low-dimensional fused feature sequences, respectively. The local classification label extraction module is used to extract features from the local groups to obtain local classification labels; The fusion module is used to fuse all the global classification labels with the local classification labels to obtain fused classification labels; The classification module is used to obtain the detection results of the retinal optical coherence tomography image based on the fused classification labels.

7. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a retinal optical coherence tomography (OCT) image detection program stored in the memory and executable on the processor. When the processor executes the retinal OCT image detection program, it implements the steps of the retinal OCT image detection method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a retinal optical coherence tomography (OCT) image detection program, which, when executed by a processor, implements the steps of the retinal OCT image detection method as described in any one of claims 1-5.

Images