A method and related products for training a domain adaptation classification model

Abstract

The present disclosure relates to a method for training a domain adaptation classification model and related products. The method comprises: obtaining a plurality of sets of training data, wherein each set of training data comprises: a non-wide-angle fundus image as a source domain and a super-wide-angle fundus image as a target domain, the super-wide-angle fundus image capturing a larger fundus area than the non-wide-angle fundus image; centering the non-wide-angle fundus image and the super-wide-angle fundus image in each set of training data; and training the domain adaptation classification model using the centered training data.

Description

Technical Field

[0001] This disclosure generally relates to the field of deep learning. More specifically, this disclosure relates to a method, apparatus, and computer-readable storage medium for training a domain-adaptive classification model. Furthermore, this disclosure also relates to a method, apparatus, and computer-readable storage medium for classifying the state of fundus images. Background Technology

[0002] Ultra-wide-angle fundus imaging is a more efficient method of fundus acquisition than ordinary non-wide-angle fundus imaging. A single ultra-wide-angle fundus image can cover 80% of the retina, while ordinary non-wide-angle fundus images can only cover about 15%. Therefore, using ultra-wide-angle fundus cameras to acquire fundus images can help radiologists better understand the development of related diseases (for example, retinal detachment may start from the periphery, which non-wide-angle fundus cameras cannot capture in the early stages).

[0003] Since ultra-wide-angle fundus cameras have only become widely used in recent years, the publicly available datasets are often very small, making it impossible to train robust deep networks.

[0004] Domain adaptation models typically transfer data features from a data-rich source domain (e.g., non-wide-angle fundus images in this application) to a target domain (e.g., ultra-wide-angle fundus images in this application) through various methods. This allows the model to utilize features from both the source and target domains for classification training, improving the accuracy of classification tasks. However, ultra-wide-angle and non-wide-angle fundus images differ significantly in their coverage area of ​​the retina, resulting in different scales. Directly importing both into the model for training often yields inaccurate results. Summary of the Invention

[0005] To at least partially address the technical problems mentioned in the background section, this disclosure provides a scheme for training domain-adaptive classification models. Using the scheme of this disclosure, ultra-wide-angle fundus images can be effectively applied to the training of classification models to improve their recognition accuracy. To this end, this disclosure provides solutions in several aspects, including...

[0006] In a first aspect, this disclosure provides a method for training a domain adaptation classification model. The method includes: acquiring multiple sets of training data, wherein each set of training data includes: a non-wide-angle fundus image as a source domain and an ultra-wide-angle fundus image as a target domain, wherein the fundus region captured by the ultra-wide-angle fundus image is larger than the fundus region captured by the non-wide-angle fundus image; center-aligning the non-wide-angle fundus image and the ultra-wide-angle fundus image in each set of training data; and importing the aligned training data into the domain adaptation classification model for training.

[0007] In one embodiment, the step of centering a non-wide-angle fundus image and an ultra-wide-angle fundus image in a set of training data includes: acquiring a solid color image with the same size as the ultra-wide-angle fundus image; scaling the non-wide-angle fundus image to match the size of the same fundus region in the ultra-wide-angle fundus image; embedding the scaled non-wide-angle fundus image into the central region of the solid color image to obtain an embedded fundus image; and centering the embedded fundus image and the ultra-wide-angle fundus image.

[0008] In another embodiment, after center-aligning the embedded fundus image and the ultra-wide-angle fundus image, the method further includes: performing image segmentation on the embedded fundus image and the ultra-wide-angle fundus image respectively to obtain N*N first image blocks and N*N second image blocks of equal size, where the first image blocks are image blocks obtained from the segmentation of the embedded fundus image and the second image blocks are image blocks obtained from the segmentation of the ultra-wide-angle fundus image, and N is an integer greater than 1; performing mask encoding on the first image blocks and the second image blocks respectively, so that each image block obtains a mask; and importing the aligned training data into the domain adaptation classification model for training, including: importing the image blocks with consistent masks from the N*N first image blocks and the N*N second image blocks into the domain adaptation classification model for training.

[0009] In another embodiment, mask encoding is performed on the first image block and the second image block respectively, including: setting the mask of the image block corresponding to the region of the non-wide-angle fundus image embedded in the N*N first image blocks to 1, setting the mask of the image block corresponding to the solid color region to 0; and setting the mask of all N*N second image blocks to 1.

[0010] In another embodiment, after obtaining N*N first image blocks and N*N second image blocks, the method further includes: converting the N*N first image blocks into a one-dimensional sequence of first image blocks; converting the N*N second image blocks into a one-dimensional sequence of second image blocks; and importing image blocks with consistent masks from the N*N first image blocks and N*N second image blocks into a domain adaptation classification model for training, including: importing image blocks with consistent masks from the converted one-dimensional sequence of first image blocks and one-dimensional sequence of second image blocks into the domain adaptation classification model for training.

[0011] In another embodiment, the domain adaptation classification model includes: a feature extraction module, a domain adaptation module, and a fundus state classification module; the feature extraction module is connected to both the domain adaptation module and the fundus state classification module; the feature extraction module is used to extract features from the input training data to obtain fundus image features; the domain adaptation module is used to remove the domain information between the fundus image features of the source domain and the fundus image features of the target domain, so that the non-wide-angle fundus image of the source domain and the ultra-wide-angle fundus image of the target domain achieve domain adaptation; the fundus state classification module is used to classify and identify the fundus image state based on the fundus image features adjusted by domain adaptation.

[0012] In another embodiment, the domain adaptation module includes: a gradient inversion layer and a domain classifier; the gradient inversion layer is used to multiply the gradient in the fundus image features by a negative constant during the backpropagation training process; the domain classifier is used to perform domain classification and recognition on the fundus image features of the source domain and the target domain after the gradient inversion layer is processed, until the image features of the source domain and the target domain cannot be distinguished.

[0013] In a second aspect, this disclosure also provides an apparatus for training a domain adaptation classification model, comprising: a processor; and a memory storing program instructions for training the domain adaptation classification model, wherein when the program instructions are executed by the processor, the apparatus implements the various embodiments of the first aspect described above.

[0014] In a third aspect, this disclosure also provides a computer-readable storage medium having stored thereon computer-readable instructions for training a domain-adaptive classification model, which, when executed by one or more processors, implement the various embodiments of the first aspect described above.

[0015] In a fourth aspect, this disclosure also provides a method for classifying the state of fundus images, comprising: acquiring a non-wide-angle fundus image or an ultra-wide-angle fundus image; importing the non-wide-angle fundus image or the ultra-wide-angle fundus image into the aforementioned trained model to obtain a classification result of the fundus image state.

[0016] In a fifth aspect, this disclosure also provides an apparatus for classifying the state of fundus images, comprising: a processor; and a memory storing program instructions for classifying the state of fundus images, wherein when the program instructions are executed by the processor, the apparatus implements the embodiments of the aforementioned fourth aspect.

[0017] In a sixth aspect, this disclosure also provides a computer-readable storage medium having stored thereon computer-readable instructions for classifying fundus image states, which, when executed by one or more processors, implement the embodiments of the fourth aspect described above.

[0018] The scheme disclosed herein centers the non-wide-angle fundus images (source domain) and the ultra-wide-angle fundus images (target domain) in the training data, and then imports the aligned training data into the domain adaptation classification model for training. This allows for full utilization of the effective features of the non-wide-angle and ultra-wide-angle fundus images (e.g., image features of the central region of the fundus in the non-wide-angle fundus image and image features of the peripheral region of the fundus in the ultra-wide-angle fundus image) during the training process, thereby improving the recognition accuracy of the classification model. Attached Figure Description

[0019] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0020] Figure 1 This is a simplified flowchart illustrating a method for training a domain adaptation classification model according to an embodiment of the present disclosure;

[0021] Figure 2 This is a schematic diagram showing a comparison of retinal coverage areas between non-wide-angle fundus images and ultra-wide-angle fundus images according to embodiments of this disclosure;

[0022] Figure 3 This is a detailed flowchart illustrating a method for training a domain adaptation classification model according to an embodiment of the present disclosure;

[0023] Figure 4 This is a schematic diagram showing image segmentation after embedding fundus images and ultra-wide-angle fundus images according to embodiments of this disclosure;

[0024] Figure 5a This is a schematic diagram illustrating the structure of inputting an ultra-wide-angle fundus image to a VIT according to an embodiment of this disclosure;

[0025] Figure 5b This is a schematic diagram illustrating the structure of inputting a non-wide-angle fundus image to the VIT according to an embodiment of the present disclosure;

[0026] Figure 6 This is a schematic diagram illustrating the structure of a domain adaptation classification model according to an embodiment of the present disclosure; and

[0027] Figure 7 This is an exemplary structural block diagram illustrating an apparatus for training a domain adaptation classification model or for classifying fundus image states according to embodiments of the present disclosure. Detailed Implementation

[0028] The technical solutions in the embodiments of this disclosure will now be clearly and completely described with reference to the accompanying drawings. It should be understood that the embodiments described in this specification are only some embodiments provided by this disclosure for the purpose of facilitating a clear understanding of the solutions and complying with legal requirements, and are not all embodiments that can be implemented by this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments disclosed in this specification without creative effort are within the scope of protection of this disclosure.

[0029] In medical applications, ultra-wide-angle fundus images, due to their wider coverage compared to non-wide-angle fundus images, help radiologists better understand the progression of related diseases (e.g., retinal detachment may begin peripherally, which non-wide-angle fundus cameras cannot capture in the early stages). However, ultra-wide-angle fundus cameras have only recently become widely used, and the available public datasets are often small in size, making it difficult to train robust deep networks. Therefore, this disclosure proposes a domain transfer scheme based on ultra-wide-angle and non-wide-angle fundus images. This scheme fully utilizes the rich features of non-wide-angle fundus images in existing training databases, transferring image features from the non-wide-angle fundus image domain to the ultra-wide-angle fundus image domain. This effectively expands the training dataset used for classification model training and improves the recognition accuracy of the classification model.

[0030] Figure 1 This is a simplified flowchart illustrating a method 100 for training a domain adaptation classification model according to an embodiment of the present disclosure. Figure 1 As shown:

[0031] In step 102, multiple sets of training data are acquired, wherein one set of training data includes: a non-wide-angle fundus image as the source domain and an ultra-wide-angle fundus image as the target domain.

[0032] In this embodiment of the disclosure, the fundus area captured by the ultra-wide-angle fundus image is larger than the fundus area captured by the non-wide-angle fundus image. Figure 2 A schematic diagram comparing the retinal coverage area of ​​non-wide-angle fundus images and ultra-wide-angle fundus images is shown. Figure 2As shown, the left side of the image is a non-wide-angle fundus image, and the right side is an ultra-wide-angle fundus image. In this field, non-wide-angle fundus images, also known as conventional fundus images or color fundus images, are fundus images captured by a typical retinal camera with a field of view of approximately 30° to 60°, typically covering about 15% of the fundus area. Ultra-wide-angle fundus images have a field of view of approximately 200°, typically covering about 80% of the fundus area. The main difference between the two types of fundus images is the range of the field of view and the different imaging equipment used; these can be referred to as the two fundus images having different modalities. The two fundus images in the same training data can be taken from the same eye of the same person, meaning the two images are highly matched; or, the two fundus images in the same training data can also be taken from the eyes of different people.

[0033] In this embodiment of the disclosure, the source domain and the target domain refer to two domains in the domain adaptation model. The source domain is the source image domain of the image features to be transferred, and the target domain is the target image domain to which the image features are transferred. The purpose of the domain adaptation model is to find or construct a common representation space for these two domains.

[0034] In step 104, the non-wide-angle fundus images and ultra-wide-angle fundus images in each group of training data are center-aligned.

[0035] like Figure 2 As shown, in practical applications, since the retinal coverage area of ​​ultra-wide-angle fundus images is much larger than that of non-wide-angle fundus images, ultra-wide-angle fundus images generally correspond to a larger resolution (e.g., 800x800 or higher). According to the coverage area conversion, if the scale of the same fundus region of non-wide-angle fundus images and ultra-wide-angle fundus images is scaled to a similar size (i.e., converted to a scale-equivalent resolution), the side length of non-wide-angle fundus images is about 1 / 2 to 1 / 3 of that of ultra-wide-angle images, and the resolution is generally 384x384 or higher. After converting to scale-equivalent resolution, the central regions of ultra-wide-angle fundus images can be aligned with those of non-wide-angle fundus images. This allows for the transfer of image features from the non-wide-angle fundus image domain to the ultra-wide-angle fundus image domain during subsequent training, thereby fully utilizing the effective features of both images (e.g., image features of the central fundus region in non-wide-angle fundus images and image features of the peripheral fundus region in ultra-wide-angle fundus images). Effective features refer to image features that help identify pathological conditions in fundus images.

[0036] As an example, a Vision Transformer (VIT) model can be used as the base network for center alignment to achieve alignment of non-wide-angle fundus images with ultra-wide-angle fundus images in terms of size and position, thereby improving the ability to perform subsequent domain adaptation (i.e., transferring image features from the map domain of the non-wide-angle fundus image to the map domain of the ultra-wide-angle fundus image). A specific implementation method for center alignment using the Vision Transformer model is provided in subsequent embodiments, which will be described in detail later.

[0037] In step 106, the aligned training data is imported into the domain adaptation classification model for training. Specifically, the domain adaptation classification model can include two parts: a domain adaptation part and a classification training part. The domain adaptation part is mainly used to find or construct a common representation space (composed of multiple image features) between the two domains (source domain and target domain). The goal is to obtain a space where the domains are close to each other, which can be achieved by using adversarial machine learning, encouraging feature representations of training samples from different domains to be difficult to distinguish. The classification training part then uses the domain-adapted image features for classification training.

[0038] In this embodiment, classification training mainly refers to training on the classification and recognition of various fundus image states to assist doctors in identifying different fundus image states corresponding to different diseases, or fundus image states corresponding to different stages of a disease. For example, fundus images can be used to detect retinal diseases such as diabetic retinopathy (DR), age-related macular degeneration (AMD), and glaucoma. However, in practical medical applications, diseases with weak refractive media, such as cataracts and vitreous opacities, are often difficult to examine with ordinary non-wide-angle fundus cameras. Ultra-wide-angle fundus cameras can cover a larger retinal area and can screen for more pathological changes from the peripheral retina, such as retinal degeneration, detachment, hemorrhage, and exudation. Therefore, the classification model trained using ultra-wide-angle fundus images can more accurately identify different fundus image states, thereby effectively assisting medical personnel in identifying different diseases or different pathological stages of diseases.

[0039] Figure 3 This is a detailed flowchart illustrating a method 300 for training a domain adaptation classification model according to an embodiment of the present disclosure. Figure 3 As shown:

[0040] In step 302, multiple sets of training data are acquired. One set of training data includes a non-wide-angle fundus image as the source domain and an ultra-wide-angle fundus image as the target domain. The content of step 302 is the same as that of step 102 above, and will not be repeated here.

[0041] In step 304, a solid color image with the same size as the ultra-wide-angle fundus image is obtained.

[0042] Specifically, the solid color image can be a pure black image to better serve as an image background.

[0043] In step 306, the non-wide-angle fundus image is scaled to the same size as the same fundus region in the ultra-wide-angle fundus image.

[0044] like Figure 2 As shown, within the same fundus region, the non-wide-angle fundus images and ultra-wide-angle fundus images from the original training data have inconsistent image sizes. This may lead to the larger fundus region of the non-wide-angle fundus image covering some of the image features of the peripheral fundus region in the ultra-wide-angle fundus image during image feature transfer, thus preventing the full utilization of the effective features in the ultra-wide-angle fundus image. Therefore, the non-wide-angle fundus image needs to be scaled before image alignment.

[0045] It should be noted that, due to differences in different scenarios and operational errors in actual applications, expressions such as "consistent size" or "same size" in the embodiments of this disclosure refer to approximately the same size within an acceptable range of error, and do not imply precise consistency in size.

[0046] In step 308, the scaled non-wide-angle fundus image is embedded into the central region of the solid color image to obtain the embedded fundus image.

[0047] like Figure 4 As shown, Figure 4 The image on the left is the embedded fundus image after "embedding the non-wide-angle fundus image into the central region of a pure black image". The embedded fundus image retains the effective features of the non-wide-angle fundus image (which can be represented by mask encoding). Specifically, in one implementation, after obtaining the embedded fundus image, image segmentation can be performed on both the embedded fundus image and the ultra-wide-angle fundus image to obtain N*N first image blocks and N*N second image blocks respectively (as an example). Figure 4The embedded fundus image and the ultra-wide-angle fundus image are segmented into 8*8 image patches. The first image patch is the image patch obtained from the segmentation of the embedded fundus image, and the second image patch is the image patch obtained from the segmentation of the ultra-wide-angle fundus image, where N is an integer greater than 1. After image segmentation, the first and second image patches are masked, so that each image patch receives a mask (i.e., a Mask parameter). As an example, the mask of the image patch corresponding to the region of the non-wide-angle fundus image embedded in the N*N first image patches can be set to 1, and the mask of the image patch corresponding to the solid color region can be set to 0; the masks of the N*N second image patches can all be set to 1. In this embodiment, the image patch 1 with the mask set can be regarded as an image region with effective features.

[0048] In step 310, the embedded fundus image and the ultra-wide-angle fundus image are center-aligned.

[0049] As an example, VIT can be used as the base network for center alignment to achieve alignment of non-wide-angle fundus images with ultra-wide-angle fundus images in terms of size and position. After the mask encoding operation is completed as described above, only the image blocks with the same mask among the N*N first image blocks and N*N second image blocks (i.e., image blocks with a mask of 1 in all of the embodiments of this disclosure) are imported into VIT for alignment processing.

[0050] As an example, Figure 5a This diagram illustrates the structure of inputting an ultra-wide-angle fundus image to a VIT in an embodiment of this disclosure. Figure 5b This diagram illustrates the structure of inputting a non-wide-angle fundus image to the VIT in an embodiment of this disclosure. Figure 5a and Figure 5bAs shown, VIT can include an embedding module, a transformational encoding module, and an MLP Head module (MLP, Multilayer Perceptron). The embedding module consists of patch embedding and position embedding. Patch embedding extracts information from image patches through linear projection. Position embedding is a learnable parameter used to learn the position of the corresponding patch in the image. The embedding module (Linear Projection of Flattened in the diagram) is mainly used to perform linear projection of flattened patches on the input N*N image patches, that is, to convert N*N first image patches into a one-dimensional sequence of first image patches, and to convert N*N second image patches into a one-dimensional sequence of second image patches. The transformational encoding module (Transformer Encoder) consists of multiple multi-head self-attention layers alternating with multilayer perceptrons, mainly used for global interaction and transmission of image information. The output of VIT also includes an MLP Head module, which is used for classification. This classification head is an MLP with a hidden layer during the pre-training stage.

[0051] like Figure 5a As shown, since the mask of the second image block corresponding to all ultra-wide-angle fundus images in this embodiment is "1", all second image blocks are input into the embedding module for processing. Figure 5b As shown, since the mask of the first image block corresponding to the pure black area in this embodiment is "0" and the mask of the second image block corresponding to the non-wide-angle fundus image area is "1", the pure black image block in the figure is not input to the embedding module, while the image block showing the fundus image area can be input to the embedding module.

[0052] Specifically, each image block can have a corresponding position identifier within an N*N image region, ensuring that after the first and second image blocks are input into the VIT, they can accurately correspond to their respective image positions, achieving more accurate image position alignment. Figure 5a and Figure 5b In the process, since the first image block and the second image block of the one-dimensional sequence are input sequentially, the position identifiers of the image blocks can be sequentially marked as "0", "1", "2", "3", "4", "5", "6", "7", "8" and "9", which enables VIT to identify the position information corresponding to each image block, thereby achieving more accurate image feature extraction.

[0053] In step 312, the aligned training data is imported into the domain adaptation classification model for training. Specifically, Figure 6 The diagram shown is a structural schematic of the domain adaptation classification model in an embodiment of this disclosure. Figure 6 As shown, the embedded fundus image as the source domain and the ultra-wide-angle fundus image as the target domain are input into the visual transformation model VIT for center alignment, and the aligned image features are input into the feature extraction module 601 in the domain adaptation classification model 610 for further image feature extraction.

[0054] In this embodiment, the domain adaptation classification model 610 may include: a feature extraction module 601, a domain adaptation module 602, and a fundus state classification module 603. The feature extraction module 601 is connected to both the domain adaptation module 602 and the fundus state classification module 603. The feature extraction module 601 extracts features from the input training data to obtain fundus image features (including image features from ultra-wide-angle fundus images and non-wide-angle fundus images). The domain adaptation module 602 removes domain information between the fundus image features of the source domain and the fundus image features of the target domain, enabling domain adaptation between the non-wide-angle fundus image of the source domain and the ultra-wide-angle fundus image of the target domain. Specifically, the domain adaptation module 602 includes: a gradient inversion layer 6021 and a domain classifier 6022; the gradient inversion layer 6021 is used to multiply the gradient in the fundus image features by a negative constant during the backpropagation training process; the domain classifier 6022 is used to perform domain classification and recognition on the fundus image features of the source domain and the target domain after the gradient inversion layer is processed, until the image features of the source domain and the target domain cannot be distinguished. In practical applications, the training objectives of the feature extraction module 601 and the neighborhood classifier 6022 are adversarial. That is, the image features output by the feature extraction module 601, after being recognized by the neighborhood classifier 6022, can distinguish which image features correspond to non-wide-angle fundus images and which correspond to ultra-wide-angle fundus images. However, a gradient inversion layer 6021 is set between the feature extraction module 601 and the neighborhood classifier 6022. When the neighborhood classifier 6022 obtains the loss (the difference between the predicted value and the true value) and backpropagates (i.e., propagates from the neighborhood classifier towards the feature extraction module), it passes the loss layer by layer. Each layer calculates the gradient based on the returned error and updates its parameters. The gradient inversion layer 6021 multiplies the error passed to it by a negative number, thus making the training objectives of the networks before and after the gradient inversion layer opposite, achieving an adversarial effect. After repeated iterations, the training objective is achieved (i.e., neighborhood adaptation is realized) until the neighborhood classifier 6022 can no longer distinguish between image features in the source and target domains.

[0055] Furthermore, the fundus state classification module 603 is used to classify and recognize fundus image states based on fundus image features adjusted by domain adaptation. During training, the domain adaptation module 602 continuously adjusts the output of the feature extraction module 601 through a loss function, and the fundus state classification module 603 continuously trains to classify and recognize fundus image states based on the updated image features. When the image features of the source and target domains achieve domain adaptation, the fundus state classification module 603 gradually achieves the classification training goal (e.g., the error value for identifying which category of fundus image state the target image belongs to meets a set threshold). As an example, the classification result of fundus image states can be fundus image states corresponding to different types of diseases, such as fundus image states corresponding to diabetic retinopathy (DR), age-related macular degeneration (AMD), and glaucoma. In another implementation, the classification result of fundus image status can also be the fundus image status corresponding to different stages of a disease, such as the fundus image status corresponding to the three pathological stages of diabetic retinopathy. Specifically, the fundus image status of stage I has microaneurysms or small hemorrhages, the fundus image status of stage II has yellowish-white hard exudates or hemorrhages, and the fundus image status of stage III has white soft exudates or hemorrhages.

[0056] In this embodiment of the disclosure, by embedding a non-wide-angle fundus image into a solid color image, the size matching with the ultra-wide-angle fundus image is achieved. The effective feature regions of the embedded fundus image and the ultra-wide-angle fundus image are distinguished by mask encoding. This enables accurate and effective image alignment during the image center alignment process, thereby providing effective data support for subsequent training of domain adaptation and fundus image state classification.

[0057] After executing as follows Figure 3 Following the illustrated implementation process, a trained domain-adaptive classification model is obtained, which can then be used to classify and recognize fundus image states. Specifically, non-wide-angle or ultra-wide-angle fundus images can be imported into the trained domain-adaptive classification model to obtain classification results for fundus image states. As an example, the classification results for fundus image states can be fundus image states corresponding to different types of diseases, such as fundus image states corresponding to retinal diseases such as diabetic retinopathy (DR), age-related macular degeneration (AMD), and glaucoma. In another implementation, the classification results for fundus image states can also be fundus image states corresponding to different stages of a disease, such as fundus image states corresponding to the three pathological stages of diabetic retinopathy. Specifically, the fundus image state in stage I has microaneurysms or small hemorrhages; the fundus image state in stage II has yellowish-white hard exudates or hemorrhages; and the fundus image state in stage III has white soft exudates or hemorrhages.

[0058] Figure 7 This is an exemplary structural block diagram illustrating a device 700 for training a domain adaptation classification model or for classifying fundus image states according to embodiments of the present disclosure. It will be understood that the device implementing the solutions of this disclosure may be a single device (e.g., a computing device) or a multifunctional device including various peripheral devices.

[0059] like Figure 7 As shown, the device disclosed herein may include a central processing unit (“CPU”) 711, which may be a general-purpose CPU, a special-purpose CPU, or other information processing and program execution unit. Furthermore, the device 700 may also include a mass storage 712 and a read-only memory (“ROM”) 713, wherein the mass storage 712 may be configured to store various types of data, including various fundus images and multimodal information related to the patient, algorithm data, intermediate results, and various programs required to run the device 700. The ROM 713 may be configured to store power-on self-tests for the device 700, initialization of various functional modules in the system, drivers for the system's basic input / output, and data and instructions required to boot the operating system.

[0060] Optionally, device 700 may also include other hardware platforms or components, such as the tensor processing unit (“TPU”) 714, graphics processing unit (“GPU”) 715, field-programmable gate array (“FPGA”) 716, and machine learning unit (“MLU”) 717 shown. It is understood that although various hardware platforms or components are shown in device 700, they are merely exemplary and not limiting, and those skilled in the art can add or remove appropriate hardware as needed. For example, device 700 may include only a CPU, associated storage devices, and interface devices to implement the device of this disclosure for training a domain adaptation classification model or for classifying fundus image states.

[0061] In some embodiments, to facilitate data transmission and interaction with external networks, the device 700 of this disclosure further includes a communication interface 718, through which it can connect to a local area network / wireless local area network (“LAN / WLAN”) 705, and further through the LAN / WLAN to connect to a local server 706 or to the Internet (“Internet”) 707. Alternatively or additionally, the device 700 of this disclosure can also directly connect to the Internet or cellular network via the communication interface 718 based on wireless communication technology, such as 3G (“3G”), 4G (“4G”), or 5G (“5G”) wireless communication technology. In some application scenarios, the device 700 of this disclosure can also access a server 708 and a database 709 on an external network as needed to obtain various known algorithms, data, and modules, and can remotely store various data, such as various data or instructions for presenting, for example, patient fundus images, multimodal information, embedding vectors, or clustering results.

[0062] Peripherals of device 700 may include a display device 702, an input device 703, and a data transmission interface 704. In one embodiment, the display device 702 may include, for example, one or more speakers and / or one or more visual displays, configured to perform clustering on a model trained according to this disclosure, or to provide voice prompts and / or display images and videos during clustering. The input device 703 may include, for example, a keyboard, mouse, microphone, gesture capture camera, or other input buttons or controls, configured to receive audio data input and / or user commands. The data transmission interface 704 may include, for example, a serial interface, parallel interface, or Universal Serial Bus interface (“USB”), Small Computer System Interface (“SCSI”), Serial ATA, FireWire (“FireWire”), PCI Express, and High Definition Multimedia Interface (“HDMI”), configured for data transmission and interaction with other devices or systems. According to the scheme of this disclosure, the data transmission interface 704 can receive non-wide-angle or ultra-wide-angle fundus images captured by a fundus camera and transmit various other types of data or results to device 700.

[0063] The CPU 711, mass storage 712, ROM 713, TPU 714, GPU 715, FPGA 716, MLU 717, and communication interface 718 of the device 700 disclosed herein can be interconnected via bus 719, and can interact with peripheral devices through this bus. In one embodiment, the CPU 711 can control other hardware components in the device 700 and their peripheral devices through this bus 719.

[0064] The above combination Figure 7This document describes a device that can be used to train a domain-adaptive classification model or to classify fundus image states, as described herein. It should be understood that the device structure or architecture described herein is merely exemplary, and the implementation methods and entities of this disclosure are not limited thereto, but can be modified without departing from the spirit of this disclosure.

[0065] Based on the foregoing description in conjunction with the accompanying drawings, those skilled in the art will understand that the embodiments of this disclosure can also be implemented by a software program. Therefore, this disclosure also provides a computer program product. This computer program product can be used to implement the embodiments of this disclosure in conjunction with the accompanying drawings. Figures 1-6 The device described is used for training a domain-adaptive classification model or for classifying the state of fundus images.

[0066] It should be noted that although the operations of the methods of this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0067] It should be understood that when the terms "first," "second," "third," and "fourth," etc., are used in the claims, specification, and drawings of this disclosure, they are used only to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0068] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0069] While the embodiments of this disclosure are described above, the content is merely an example for the purpose of facilitating understanding of this disclosure and is not intended to limit the scope or application scenarios of this disclosure. Any person skilled in the art can make any modifications and changes in form and detail of the implementation without departing from the spirit and scope disclosed herein; however, the patent protection scope of this disclosure shall still be determined by the scope defined in the appended claims.

Claims

1. A method for training a domain-adaptive classification model, characterized in that, The method includes: Acquire multiple sets of training data, wherein one set of training data includes: a non-wide-angle fundus image as the source domain and an ultra-wide-angle fundus image as the target domain, wherein the fundus area captured by the ultra-wide-angle fundus image is larger than the fundus area of ​​the non-wide-angle fundus image; Center-align the non-wide-angle fundus images and ultra-wide-angle fundus images in each training data set; The aligned training data is then imported into the domain adaptation classification model for training. The method further includes: Image segmentation was performed on the embedded fundus image and the ultra-wide-angle fundus image respectively, resulting in N images of equal size. N first image blocks and N N second image blocks, wherein the first image block is the image block obtained by segmenting the embedded fundus image, and the second image block is the image block obtained by segmenting the ultra-wide-angle fundus image, and N is an integer greater than 1; Mask encoding is performed on the first image block and the second image block respectively, so that each image block obtains a mask; The step of importing the aligned training data into the domain adaptation classification model for training includes: N N first image blocks and N Image patches with identical masks from N second image patches are imported into the domain adaptation classification model for training; The obtained N N first image blocks and N After N second image patches, it also includes: The N N first image blocks are converted into a one-dimensional sequence of first image blocks; The N N second image blocks are converted into a one-dimensional sequence of second image blocks; The N N first image blocks and N Image patches with identical masks from N second image patches are imported into the domain adaptation classification model for training, including: Image patches with identical masks from the first and second image patches converted into one-dimensional sequences are imported into the domain adaptation classification model for training. The steps for center-aligning non-wide-angle fundus images with ultra-wide-angle fundus images in one set of training data include: Obtain a solid color image with the same size as the ultra-wide-angle fundus image; The non-wide-angle fundus image is scaled to the same size as the same fundus region in the ultra-wide-angle fundus image; The scaled non-wide-angle fundus image is embedded into the central region of the solid color image to obtain the embedded fundus image; The embedded fundus image and the ultra-wide-angle fundus image are center-aligned.

2. The method according to claim 1, characterized in that, The step of performing mask encoding on the first image block and the second image block respectively includes: The N The mask of the image block corresponding to the region of the non-wide-angle fundus image embedded in the N first image blocks is set to 1, and the mask of the image block corresponding to the solid color region is set to 0. The N The masks for all N second image patches are set to 1.

3. The method according to claim 1, characterized in that, The domain-adaptive classification model includes: Feature extraction module, domain adaptation module, and fundus state classification module; The feature extraction module is connected to the domain adaptation module and the fundus state classification module, respectively. The feature extraction module is used to extract features from the input training data to obtain fundus image features; The domain adaptation module is used to remove the domain information between the fundus image features of the source domain and the fundus image features of the target domain, so that the non-wide-angle fundus image of the source domain and the ultra-wide-angle fundus image of the target domain can achieve domain adaptation. The fundus state classification module is used to classify and identify the state of fundus images based on the fundus image features adjusted by domain adaptation.

4. The method according to claim 3, characterized in that, The domain adaptation module includes: a gradient inversion layer and a domain classifier; The gradient inversion layer is used to multiply the gradient in the fundus image features by a negative constant during the backpropagation training process; The domain classifier is used to classify and identify the features of the source and target domain fundus images after gradient inversion layer processing, until the image features of the source and target domains can no longer be distinguished.

5. An apparatus for training a domain-adaptive classification model, characterized in that, include: processor; as well as A memory storing program instructions for training a domain-adaptive classification model, wherein when the program instructions are executed by the processor, the device implements the method according to any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The system stores computer-readable instructions for training a domain-adaptive classification model, which, when executed by one or more processors, implement the method as described in any one of claims 1-4.

7. A method for classifying the state of fundus images, characterized in that, include: Acquire non-wide-angle or ultra-wide-angle fundus images; The non-wide-angle fundus image or the ultra-wide-angle fundus image is imported into the trained model according to claims 1-4 to obtain the classification result of the fundus image state.

8. A device for classifying the state of fundus images, characterized in that, include: processor; as well as The memory stores program instructions for classifying the state of fundus images, which, when executed by the processor, cause the device to implement the method according to claim 7.

9. A computer-readable storage medium, characterized in that, The system stores computer-readable instructions for classifying the state of fundus images, which, when executed by one or more processors, implement the method of claim 7.

Images