Fundus color photograph processing device based on attention mechanism

Abstract

The application relates to an attention mechanism-based fundus color photograph processing device, which comprises an image preprocessing module, a blood vessel feature extraction module and a convolutional neural network module; wherein the image preprocessing module comprises a mask extraction module and an image enhancement module; the mask extraction module is used for extracting a mask, and the mask is used for segmenting an image of a region of interest and an image of a non-region of interest in a fundus color photograph; the image enhancement module is used for performing contrast enhancement on the fundus color photograph; the image preprocessing module performs clipping on the fundus color photograph by using the mask to obtain a preprocessed fundus color photograph image; the blood vessel feature extraction module extracts a blood vessel distribution saliency map according to the preprocessed fundus color photograph image; the blood vessel distribution saliency map and the preprocessed fundus color photograph image are multiplied after point-to-point activation of each color channel to serve as input of the convolutional neural network module; and the convolutional neural network model focuses on learning an image region related to a hypertension symptom.

Description

Technical Field

[0001] This invention relates to an image data processing technology, and more particularly to a method and apparatus for processing fundus color image data based on an attention mechanism, specifically to a method and system for processing fundus color image data based on an attention mechanism for early screening of hypertension. Background Technology

[0002] The microvessels in the retina are the only microvessels in the human body that are not covered by skin or tissue and can be directly observed. Fundus photography can be obtained in a non-invasive and very economical way, making it more suitable for use in large-scale screening. With the development of artificial intelligence technology in the field of medical imaging, fundus photography-based screening for fundus diseases such as diabetic retinopathy, age-related macular degeneration, and glaucoma has been widely used. Therefore, systemic disease screening based on fundus photography has become a challenging area of ​​technological development.

[0003] In recent years, data-driven deep learning methods have been widely applied to disease screening using fundus photography. This method, which automatically learns features from fundus photography using deep learning models to achieve disease classification and screening, can automatically learn more complex features from the input raw image data compared to traditional methods that use manually designed features. Through automatic optimization, it achieves better performance for end-to-end screening. Since diseases such as diabetic retinopathy (DR), glaucoma, and age-related macular degeneration (AMD) directly cause varying degrees of specific lesions in the fundus, resulting in more semantic features, deep learning models can converge quickly and focus on the lesion area when screening for these diseases, often achieving better results.

[0004] For systemic chronic diseases such as hypertension, the lack of strong annotations of lesion areas makes direct screening through models difficult, rendering current algorithms ineffective in such applications. Specifically, the changes in the retina caused by hypertension are a long and slow process, so obvious lesion features are usually not visible in the early stages of the disease; instead, subtle changes gradually occur in areas such as the optic disc and microvessels. Therefore, applying deep learning methods to hypertension using fundus photography often faces greater challenges due to difficulties in feature extraction and semantic ambiguity. Consequently, there are currently few studies using deep learning methods to screen for hypertension using fundus photography. In 2020, Dai et al. used fundus photography and fundus vessel segmentation images to perform cross-sectional prediction screening for hypertension, using a simplified five-layer CNN network for classification experiments. The AUC obtained in the screening of fundus photography and fundus vessel segmentation images were only 0.65 and 0.60, respectively. In the same year, Zhang L et al. used neural network models to predict hypertension, hyperglycemia, dyslipidemia, and a series of risk factors. Their Innceoptin-V3 model, used for screening, showed an AUC of only 0.76 for predicting hypertension. Therefore, because early-stage hypertension typically does not lead to obvious lesions in the microvessels of the retina, deep learning models cannot effectively extract and learn from these features. This has resulted in the limited prevalence of research on early hypertension screening based on fundus photography. Furthermore, existing studies on hypertension screening utilize relatively limited methods with unsatisfactory results. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a fundus color image processing device based on an attention mechanism, comprising an image preprocessing module, a blood vessel feature extraction module, and a convolutional neural network module. The image preprocessing module includes a mask extraction module and an image enhancement module. The mask extraction module extracts a mask used to segment the region of interest (ROI) and non-ROI images in the fundus color image. The image enhancement module enhances the contrast of the fundus color image. The image preprocessing module uses the mask extracted by the mask extraction module to crop the contrast-enhanced fundus color image to obtain a preprocessed fundus color image. The blood vessel feature extraction module extracts a saliency map of blood vessel distribution from the preprocessed fundus color image. The convolutional neural network module performs automatic feature extraction. Furthermore, the saliency map of blood vessel distribution and the preprocessed fundus color image are multiplied by point-to-point activation of each color channel and used as the input to the convolutional neural network module.

[0006] In the above technical solution, the features automatically extracted by the convolutional neural network module are image data features related to hypertension in fundus color photographs.

[0007] In the above technical solution, the step of performing point-to-point activation multiplication of each color channel of the vascular distribution saliency map and the preprocessed fundus color image is specifically implemented using the following formula:

[0008]

[0009] Wherein, SM represents the data from the vascular distribution significance map; I k This represents the preprocessed fundus color image data; k is the number of channels in the fundus color image, with 0, 1, and 2 representing the RGB channels respectively; ⊙ represents the pixel-level product; C is a constant matrix; Mask is the mask layer data; I SM To obtain the image data after saliency-weighted activation.

[0010] In the above technical solution, the convolutional neural network module will obtain the image data I after saliency-weighted activation. SM After processing by the convolutional layers, the data is input into a stacked ResNeSt module for feature extraction.

[0011] In the above technical solution, the ResNeSt module splits the input data of size H×W×C into K structurally consistent basis arrays along the channel dimension. In each basis array, each set of features is further input into R branches, thus obtaining G=KR branches. Convolutions with kernel sizes of 1×1 and 3×3 are applied twice to each branch to obtain G deep features in different channel dimensions. In each basis array, the outputs of the R branches are used as inputs to the split attention module. The outputs of the R branches are concatenated along the channel dimension and passed through a global pooling layer to obtain a global representation. Then, they are compressed along the channel dimension through two fully connected layers to obtain the attention factors of each branch feature group. These R attention factors represent the importance of the corresponding branch in the channel dimension. After calculation by the r-Softmax function, the attention weight of each branch in the channel dimension of the basis array is calculated. Finally, the weights are multiplied by the corresponding branch outputs and summed to obtain the output of the basis array. Here, H, W, and C are the height, width, and color values ​​of each channel of the input data image, respectively, and K and R are positive integers greater than 2.

[0012] In the above technical solution, the mask extraction module extracts the image data of the red channel in the RGB color image of the fundus photo; the image data of the red channel is segmented at a low threshold to obtain a background segmentation map, which is divided into two parts: region of interest and region of non-interest; and a mask is extracted from the background segmentation map to separate the region of interest image and the region of non-interest image.

[0013] In the above technical solution, the region of interest image is a roughly circular region, and the non-region of interest is connected.

[0014] In the above technical solution, the image enhancement module transforms the fundus color photo to the LAB color space and separates the image data of the luminance channel; it uses the CLAHE algorithm to enhance the contrast of the image data of the luminance channel; it reassembles the contrast-enhanced luminance channel image data and converts it back from the LAB color space to the RGB color space.

[0015] The above technical solution further includes a multilayer perceptron module for classification by fusing image features and non-image features from clinical metadata through a multilayer perceptron model; preferably, the multilayer perceptron module includes three sequentially connected hidden layers, wherein the first hidden layer includes two input parts; wherein the first input uses preprocessed metadata as input, which includes one or more of age, gender, height, weight, heart rate, and BMI index; the second input uses the 512-dimensional feature map output by the convolutional neural network module as input.

[0016] The present invention achieves the following technical effects:

[0017] This invention uses a vascular distribution feature map obtained through vascular segmentation and post-processing as prior knowledge to guide a convolutional neural network, enabling more targeted feature extraction. Furthermore, it incorporates clinical metadata to further mitigate the influence of individual factors on screening results. Compared to existing technologies, this invention introduces a vascular feature extraction module to extract vascular distribution information, helping the deep learning model focus more on important areas such as the main retinal vessels for more targeted feature extraction. In screening, it integrates non-image features based on clinical metadata for classification, providing more diverse feature information for early hypertension screening while correcting the impact of individual factors. This further improves the accuracy of early hypertension screening, enabling more efficient large-scale early hypertension screening, helping patients detect their condition promptly and receive early intervention and treatment, and has significant social value. Attached Figure Description

[0018] Figure 1This is a system framework diagram of the present invention;

[0019] Figure 2 This is a flowchart of the fundus color image preprocessing process of the present invention;

[0020] Figure 3 This is a flowchart of the vascular feature extraction method of the present invention;

[0021] Figure 4 This is a flowchart of the post-processing procedure for vascular feature extraction according to the present invention;

[0022] Figure 5 This is a schematic diagram of the convolutional neural network module structure of the present invention;

[0023] Figure 6 This is a schematic diagram of the ResNeSt module structure of the present invention;

[0024] Figure 7 This is a schematic diagram of the multilayer sensor module structure of the present invention. Detailed Implementation

[0025] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] To address the limitations of existing technologies for hypertension screening based on fundus photography, which suffer from limited research, simplistic methods, and unsatisfactory results, this invention provides a fundus photography data processing method based on attention mechanisms, particularly suitable for early screening of hypertension symptoms. The main steps of this method are as follows: Figure 1 As shown, the method includes: preprocessing fundus color images, extracting refined vascular features using a pre-trained segmentation network, and extracting saliency maps representing vascular distribution using post-processing; fusing the saliency maps and fundus color images as input to a convolutional neural network to achieve targeted and automatic feature extraction from fundus color images; and classifying the images by fusing image features and non-image features from clinical metadata through a multilayer perceptron model to achieve early screening of hypertension symptoms.

[0027] According to another preferred embodiment of the present invention, the present invention also provides a fundus color imaging data processing system based on attention mechanisms, characterized in that it includes:

[0028] The image preprocessing module is used to preprocess fundus color images;

[0029] The vascular feature extraction module is used to extract refined vascular features through a pre-trained segmentation network and to extract a saliency map representing the distribution of blood vessels using post-processing.

[0030] The convolutional neural network module is used to selectively and automatically extract features from fundus images by fusing saliency maps and fundus images as input to the convolutional neural network.

[0031] The multilayer perceptron module is used for classification by fusing image features and non-image features from clinical metadata through a multilayer perceptron model.

[0032] Preferably, this attention-based fundus color imaging data processing system is particularly suitable for early screening of hypertension.

[0033] According to another preferred embodiment of the present invention, the present invention also provides a system for early screening of hypertension based on fundus color images, characterized in that it includes:

[0034] The image preprocessing module is used to preprocess fundus color images;

[0035] The vascular feature extraction module is used to extract refined vascular features through a pre-trained segmentation network and to extract a saliency map representing the distribution of blood vessels using post-processing.

[0036] The convolutional neural network module is used to selectively and automatically extract features from fundus images by fusing saliency maps and fundus images as input to the convolutional neural network.

[0037] The multilayer perceptron module is used to classify hypertension by fusing image features and non-image features from clinical metadata through a multilayer perceptron model.

[0038] According to another preferred embodiment of the present invention, the fundus color imaging data processing system based on the attention mechanism proposed in the present invention, such as Figure 1 As shown, it includes an image preprocessing module, a blood vessel feature extraction module, a convolutional neural network module, and a multilayer perceptron module.

[0039] (1) Image preprocessing

[0040] The preprocessing method for fundus color images in this invention mainly consists of region of interest extraction and contrast enhancement of fundus color images, such as... Figure 2 As shown.

[0041] Image preprocessing of fundus color images includes extraction of regions of interest (ROIs) and image cropping. A ROI is a subset of a fundus color image containing the desired information to be processed; a non-ROI is a subset of the image that does not contain the desired information. Without affecting the ROIs, non-ROIs should be removed as much as possible to reduce computational costs in image processing and to facilitate size standardization for images of different specifications. Therefore, the purpose of extracting ROIs in this invention is twofold: first, to avoid wasting time and computational resources due to an excessive number of non-ROIs; and second, to use the extracted non-ROIs as masks in subsequent data processing.

[0042] For fundus images obtained by fundus cameras, i.e., color fundus images, the circular area in the center is the part that needs to be focused on, i.e., the region of interest.

[0043] Extracting regions of interest typically employs traditional image processing methods such as segmentation and morphological processing, including manual thresholding, iterative thresholding, bimodal thresholding, and Otsu's method. Otsu's method is a segmentation method that automatically selects a threshold based on global information.

[0044] The region of interest extraction method used in this invention mainly includes the following steps:

[0045] Step S1110: Extract the red channel image data from the RBG color image of the fundus color photograph.

[0046] Step S1120: The image data of the red channel is segmented at a lower threshold to obtain a background segmentation map.

[0047] Specifically, the Otsu algorithm is preferred. Let u be the overall mean of the image data obtained in step S1110, and let B be the variance between the region of interest and the region of non-interest. Take a threshold t, and the mean obtained after extracting the region of interest is u0. Then the ratio of the number of pixels contained in the region of interest to the total number of pixels in the original image is w0. The mean obtained after extracting the region of non-interest is u1, and the corresponding ratio of the number of pixels in the region of non-interest is w1. At this time, w0 + w1 = 1 is satisfied.

[0048] At this point, the variance B between the two parts is expressed as follows:

[0049] B = w0(t) × (u0(t) - u) 2 +w1(t)×(u1(t)-u) 2

[0050] By iterating through the threshold t using the above formula, we can find the t that maximizes the variance B through a search. sThe value is used to obtain the final Otsu threshold, and then t is used to calculate the value. s The value is used as a threshold to divide the grayscale image into a binary image, thereby obtaining the best segmentation effect.

[0051] The non-interest regions extracted through the above steps are usually connected, and the extracted interest regions are usually circular. Therefore, morphological processing methods can be used to delete small areas of interest that may be misclassified, and opening and closing operations can be used to remove burrs, making the edges of the obtained binary mask smooth.

[0052] Therefore, step S1120 may further include using morphological methods to delete small connected regions, thereby removing areas in the foreground that may be misclassified as background. Finally, dilation and erosion operations are performed to smooth the field of view boundaries and obtain a more complete image of the non-interest-bearing region. The dilation and erosion operations preferably use a 5x5 kernel.

[0053] Step S1130: Extract the mask from the segmented region of interest image and the non-region of interest image.

[0054] The maximum bounding rectangle of the segmented, roughly circular region of interest (ROI) is extracted, and then expanded outwards by 10 pixels on each side of the ROI to improve the error tolerance of the extraction. Finally, binarization is performed using a threshold truncation method, where the background region value is set to 0 and the ROI value is set to 1, resulting in the final mask.

[0055] Image preprocessing of fundus color images also includes image enhancement. Image enhancement involves adding random information or performing a series of transformations on the original image to purposefully highlight regions of interest or contextual information relevant to a specific task, repair lost information, and suppress non-interesting or interfering regions. The specific purpose of image enhancement in this invention is to: expand the dataset to obtain image data with richer features; and highlight information more important to downstream tasks, such as the optic disc and vascular structures, through feature enhancement algorithms.

[0056] The method for enhancing fundus color photographs used in this invention mainly includes the following steps:

[0057] Step S1210: Convert the fundus color image to the LAB color space and extract the luminance (L) channel image data.

[0058] Step S1220: Use the CLAHE algorithm to enhance the contrast of the luminance channel image data.

[0059] In step S1230, the contrast-enhanced luminance channel image data obtained in step S1220 is recombined, and the image is converted back from the LAB color space to the RGB color space. The image is then cropped using the mask extracted in step S1130 of the preprocessing process to obtain the preprocessed image.

[0060] Preprocessing using the above method can significantly enhance the texture features in fundus photographs while preserving the color features of the image to the greatest extent.

[0061] (2) Extraction of blood vessel features

[0062] The vascular feature extraction module extracts retinal segmentation maps from fundus images and further converts them into saliency maps that characterize vascular distribution through post-processing. Figure 3 As shown, this method mainly consists of a network training phase and an application phase. In the training phase, the idea of ​​transfer learning is primarily utilized. The model is first trained and warmed up using a general public dataset, and then debugged using a small amount of private dataset to avoid poor feature extraction performance due to inconsistent data size and specifications when applied to subsequent screening tasks. In the application phase, the weights of the trained segmentation network selected in the previous phase are frozen, the threshold truncation operation at the network ends is removed, and a post-processing module is added to convert the refined vascular feature mapping information into a saliency map of a larger area that can characterize vascular distribution. Using the saliency map of vascular distribution, targeted saliency weights can be assigned to the vascular-rich parts of fundus images, thereby helping the downstream convolutional neural network to focus more on these regions.

[0063] This invention employs U-Net and its variants as the segmentation network. After training with refined fundus vascular annotations, the network is frozen and used as a feature extractor. Preferably, this invention uses an improved multi-scale Res-UNet segmentation network as the backbone network of the module. Training the network enables the model to obtain refined vascular feature maps, and a pooling-upsampling post-processing method converts the refined feature mapping information into a larger saliency map. The multi-scale Res-UNet segmentation network used in this invention adds residual shortening structures to the convolutional modules of the traditional U-Net network. While simplifying network training, the skip connections within the residual units ensure better integration of shallow and deep information, preventing gradient vanishing while adaptively learning the required scale of deep information. Specifically, in the traditional U-Net network, convolutions with a kernel size of 3 are typically used to connect the encoding and decoding parts. However, to extract more multi-scale information, this invention improves the bridging part to a dilated convolutional spatial pyramid pooling module (ASPP). The ASPP module performs parallel convolutions on the input using dilated convolutions with dilation scales of 6, 12, and 18 padding sizes, respectively. The resulting features are then concatenated along the channel dimension. Finally, a 1×1 convolution is used to fuse the features across multiple channel dimensions. By applying dilated convolutions with different dilation scales in parallel to the mapping of input features, feature information at different receptive scales can be obtained, thus improving the model's ability to capture multi-scale feature context.

[0064] The vascular feature extraction method of the present invention is as follows: Figure 4 As shown, the main steps include the following:

[0065] Step S2110: Slice the preprocessed fundus image obtained from the preprocessing process;

[0066] Step S2120: Use a pre-trained segmentation network to extract blood vessel features from the slice image;

[0067] Step S2130: Reconstruct the extracted vascular features to obtain a vascular feature map;

[0068] Step S2140: Post-process the vascular feature map to obtain a vascular distribution saliency map.

[0069] After the segmentation network is trained, the feature layers output by the network's end are post-processed. The post-processing operations after reconstructing the blood vessel feature map include pooling and upsampling operations. The pooling operation includes three average pooling operations on the blood vessel feature map, with kernel size of 4x4 and stride of 2, kernel size of 8x8 and stride of 2, and kernel size of 8x8 and stride of 8, respectively, pooling the image to a size of 16x16. The upsampling operation includes using a bilinear interpolation upsampling algorithm to stretch the grayscale image to the size of the input image.

[0070] In this invention, the average pixel density within a sliding window can be obtained through average pooling, thereby characterizing the density of blood vessels within that window; bilinear interpolation is used as an upsampling method to make the pixel changes between different windows smoother, thereby obtaining a more gentle saliency map that can characterize the global distribution of blood vessels.

[0071] The vascular feature map contains refined feature mapping information representing the vascular network at the original image scale. After post-processing using the method described above, this refined information can be converted into a saliency map covering a larger area. The main blood vessels and vascular intersections will obtain high saliency values. Therefore, in subsequent hypertension screening experiments, this feature map will be used to activate the original fundus color image, thereby guiding the saliency of the depth feature extraction module. This provides additional semantic prior knowledge for downstream depth information extraction, helping the model to focus more on important parts such as the main blood vessels and vascular intersections.

[0072] (3) Automatic feature extraction using convolutional neural networks

[0073] This invention employs a convolutional neural network (CNN) module to automatically extract features related to hypertension from fundus photographs. CNNs are a type of feedforward neural network, typically composed of convolutional layers, pooling layers, and fully connected layers. Specifically, the backbone network of the CNN module used in this invention is as follows: Figure 5 As shown, the main steps for automatically extracting features using a convolutional neural network are as follows:

[0074] Step S3110: Using the saliency map of blood vessel distribution obtained in step S2140, perform point-to-point activation multiplication on each color channel of the preprocessed fundus color image obtained in step S2130.

[0075] In this invention, the activation operation of fundus images using saliency maps is achieved by adjusting the weights of preprocessed fundus color images using saliency maps to obtain saliency-weighted activated image data I. SM , as input for downstream screening tasks.

[0076]

[0077] Wherein, SM is the vascular distribution significance map data obtained in step S2140 above; I k The data is the preprocessed fundus color image obtained in step S2130; k is the number of channels in the fundus color image, with 0, 1, and 2 representing RGB channels respectively; ⊙ represents the product at the pixel level; C is a constant matrix used to prevent information loss in areas with fewer blood vessels in the fundus image; Mask is the mask data obtained in step S1130.

[0078] Step S3120: The input data is passed through three convolutional layers with a kernel size of 3x3, and then input into a stacked ResNeSt module for feature extraction.

[0079] The ResNeSt module, for example Figure 6 As shown, the input feature map (input size H×W×C, where H, W, and C are the height, width, and color values ​​of each channel of the input data image, respectively) is split into K cardinals along the channel dimension. Within each cardinal, each feature input is further divided into R splits, resulting in G = KR branches. The number of feature map channels in each branch is C. k The algorithm applies two convolutions (conv) of 1×1 and 3×3 kernel sizes to each branch, respectively, to obtain G deep features in different channel dimensions. In each basis array, the outputs of the R branches serve as input to the Split Attention module. The outputs of the R branches are concatenated along the channel dimension and passed through a global pooling layer to obtain a global representation. Then, two fully connected (Dense) layers compress the feature maps along the channel dimension, obtaining attention factors for each sub-feature group. These R attention factors represent the importance of the corresponding branch in the channel dimension. After calculation using the r-Softmax function, the attention weight of each branch in the channel dimension of the basis array is calculated. Finally, the weights are multiplied by the corresponding branch outputs and summed to obtain the output of the basis array. This split attention mechanism ensures that each output channel is a weighted average of the corresponding channels of the input on different branches. Through model training, the contribution of each channel is automatically learned, forming a channel attention mechanism similar to SENet. Finally, the outputs of the K basis arrays are summed and passed through a 1×1 convolution kernel to return to a feature map of the same size as the original input. Then, the feature map is short-circuited with the original input feature map to form a residual module.

[0080] That is, after the fundus color image data is processed upstream, it is input into the convolutional neural network of the present invention, and then passes through three convolutional layers with a kernel size of 3×3 (accompanied by BN layer and ReLU activation function) and then input into the ResNeSt module for high-dimensional feature extraction. Then, the features are compressed by a global average pooling layer, and the high-dimensional depth features are output after dimensionality reduction by a fully connected layer for use as input to the subsequent classification module.

[0081] Step S3130: Feature compression is performed through a global average pooling layer, and finally a fully connected layer is used to compress the depth feature dimension to a size of 512x1.

[0082] Clinical studies have shown that hypertensive patients, due to long-term high blood pressure and significant blood pressure fluctuations, are prone to functional stenosis of the retinal arteries and a tendency towards arteriosclerosis. In fundus images, this typically manifests as subtle arterial diameter inconsistencies and arteriovenous crossing compression. Long-term uncontrolled hypertension can also lead to changes such as optic disc papilledema, hemorrhage, and reactive leukoplakia. In the saliency map generated by this invention, areas rich in blood vessels, such as the optic disc and main vessels, are assigned higher activation values. After weighted activation of preprocessed fundus color images using the saliency map, important areas rich in blood vessels, such as the main vessels and optic disc, receive higher activation weight values. Furthermore, the arteriovenous crossings after secondary bifurcation of blood vessels are also assigned higher activation weight values, thus guiding downstream models to focus more on these important areas. From a clinical perspective, these areas with high activation weight values ​​are also key areas of focus for physicians observing fundus images. Therefore, this method transforms the relatively fine vascular feature map after pixel-level segmentation into a saliency map of vascular distribution, which is equivalent to the key areas that professional physicians focus on when observing images. This achieves the goal of using prior knowledge to "guide" where the subsequent feature extraction module should "look".

[0083] During training, the convolutional neural network module uses backpropagation to automatically extract deep feature information that is highly correlated with hypertension from fundus color images. After activation through saliency maps, it can guide the convolutional neural network model to focus more on areas with larger gray values ​​(higher brightness) in the saliency maps, that is, areas with denser blood vessel distribution, especially structures such as optic discs, main blood vessels, and vascular bifurcation and intersection, and their distribution areas. This information is used as prior information to guide the convolutional neural network model to focus on learning areas that are more likely to be correlated with hypertension.

[0084] (4) Multilayer Perceptron Module

[0085] This invention further employs a multilayer perceptron module to fuse image features and non-image features for classification. A multilayer perceptron (MLP) is a neural network model based on the perceptron learning algorithm; in addition to the input and output layers, an MLP can contain multiple hidden layers.

[0086] The MLP structure used in this invention is as follows: Figure 7 As shown, this MLP consists of three hidden layers: the first hidden layer a contains 64 nodes and takes preprocessed metadata including clinical data as input, such as age, gender, height, weight, heart rate, and BMI. The feature map obtained in step S3130, after network processing, is compressed into a 512-dimensional feature map through a global average pooling layer and a fully connected layer, serving as the second part of the MLP input. The two inputs are concatenated and then passed through a hidden layer b with 512 nodes and a hidden layer c with 128 nodes. All these hidden layers use ReLU activation to achieve non-linear affine transformation. The MLP output layer contains two nodes, providing the final classification result.

[0087] MLPs are typically trained using backpropagation and supervised learning methods. Therefore, combining an MLP with an upstream convolutional neural network module allows for end-to-end training via gradient descent optimization. Furthermore, the activation function used in the MLP exhibits non-linear affine transformation characteristics, enabling the fusion of high-dimensional features from the convolutional neural network output with non-image features obtained from clinical metadata. During supervised learning, gradient updates are continuously applied to combine different features. After applying the Softmax function to the features of the two final output nodes of the MLP module, different predicted probabilities for different categories are obtained, with the larger one representing the final classification result.

[0088] Test results:

[0089] To validate the method and model of this invention, the THCS dataset used in the test was compiled from fundus images and related demographic information of hypertensive patients from hospitals. It includes 1507 fundus images from 822 individuals diagnosed with hypertension in northern China, and 2540 fundus images from 1340 non-hypertensive patients, along with clinical metadata such as age, sex, height, ethnicity, weight, and heart rate. The fundus images in the dataset are high-resolution color images of the fundus at 2464*2248 pixels, captured using a Canon CR-2 fundus camera with a 45° field of view. Community health checkup data provided by hospitals was also used as an additional supplementary dataset. This dataset also includes an expanded dataset from the Beijing community population, containing fundus images and a series of clinical metadata such as blood pressure, blood glucose, serum creatinine, and self-reported medical history. The fundus images include 1344 images from 699 suspected hypertensive positive samples.

[0090] Experimental results show that the present invention achieves an average AUC of 0.870, an average precision of 0.805, an average accuracy of 0.830, and an average recall of 0.761 on the aforementioned THCS dataset. Compared with methods using only convolutional neural networks, the average AUC, average precision, average accuracy, and average recall are improved by 6.8%, 4.2%, 14.9%, and 10.9%, respectively, significantly enhancing the early screening effect for hypertension.

[0091] To achieve the above objectives, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the above method.

[0092] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0093] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and units, such as the program units corresponding to the above-described method embodiments of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods described in the above-described method embodiments.

[0094] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] The one or more units are stored in the memory and, when executed by the processor, perform the methods described in the above embodiments.

[0096] The specific details of the aforementioned computer equipment can be understood by referring to the relevant descriptions and effects in the above embodiments, and will not be repeated here.

[0097] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed in a computer processor, implements the steps in the decentralized storage of personal health information described above. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0098] Obviously, those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination.

[0099] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the concept and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fundus color image processing device based on an attention mechanism, characterized in that: It includes an image preprocessing module, a blood vessel feature extraction module, and a convolutional neural network module; among which, The image preprocessing module includes a mask extraction module and an image enhancement module; The mask extraction module is used to extract a mask, which is used to segment the region of interest (ROI) and non-ROI images in a fundus color photograph. The mask extraction module extracts the image data of the red channel in the RGB color image of the fundus color photograph. The image data of the red channel is segmented to obtain a background segmentation map that is divided into two parts: the ROI and the non-ROI. The image enhancement module is used to enhance the contrast of fundus color photographs; The image preprocessing module uses the mask extracted by the mask extraction module to crop the fundus color photo after the image enhancement module has performed contrast enhancement to obtain a preprocessed fundus color photo image; while preserving the color features of the image, it significantly enhances the texture features in the fundus color photo. The vascular feature extraction module extracts a saliency map of vascular distribution based on the preprocessed fundus color image; The convolutional neural network module takes the saliency map of blood vessel distribution and the preprocessed fundus color image, performs point-to-point activation multiplication on each color channel, and then uses the result as the input to the convolutional neural network module; specifically, it uses the following formula: , = 0,1,2 in, Data for a saliency map of blood vessel distribution; This refers to the data of the preprocessed fundus color image; The number of channels in the fundus color image, with 0, 1, and 2 representing RGB channels respectively; Represents a product at the pixel level; It is a constant matrix; Data for the mask; To obtain the image data after saliency-weighted activation; The convolutional neural network module will obtain the image data after saliency-weighted activation. After processing by the convolutional layers, the data is input into a stacked ResNeSt module for feature extraction. The attention-based fundus color image processing device further includes a multilayer perceptron module; The multilayer perceptron module includes an MLP output layer and three sequentially connected hidden layers. The first hidden layer includes two input parts: the first input part takes preprocessed metadata as input, which includes one or more of age, gender, height, weight, heart rate, and BMI; the second input part takes the 512-dimensional feature map output by the convolutional neural network module as input; the first and second input parts are concatenated and then passed through a second hidden layer with 512 nodes and a third hidden layer with 128 nodes. The MLP output layer contains two nodes and provides the final classification result.

2. The fundus color imaging processing device based on an attention mechanism as described in claim 1, characterized in that: The features automatically extracted by the convolutional neural network module are image data features related to hypertension in fundus color photographs.

3. The fundus color imaging processing device based on an attention mechanism as described in claim 1, characterized in that: The ResNeSt module splits the input data of size H×W×C into K structurally consistent basis arrays in the channel dimension; in each basis array, each set of features is further input into R branches, thus obtaining G=KR branches; Each branch is convolved twice, with kernel sizes of 1×1 and 3×3, to obtain G deep features in different channel dimensions. In each basis array, the outputs of R branches are used as inputs to the split attention module. The outputs of R branches are concatenated along the channel dimension and passed through a global pooling layer to obtain a global representation. Then, two fully connected layers are used to compress the feature group in the channel dimension to obtain the attention factors of each branch feature group. These R attention factors represent the importance of the corresponding branch in the channel dimension. After calculation by the r-Softmax function, the attention weight of each branch in the channel dimension of the basis array is calculated. Finally, the weights are multiplied by the corresponding branch outputs and summed to obtain the output of the basis array. Where H, W, and C are the height, width, and color values ​​of each channel of the input data image, respectively, and K and R are positive integers greater than 2.

4. The fundus color imaging processing device based on an attention mechanism as described in claim 3, characterized in that: Based on the background segmentation map, extract the mask layers from the region of interest (ROI) image and the non-ROI image.

5. The fundus color imaging processing device based on an attention mechanism as described in claim 4, characterized in that: The region of interest is a roughly circular region, and the regions of non-interest are connected.

6. The fundus color imaging processing device based on an attention mechanism as described in claim 5, characterized in that: The image enhancement module transforms the fundus photograph to the LAB color space and extracts the image data of the luminance channel; it then uses the CLAHE algorithm to enhance the contrast of the luminance channel image data. The contrast-enhanced luminance channel image data is reconstructed and converted back from the LAB color space to the RGB color space.

7. A fundus photography processing system for early screening of hypertension, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the processor, the instructions being executed by the at least one processor to make the at least one processor function as an attention-based fundus color imaging processing device as described in any one of claims 1-6.

Images