Model training methods, devices, electronic equipment and storage media

Abstract

This application provides a model training method, apparatus, electronic device, and storage medium. The method includes: acquiring sample fundus images of a user that meet model training conditions, wherein the sample fundus images are labeled with the true disease category of a preset disease; inputting the sample fundus images into a disease prediction model to be trained, the disease prediction model to be trained including: a feature dynamic selection network layer and a classification layer; calling the feature dynamic selection network layer to perform multi-scale feature selection aggregation operation on the fundus feature map corresponding to the sample fundus images to obtain an aggregated feature map; calling the classification layer to process the aggregated feature map to obtain the user's predicted disease category; calculating the loss value of the disease prediction model to be trained based on the true disease category and the predicted disease category; and, if the loss value is within a preset range, using the trained disease prediction model as the final disease prediction model. This application can improve the detection efficiency and accuracy of diabetic nephropathy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, electronic device and storage medium. Background Technology

[0002] Diabetic nephropathy (DN) is one of the most common microvascular complications of diabetes. It is a chronic disease caused by diabetes and a major cause of chronic kidney disease and end-stage renal disease (kidney failure). Early-stage diabetic nephropathy often presents with no obvious symptoms, and is only detected through early diabetes screening with trace amounts of proteinuria. In the middle and late stages, the main symptoms are hypertension and edema. Physical examinations may reveal significant proteinuria, and some patients may experience anemia. It is also frequently accompanied by other microvascular complications, such as diabetic retinopathy, as well as peripheral, cardiovascular, and cerebrovascular complications. Therefore, identifying individuals with diabetic nephropathy is becoming increasingly important to ensure appropriate treatment, especially in the early stages when it is difficult to detect through physical examinations alone.

[0003] Fundus images are projections of the fundus onto a 2D plane, captured by a monocular camera. Unlike other eye scans (such as OCT images and angiography), fundus images can be acquired non-invasively and cost-effectively, making them more suitable for large-scale screening. Therefore, artificial intelligence, especially machine learning and deep learning, holds promise for providing better solutions for disease detection and prevention using fundus images.

[0004] In recent years, many studies have effectively utilized deep learning technology on fundus images to explore the feasibility of screening and predicting some diseases. However, current work has significant limitations. To date, no studies have been able to use deep learning technology combined with fundus images to predict diabetic nephropathy, mainly due to the following major problems:

[0005] 1. Diagnosis of diabetic nephropathy mainly relies on doctors to take postpartum medical history and conduct scale assessments. The test subjects need to spend a lot of time answering questions on the scale, which results in relatively low efficiency of diabetic nephropathy detection.

[0006] 2. Obvious features associated with diabetic nephropathy are difficult to observe in fundus images, thus reducing the accuracy of diabetic nephropathy prediction. Summary of the Invention

[0007] This application provides a model training method, apparatus, electronic device, and storage medium to address the problem of low efficiency and accuracy in the prior art for manual detection of diabetic nephropathy.

[0008] To solve the above-mentioned technical problems, the embodiments of this application are implemented as follows:

[0009] In a first aspect, embodiments of this application provide a model training method, the method comprising:

[0010] Acquire sample fundus images of users that meet the model training conditions, wherein the sample fundus images are labeled with the real disease category of a preset disease;

[0011] The sample fundus images are input into a disease prediction model to be trained, the disease prediction model to be trained includes: a feature dynamic selection network layer and a classification layer;

[0012] The feature dynamic selection network layer is invoked to perform multi-scale feature selection and aggregation operations on the fundus feature map corresponding to the sample fundus image to obtain an aggregated feature map.

[0013] The classification layer is invoked to process the aggregated feature map to obtain the user's predicted disease category;

[0014] Based on the actual disease category and the predicted disease category, the loss value of the disease prediction model to be trained is calculated;

[0015] If the loss value is within a preset range, the trained disease prediction model is used as the final disease prediction model.

[0016] Optionally, inputting the sample fundus images into the disease prediction model to be trained includes:

[0017] Obtain the task coding information corresponding to the sample fundus image, which is used to indicate the prediction of a specified disease category;

[0018] The sample fundus images and the task coding information are input into the disease prediction model to be trained.

[0019] Optionally, the disease prediction model to be trained further includes: a controller.

[0020] Before invoking the feature dynamic selection network layer to perform multi-scale feature selection aggregation on the fundus feature map corresponding to the sample fundus image to obtain the aggregated feature map, the method further includes:

[0021] The controller is invoked to filter out features in the fundus features of the sample fundus image that are irrelevant to the specified disease category based on the task encoding information, thereby obtaining the fundus feature map.

[0022] Optionally, the feature dynamic selection network layer includes: a feature extraction layer, a weight calculation layer, and a feature aggregation layer.

[0023] The step of calling the feature dynamic selection network layer to perform multi-scale feature selection and aggregation operations on the fundus feature map corresponding to the sample fundus image to obtain an aggregated feature map includes:

[0024] The feature extraction layer is invoked to perform multi-scale feature extraction on the fundus feature map to obtain multi-scale fundus features;

[0025] The weight calculation layer is invoked to process the multi-scale fundus features to obtain the feature weights corresponding to the multi-scale fundus features;

[0026] The feature aggregation layer is invoked to aggregate the multi-scale fundus features according to the feature weights to obtain the aggregated feature map.

[0027] Optionally, acquiring sample fundus images of the user that meet the model training conditions includes:

[0028] Acquire multiple initial fundus images of the user;

[0029] The initial fundus image is processed by a preset model to filter out sample fundus images that meet the model training conditions. The sample fundus images are fundus images of the real fundus image category and whose image quality is greater than a quality threshold.

[0030] Optionally, the step of processing the initial fundus image using a preset model to filter out sample fundus images from the initial fundus image that meet the model training conditions includes:

[0031] The initial fundus image is preprocessed to generate a preprocessed fundus image;

[0032] The preprocessed fundus image is processed by calling the fundus image recognition model to obtain the fundus image recognition result;

[0033] Based on the fundus image recognition results, a standard fundus image is determined from the initial fundus image;

[0034] The preprocessed fundus image is processed by calling an image quality level classification model to obtain the probability that the preprocessed fundus image belongs to each preset quality level;

[0035] Based on the probability, sample fundus images that meet the model training conditions are selected from the standard fundus images.

[0036] Optionally, after setting the trained disease prediction model as the final disease prediction model when the loss value is within a preset range, the method further includes:

[0037] Acquire target fundus images of the target user that meet the preset disease detection criteria;

[0038] The target fundus image is input into the disease prediction model;

[0039] The feature dynamic selection network layer is invoked to perform multi-scale feature selection and aggregation operations on the target fundus feature map corresponding to the target fundus image to obtain the target feature map.

[0040] The classification layer is invoked to process the target feature map to obtain a prediction result of whether the user has a preset disease.

[0041] Secondly, embodiments of this application provide a model training apparatus, the apparatus comprising:

[0042] The sample image acquisition module is used to acquire sample fundus images of the user that meet the model training conditions. The sample fundus images are labeled with the real disease category of the preset disease.

[0043] The sample image input module is used to input the sample fundus image into the disease prediction model to be trained. The disease prediction model to be trained includes: a feature dynamic selection network layer and a classification layer.

[0044] The feature map acquisition module is used to call the feature dynamic selection network layer to perform multi-scale feature selection and aggregation operation on the fundus feature map corresponding to the sample fundus image to obtain an aggregated feature map.

[0045] The predicted category acquisition module is used to call the classification layer to process the aggregated feature map to obtain the predicted disease category of the user;

[0046] The loss value calculation module is used to calculate the loss value of the disease prediction model to be trained based on the real disease category and the predicted disease category.

[0047] The prediction model acquisition module is used to use the trained disease prediction model as the final disease prediction model when the loss value is within a preset range.

[0048] Optionally, the sample image input module includes:

[0049] The encoding information acquisition unit is used to acquire task encoding information corresponding to the sample fundus image, which is used to indicate the prediction of a specified disease category;

[0050] An image encoding input unit is used to input the sample fundus image and the task encoding information into the disease prediction model to be trained.

[0051] Optionally, the disease prediction model to be trained further includes: a controller.

[0052] The device further includes:

[0053] The fundus feature map acquisition module is used to call the controller to filter out features in the fundus features of the sample fundus image that are not related to the specified disease category according to the task encoding information, so as to obtain the fundus feature map.

[0054] Optionally, the feature dynamic selection network layer includes: a feature extraction layer, a weight calculation layer, and a feature aggregation layer.

[0055] The feature map acquisition module includes:

[0056] The multi-scale feature acquisition unit is used to call the feature extraction layer to perform multi-scale feature extraction on the fundus feature map to obtain multi-scale fundus features.

[0057] The feature weight acquisition unit is used to call the weight calculation layer to process the multi-scale fundus features and obtain the feature weights corresponding to the multi-scale fundus features.

[0058] The aggregated feature map acquisition unit is used to call the feature aggregation layer to aggregate the multi-scale fundus features according to the feature weights to obtain the aggregated feature map.

[0059] Optionally, the sample image acquisition module includes:

[0060] The fundus image acquisition unit is used to acquire multiple initial fundus images of the user;

[0061] The sample image filtering unit is used to call a preset model to process the initial fundus image in order to filter out the sample fundus images that meet the model training conditions from the initial fundus image. The sample fundus images are fundus images of the real fundus image category and whose image quality is greater than a quality threshold.

[0062] Optionally, the sample image screening unit includes:

[0063] The fundus image generation subunit is used to preprocess the initial fundus image to generate a preprocessed fundus image;

[0064] The recognition result acquisition subunit is used to call the fundus image recognition model to process the preprocessed fundus image and obtain the fundus image recognition result;

[0065] A standard image determination subunit is used to determine a standard fundus image in the initial fundus image based on the fundus image recognition result;

[0066] The probability acquisition subunit is used to call the image quality level classification model to process the preprocessed fundus image and obtain the probability that the preprocessed fundus image belongs to each preset quality level.

[0067] The sample image filtering subunit is used to filter sample fundus images that meet the model training conditions from the standard fundus images according to the probability.

[0068] Optionally, the device further includes:

[0069] The target image acquisition module is used to acquire target fundus images of the target user that meet the preset disease detection conditions.

[0070] The target image input module is used to input the target fundus image into the disease prediction model;

[0071] The target feature map acquisition module is used to call the feature dynamic selection network layer to perform multi-scale feature selection and aggregation operations on the target fundus feature map corresponding to the target fundus image to obtain the target feature map.

[0072] The prediction result acquisition module is used to call the classification layer to process the target feature map and obtain a prediction result of whether the user has a preset disease.

[0073] Thirdly, embodiments of this application provide an electronic device, including:

[0074] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the model training method described in any of the preceding claims.

[0075] Fourthly, embodiments of this application provide a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform any of the model training methods described above.

[0076] In this embodiment, sample fundus images of the user that meet the model training conditions are acquired. These sample fundus images are labeled with the true disease category of a preset disease. The sample fundus images are input into the disease prediction model to be trained. The disease prediction model to be trained includes a feature dynamic selection network layer and a classification layer. The feature dynamic selection network layer is invoked to perform multi-scale feature selection and aggregation operations on the fundus feature map corresponding to the sample fundus image, resulting in an aggregated feature map. The classification layer is invoked to process the aggregated feature map to obtain the user's predicted disease category. Based on the true disease category and the predicted disease category, the loss value of the disease prediction model to be trained is calculated. If the loss value is within a preset range, the trained disease prediction model is used as the final disease prediction model. This embodiment of the application improves the detection efficiency and accuracy of diabetic nephropathy by extracting multi-scale features from fundus images and using a feature dynamic selection mechanism to extract features related to diabetic nephropathy for model training, compared to manual detection methods.

[0077] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0078] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 A flowchart illustrating the steps of a model training method provided in this application embodiment;

[0080] Figure 2 A schematic diagram of a disease prediction process provided in an embodiment of this application;

[0081] Figure 3 A schematic diagram of a ResNet-50 network structure provided in an embodiment of this application;

[0082] Figure 4 This is a schematic diagram of a conventional multi-task learning structure provided in an embodiment of this application;

[0083] Figure 5 This is a schematic diagram of a multi-task learning structure provided in an embodiment of this application;

[0084] Figure 6 A schematic diagram of a multi-scale network with dynamic convolutional kernels provided in an embodiment of this application;

[0085] Figure 7 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0086] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0087] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0088] Reference Figure 1The flowchart illustrates the steps of a model training method provided in an embodiment of this application, as follows: Figure 1 As shown, the model training method may include steps 101, 102, 103, 104, 105, and 106.

[0089] Step 101: Obtain sample fundus images of the user that meet the model training conditions, wherein the sample fundus images are labeled with the real disease category of the preset disease.

[0090] In the embodiments of this application, the model training conditions refer to the conditions for selecting fundus images of fundus image categories and with image quality greater than a quality threshold as model training samples.

[0091] The true disease category refers to the category manually labeled on the sample fundus image. In practical applications, business personnel can add specified numbers to the image to indicate the true disease category of the sample fundus image.

[0092] When training a disease prediction model for a preset disease, sample fundus images of the user that meet the model training conditions can be acquired. Specifically, multiple fundus images of the user can be acquired, and then a fundus image recognition model can be used to identify the standard fundus images among them. Finally, an image quality classification model is called to classify the identified standard fundus images by image quality, and the highest quality standard fundus image is selected as the sample fundus images. This implementation process can be described in detail below with reference to the specific implementation method.

[0093] In one specific implementation of this application, step 101 may include:

[0094] Sub-step A1: Obtain multiple initial fundus images of the user.

[0095] In this embodiment, multiple initial fundus images of the user can be acquired during the training of the disease prediction model. In this example, the initial fundus images can be fundus images taken by a medical institution or fundus images taken by the user using specific fundus image capturing equipment. Specifically, the method of acquiring the initial fundus images can be determined according to business needs, and this embodiment does not impose any restrictions on it.

[0096] After obtaining multiple initial fundus images of the user, sub-step A2 is executed.

[0097] Sub-step A2: Call the preset model to process the initial fundus image to filter out the sample fundus images in the initial fundus image that meet the model training conditions, wherein the sample fundus images are fundus images of the real fundus image category and whose image quality is greater than the quality threshold.

[0098] The sample fundus images can be fundus images of the real fundus image category and whose image quality is greater than the quality threshold.

[0099] After acquiring multiple initial fundus images from the user, a preset model can be invoked to process these images, thereby filtering out sample fundus images that meet the model's training conditions. Specifically, the preset model can be a fundus image recognition model and an image quality level classification model. These two models work together to filter out sample fundus images that meet the model's training conditions. This implementation process can be described in detail below.

[0100] In another specific implementation of this application, the above sub-step A2 may include:

[0101] Sub-step B1: Preprocess the initial fundus image to generate a preprocessed fundus image.

[0102] In this embodiment, after obtaining the initial fundus image, the initial fundus image can be preprocessed to generate a preprocessed fundus image. Specifically, denoising and deduplication operations can be performed on the initial fundus image to obtain the preprocessed fundus image.

[0103] After preprocessing the initial fundus image to generate a preprocessed image, sub-steps B2 and B4 are executed.

[0104] Sub-step B2: Call the fundus image recognition model to process the preprocessed fundus image and obtain the fundus image recognition result.

[0105] After preprocessing the initial fundus image to generate a preprocessed image, a fundus image recognition model can be called to process the preprocessed fundus image to obtain the fundus image recognition result. Specifically, the preprocessed image can be input into the fundus image recognition model so that the fundus image recognition model can recognize the preprocessed fundus image to obtain the fundus image recognition result.

[0106] After the fundus image recognition model is called to process the preprocessed fundus image to obtain the fundus image recognition result, sub-step B3 is executed.

[0107] Sub-step B3: Based on the fundus image recognition results, determine the standard fundus image in the initial fundus image.

[0108] After processing the preprocessed fundus image using a fundus image recognition model to obtain the fundus image recognition result, the standard fundus image in the initial fundus image can be determined based on the fundus image recognition result. Specifically, non-fundus images, such as facial images or images without fundus features, can be removed from the initial fundus image using the fundus image recognition result. After removing the non-fundus images from the initial fundus image, the remaining initial fundus image is the standard fundus image.

[0109] After determining the standard fundus image in the initial fundus image based on the fundus image recognition results, proceed to sub-step B5.

[0110] Sub-step B4: Call the image quality level classification model to process the preprocessed fundus image and obtain the probability that the preprocessed fundus image belongs to each preset quality level.

[0111] The preset quality levels can be divided into three levels: non-fundus image level, blurred fundus image level, and high-quality fundus image level.

[0112] After preprocessing the initial fundus image to generate a preprocessed image, an image quality classification model can be used to process the preprocessed fundus image and obtain the probability that the preprocessed fundus image belongs to each preset quality level. Specifically, the image quality classification model can extract image features from the preprocessed fundus image and classify these features to obtain the probability that the preprocessed fundus image belongs to a preset quality level. That is, the image quality classification model can output the probability that the preprocessed fundus image belongs to the non-fundus image level, the blurred fundus image level, and the high-quality fundus image level.

[0113] After processing the preprocessed fundus image using the image quality level classification model and obtaining the probability that the preprocessed fundus image belongs to each preset quality level, sub-step B5 is executed.

[0114] Sub-step B5: Based on the probability, select sample fundus images that meet the model training conditions from the standard fundus images.

[0115] After obtaining the probabilities of preprocessed fundus images belonging to various preset quality levels, sample fundus images that meet the model training conditions can be selected from standard fundus images based on these probabilities. Specifically, the quality level of the preprocessed fundus image can be determined based on its probabilities, and then standard fundus images belonging to high-quality levels are selected as sample fundus images for subsequent disease prediction model training.

[0116] This application embodiment combines a fundus image recognition model and an image quality level classification model to obtain a more accurate fundus image quality classification while filtering out interference from non-fundus images, thereby improving the accuracy of fundus image quality control and thus improving the detection accuracy of the trained disease prediction model.

[0117] In this embodiment, the sample fundus image for a single user can be one image or multiple images. Specifically, the number of fundus images for a single user can be determined according to the actual situation, and this embodiment does not impose any restrictions on this.

[0118] After obtaining sample fundus images of the user that meet the model training conditions, step 102 is executed.

[0119] Step 102: Input the sample fundus image into the disease prediction model to be trained. The disease prediction model to be trained includes a feature dynamic selection network layer and a classification layer.

[0120] After obtaining sample fundus images of the user that meet the model training conditions, the sample fundus images can be input into the disease prediction model to be trained. The disease prediction model to be trained may include: a feature dynamic selection network layer and a classification layer.

[0121] Among them, the feature dynamic selection network layer can adopt a dynamic selection mechanism so that the model can automatically utilize the information captured by the receptive field in the fundus image that is effective for classifying diabetic nephropathy.

[0122] Categorical layers can be used to predict disease categories.

[0123] In practical applications, conventional multi-task learning methods can be such as Figure 4 As shown, it consists of a shared backbone network and multiple task-specific classification layers. During the training phase, each part of the data needs to be labeled before being input into the network. Therefore, conventional methods have low data utilization, wasting a lot of data that could have been used to train classification layers for other tasks. In addition, the inflexible multi-head architecture is not easily scaled to a newly labeled task.

[0124] Based on this, the embodiments of this application employ a dynamic on-demand network, which can be trained on partially labeled datasets; it is a single-head dynamic network, such as... Figure 5 As shown, it can predict multiple tasks simultaneously, just like multiple networks or multi-head networks (i.e., the conventional multi-task learning methods described above). The kernel of the dynamic head is adaptively generated by the controller, conditioned on the input image and the assigned task. Specifically, task-specific prior information is provided to the controller to guide the generation of segmented task dynamic head kernels. Due to the lightweight design of the dynamic head, the computational cost of repeated inference is negligible compared to multi-head networks.

[0125] Each labeled dataset contains key prior information (i.e., task encoding information), which tells the model which task it is handling and which region it should focus on. Intuitively, this task encoding is input into the model to achieve task awareness. The task encoding information is shown in Table 1 below:

[0126] Table 1: Task Coding

[0127] Task coding Diabetic nephropathy [1,0,0,0,0,0] eGFR [0,1,0,0,0,0] blood pressure [0,0,1,0,0,0] age [0,0,0,1,0,0] Proteinuria [0,0,0,0,1,0] Diabetic retinopathy [0,0,0,0,0,1]

[0128] As shown in Table 1 above, the codes corresponding to the diabetic nephropathy task are [1,0,0,0,0,0], the codes corresponding to the eGFR task are [0,1,0,0,0,0], the codes corresponding to the age task are [0,0,0,1,0,0], and so on.

[0129] Understandably, during the model training process using batch sample fundus images, the task coding information for each batch of samples can be only one task coding information or multiple task coding information. Specifically, the number of task coding information for each batch can be determined according to business needs, and this embodiment does not impose any restrictions on this.

[0130] In practice, task coding information corresponding to the sample fundus image can be obtained to indicate the prediction of a specific disease category. Then, the sample fundus image and the task coding information can be input together into the disease prediction model to be trained.

[0131] After inputting the sample fundus images into the disease prediction model to be trained, step 103 is executed.

[0132] Step 103: Call the feature dynamic selection network layer to perform multi-scale feature selection aggregation operation on the fundus feature map corresponding to the sample fundus image to obtain the aggregated feature map.

[0133] In this embodiment, the disease prediction model to be trained may include: a feature dynamic selection network layer and a classification layer.

[0134] Feature dynamic selection network layers can be like Figure 6 As shown, this network layer can employ a dynamic kernel selection mechanism to enable the model to automatically utilize information captured by receptive fields effective for classifying diabetic nephropathy in fundus images. In this example, a building block of selective kernel units is designed in the disease prediction model to be trained, where multiple branches with different kernel sizes are fused using softmax, guided by information from these branches. Therefore, the disease prediction model to be trained consists of multiple selective kernel units, where neurons are capable of capturing target objects of different scales.

[0135] After inputting the sample fundus images into the disease prediction model to be trained, the feature dynamic selection network layer can be called to perform multi-scale feature selection and aggregation operations on the fundus feature maps corresponding to the sample fundus images to obtain aggregated feature maps.

[0136] The fundus feature map can be the feature map output by the controller of the disease prediction model to be trained. The processing of the controller can be described in detail in conjunction with the specific implementation method below.

[0137] In one specific implementation of this application, before step 103 above, the following may also be included:

[0138] Step C1: Call the controller to filter out features in the fundus features of the sample fundus image that are not related to the specified disease category according to the task encoding information, and obtain the fundus feature map.

[0139] In this embodiment, after inputting task encoding information and sample fundus images into the disease prediction model to be trained, a controller can be invoked to filter out features in the fundus features of the sample fundus images that are irrelevant to the specified disease category, thereby obtaining a fundus feature map. Specifically, the controller can be a single convolutional layer. Image features are aggregated through global average pooling and concatenated with the task encoding vector before being input into the final classification layer of the model. During this process, the task encoding information can filter out regions in the fundus images that are irrelevant to the specified disease category indicated by the task encoding, making the model pay more attention to image regions related to the task encoding information.

[0140] The dynamic on-demand network used in this application differs from conventional multi-task learning methods. It makes full use of all data (including partially labeled data) and dynamically generates prediction results for each task based on the input task code.

[0141] After obtaining the fundus feature map of the sample fundus image, the dynamic feature selection network layer can be invoked to perform multi-scale feature selection and aggregation operations on the fundus feature map corresponding to the sample fundus image, resulting in an aggregated feature map. Specifically, the dynamic feature selection network layer can include three operations: Split, Fuse, and Select. The specific implementation process of each operation can be described in detail below in conjunction with the specific implementation method.

[0142] In another specific implementation of this application, step 103 may include:

[0143] Sub-step D1: Call the feature extraction layer to perform multi-scale feature extraction on the fundus feature map to obtain multi-scale fundus features.

[0144] In this embodiment, the feature extraction layer can perform a split operation on the fundus feature map corresponding to the sample fundus image, that is, perform multi-scale feature extraction operation on the fundus feature map to obtain multi-scale fundus features.

[0145] In this example, the feature extraction layer can consist of N convolutional kernels of different scales, each kernel extracting fundus features at one scale from the fundus feature map, such as... Figure 6 As shown, the model is designed with two convolutional kernels of different scales. In practical applications, multiple convolutional kernels with multiple branches can be designed.

[0146] After calling the feature extraction layer to perform multi-scale feature extraction layer operations on the fundus feature map to obtain multi-scale fundus features, sub-step D2 is executed.

[0147] Sub-step D2: Call the weight calculation layer to process the multi-scale fundus features and obtain the feature weights corresponding to the multi-scale fundus features.

[0148] After performing multi-scale feature extraction on the fundus feature map using the feature extraction layer to obtain multi-scale fundus features, the weight calculation layer can be called to process these multi-scale fundus features and obtain the feature weights corresponding to them. For example... Figure 6 As shown, the weight calculation layer can perform aggregation and weight calculation operations on multi-scale fundus features. Specifically, it can aggregate information from multiple paths to obtain a global and comprehensive representation for weight selection. In practical applications, the weight calculation layer has its own attention mechanism, through which the feature weights corresponding to fundus features at different scales can be calculated.

[0149] After calling the weight calculation layer to process the multi-scale fundus features and obtain the feature weights corresponding to the multi-scale fundus features, sub-step D3 is executed.

[0150] Sub-step D3: Call the feature aggregation layer to aggregate the multi-scale fundus features according to the feature weights to obtain the aggregated feature map.

[0151] After processing the multi-scale fundus features using the weight calculation layer to obtain the feature weights corresponding to the multi-scale fundus features, the feature aggregation layer can be called to aggregate the multi-scale fundus features based on the feature weights, resulting in an aggregated feature map. For example... Figure 6 As shown, the feature aggregation layer can perform a select operation to aggregate feature maps of different kernel sizes according to the selection weights, thus obtaining an aggregated feature map.

[0152] This application embodiment uses a multi-scale network with dynamic convolutional kernels as the backbone network for diabetic nephropathy detection. It can adaptively adjust the size of its receptive field (convolutional kernel) according to the multi-scale of the input information, and extract lesion-related features in fundus images more accurately.

[0153] After calling the feature dynamic selection network layer to perform multi-scale feature selection and aggregation operations on the fundus feature maps corresponding to the sample fundus images to obtain aggregated feature maps, step 104 is executed.

[0154] Step 104: Call the classification layer to process the aggregated feature map to obtain the user's predicted disease category.

[0155] After calling the feature dynamic selection network layer to perform multi-scale feature selection and aggregation operations on the fundus feature maps corresponding to the sample fundus images to obtain the aggregated feature maps, the classification layer can be called to process the aggregated feature maps to obtain the user's predicted disease category.

[0156] After calling the classification layer to process the aggregated feature map to obtain the user's predicted disease category, step 105 is executed.

[0157] Step 105: Calculate the loss value of the disease prediction model to be trained based on the real disease category and the predicted disease category.

[0158] After the classification layer processes the aggregated feature map to obtain the user's predicted disease category, the loss value of the disease prediction model to be trained can be calculated based on the real disease category and the predicted disease category.

[0159] In this example, the loss value of the disease prediction model to be trained is shown in the following formula (1):

[0160]

[0161] In formula (1) above, α and β are pre-set constants, and α>1, β<1. cls Let L be the cross-entropy loss function. rg The MSE loss function is used to calculate the predicted class probability p. i * and the actual category label p i Calculate the loss value, L DN L eGFR L BP L age L PRO L DR The loss values ​​are respectively for six tasks: diabetic nephropathy, eGFR, blood pressure, age, proteinuria, and diabetic retinopathy.

[0162] The cross-entropy loss function and the MSE loss function can be calculated as shown in formulas (2) and (3) below, respectively:

[0163]

[0164]

[0165] Understandably, the loss value calculation in the above formula (1) is performed when the model training task is six tasks: diabetic nephropathy, eGFR, blood pressure, age, proteinuria, and diabetic retinopathy. When the model training task is not all six tasks mentioned above, the loss value of the corresponding task in the above formula (1) can be reduced accordingly.

[0166] After calculating the loss value of the disease prediction model to be trained based on the real disease category and the predicted disease category, proceed to step 106.

[0167] Step 106: If the loss value is within a preset range, the trained disease prediction model is used as the final disease prediction model.

[0168] After calculating the loss value of the disease prediction model to be trained based on the real disease category and the predicted disease category, it can be determined whether the loss value is within the preset range.

[0169] If the loss value is not within the preset range, it means that the disease prediction model to be trained has not converged and cannot meet the preset disease prediction requirements. In this case, the disease prediction model to be trained can be continued to be trained by combining sample fundus images.

[0170] If the loss value is within the preset range, it means that the disease prediction model to be trained has converged. At this time, the trained disease prediction model can be used as the final disease prediction model, which can be used in subsequent prediction scenarios of preset diseases (such as diabetes, kidney disease, etc.).

[0171] The disease prediction model trained in this application can be applied to medical institutions (such as hospitals) to assist healthcare workers in determining whether patients have diabetes or kidney disease. It can also be applied in home and work settings to assist in determining whether family members or employees have diabetes or kidney disease. This application, through a deep learning-based method for predicting diabetic nephropathy from fundus images, overcomes the limitation of traditional methods that struggle to directly identify disease characteristics from fundus images, significantly improving the prediction accuracy for diabetic nephropathy.

[0172] After training the disease prediction model, it can be used to detect whether a user has a preset disease (such as diabetes, kidney disease, etc.). The model inference process can be described in detail below with reference to the specific implementation method.

[0173] In one specific implementation of this application, after step 106 above, the following may also be included:

[0174] Step E1: Obtain the target fundus image of the target user that meets the preset disease detection conditions.

[0175] In this embodiment of the application, when predicting a preset disease for a target user, a target fundus image of the target user that meets the preset disease detection conditions can be obtained.

[0176] The preset disease detection conditions are similar to the model training conditions described above, namely, selecting fundus images of the true fundus image category and with image quality greater than a quality threshold from multiple fundus images of the target user, and using these as the target fundus images. For example... Figure 2 As shown, when predicting a preset disease, a fundus image can be input into the quality control model, which will then filter out the target fundus image.

[0177] After obtaining the target fundus image of the target user that meets the preset disease detection conditions, step E2 is executed.

[0178] Step E2: Input the target fundus image into the disease prediction model.

[0179] After obtaining the target fundus image of the target user that meets the preset disease detection conditions, the target fundus image can be input into the disease prediction model trained in the above process.

[0180] After inputting the target fundus image into the disease prediction model, step E3 is executed.

[0181] Step E3: Call the feature dynamic selection network layer to perform multi-scale feature selection and aggregation operation on the target fundus feature map corresponding to the target fundus image to obtain the target feature map.

[0182] After inputting the target fundus image into the disease prediction model, the feature dynamic selection network layer can be invoked to perform multi-scale feature selection and aggregation operations on the target fundus feature map corresponding to the target fundus image, thereby obtaining the target feature map. This implementation process is similar to the description of step 103 above, and will not be repeated here in this embodiment.

[0183] After calling the feature dynamic selection network layer to perform multi-scale feature selection and aggregation operations on the target fundus feature map corresponding to the target fundus image to obtain the target feature map, step E4 is executed.

[0184] Step E4: Call the classification layer to process the target feature map to obtain a prediction result of whether the user has a preset disease.

[0185] After calling the feature dynamic selection network layer to perform multi-scale feature selection and aggregation on the target fundus feature map corresponding to the target fundus image to obtain the target feature map, the classification layer can then be called to process the target feature map to obtain a prediction result of whether the user has a preset disease. For example... Figure 2 As shown, after the target fundus image is selected through the quality control model, it can be input into the prediction network to obtain the prediction result.

[0186] This application embodiment employs a feature dynamic selection mechanism to extract features related to diabetic nephropathy for model training, which can improve the detection efficiency and accuracy of diabetic nephropathy compared to manual detection methods.

[0187] The model training method provided in this application involves acquiring sample fundus images of users that meet the model training conditions. These sample fundus images are labeled with the true disease category of a preset disease. The sample fundus images are input into the disease prediction model to be trained. The disease prediction model to be trained includes a feature dynamic selection network layer and a classification layer. The feature dynamic selection network layer is invoked to perform multi-scale feature selection and aggregation operations on the fundus feature map corresponding to the sample fundus image, resulting in an aggregated feature map. The classification layer is invoked to process the aggregated feature map to obtain the user's predicted disease category. Based on the true disease category and the predicted disease category, the loss value of the disease prediction model to be trained is calculated. If the loss value is within a preset range, the trained disease prediction model is used as the final disease prediction model. This application embodiment, by extracting multi-scale features from fundus images and employing a feature dynamic selection mechanism to extract features related to diabetic nephropathy for model training, can improve the detection efficiency and accuracy of diabetic nephropathy compared to manual detection methods.

[0188] Reference Figure 7 The diagram shows a schematic representation of a model training device provided in an embodiment of this application. Figure 7 As shown, the model training device 700 may include the following modules:

[0189] The sample image acquisition module 710 is used to acquire sample fundus images of the user that meet the model training conditions, wherein the sample fundus images are labeled with the real disease category of a preset disease;

[0190] The sample image input module 720 is used to input the sample fundus image into the disease prediction model to be trained. The disease prediction model to be trained includes: a feature dynamic selection network layer and a classification layer.

[0191] The feature map acquisition module 730 is used to call the feature dynamic selection network layer to perform multi-scale feature selection and aggregation operation on the fundus feature map corresponding to the sample fundus image to obtain an aggregated feature map.

[0192] The predicted category acquisition module 740 is used to call the classification layer to process the aggregated feature map to obtain the predicted disease category of the user;

[0193] The loss value calculation module 750 is used to calculate the loss value of the disease prediction model to be trained based on the real disease category and the predicted disease category.

[0194] The prediction model acquisition module 760 is used to use the trained disease prediction model as the final disease prediction model when the loss value is within a preset range.

[0195] Optionally, the sample image input module includes:

[0196] The encoding information acquisition unit is used to acquire task encoding information corresponding to the sample fundus image, which is used to indicate the prediction of a specified disease category;

[0197] An image encoding input unit is used to input the sample fundus image and the task encoding information into the disease prediction model to be trained.

[0198] Optionally, the disease prediction model to be trained further includes: a controller.

[0199] The device further includes:

[0200] The fundus feature map acquisition module is used to call the controller to filter out features in the fundus features of the sample fundus image that are not related to the specified disease category according to the task encoding information, so as to obtain the fundus feature map.

[0201] Optionally, the feature dynamic selection network layer includes: a feature extraction layer, a weight calculation layer, and a feature aggregation layer.

[0202] The feature map acquisition module includes:

[0203] The multi-scale feature acquisition unit is used to call the feature extraction layer to perform multi-scale feature extraction on the fundus feature map to obtain multi-scale fundus features.

[0204] The feature weight acquisition unit is used to call the weight calculation layer to process the multi-scale fundus features and obtain the feature weights corresponding to the multi-scale fundus features.

[0205] The aggregated feature map acquisition unit is used to call the feature aggregation layer to aggregate the multi-scale fundus features according to the feature weights to obtain the aggregated feature map.

[0206] Optionally, the sample image acquisition module includes:

[0207] The fundus image acquisition unit is used to acquire multiple initial fundus images of the user;

[0208] The sample image filtering unit is used to call a preset model to process the initial fundus image in order to filter out the sample fundus images that meet the model training conditions from the initial fundus image. The sample fundus images are fundus images of the real fundus image category and whose image quality is greater than a quality threshold.

[0209] Optionally, the sample image screening unit includes:

[0210] The fundus image generation subunit is used to preprocess the initial fundus image to generate a preprocessed fundus image;

[0211] The recognition result acquisition subunit is used to call the fundus image recognition model to process the preprocessed fundus image and obtain the fundus image recognition result;

[0212] A standard image determination subunit is used to determine a standard fundus image in the initial fundus image based on the fundus image recognition result;

[0213] The probability acquisition subunit is used to call the image quality level classification model to process the preprocessed fundus image and obtain the probability that the preprocessed fundus image belongs to each preset quality level.

[0214] The sample image filtering subunit is used to filter sample fundus images that meet the model training conditions from the standard fundus images according to the probability.

[0215] Optionally, the device further includes:

[0216] The target image acquisition module is used to acquire target fundus images of the target user that meet the preset disease detection conditions.

[0217] The target image input module is used to input the target fundus image into the disease prediction model;

[0218] The target feature map acquisition module is used to call the feature dynamic selection network layer to perform multi-scale feature selection and aggregation operations on the target fundus feature map corresponding to the target fundus image to obtain the target feature map.

[0219] The prediction result acquisition module is used to call the classification layer to process the target feature map and obtain a prediction result of whether the user has a preset disease.

[0220] The model training apparatus provided in this application acquires sample fundus images from users that meet the model training conditions. These sample fundus images are labeled with the true disease category of a preset disease. The sample fundus images are input into a disease prediction model to be trained. The disease prediction model includes a dynamic feature selection network layer and a classification layer. The dynamic feature selection network layer performs multi-scale feature selection and aggregation operations on the fundus feature map corresponding to the sample fundus image to obtain an aggregated feature map. The classification layer processes the aggregated feature map to obtain the user's predicted disease category. Based on the true disease category and the predicted disease category, the loss value of the disease prediction model to be trained is calculated. If the loss value is within a preset range, the trained disease prediction model is used as the final disease prediction model. This application embodiment, by extracting multi-scale features from fundus images and employing a dynamic feature selection mechanism to extract features related to diabetic nephropathy for model training, can improve the detection efficiency and accuracy of diabetic nephropathy compared to manual detection methods.

[0221] Additionally, this application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the above-described model training method when executed by the processor.

[0222] Figure 8 A schematic diagram of the structure of an electronic device 800 according to an embodiment of the present invention is shown. Figure 8 As shown, the electronic device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. The RAM 803 can also store various programs and data required for the operation of the electronic device 800. The CPU 801, ROM 802, and RAM 803 are interconnected via bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0223] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, microphone, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0224] The various processes and handling described above can be executed by processing unit 801. For example, the methods of any of the above embodiments can be implemented as computer software programs tangibly contained in a computer-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions of the methods described above can be performed.

[0225] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described model training method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0226] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0227] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0228] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0229] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0230] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0231] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0232] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0233] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0234] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0235] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training method, characterized in that, The method comprises: obtaining a sample fundus image of a user meeting a model training condition, the sample fundus image being labeled with a true disease category of a preset disease; obtaining task encoding information corresponding to the sample fundus image and used for indicating prediction of a specified disease category; inputting the sample fundus image and the task encoding information into a to-be-trained disease prediction model, the to-be-trained disease prediction model comprising a feature dynamic selection network layer and a classification layer, the feature dynamic selection network layer comprising a feature extraction layer, a weight calculation layer and a feature aggregation layer; calling the feature extraction layer to perform a multi-scale feature extraction operation on a fundus feature map corresponding to the sample fundus image to obtain a multi-scale fundus feature; calling the weight calculation layer to process the multi-scale fundus feature to obtain a feature weight corresponding to the multi-scale fundus feature; calling the feature aggregation layer to aggregate the multi-scale fundus feature according to the feature weight to obtain an aggregated feature map; calling the classification layer to process the aggregated feature map to obtain a predicted disease category of the user; based on the true disease category and the predicted disease category, calculating a loss value of the to-be-trained disease prediction model; in a case where the loss value is within a preset range, taking the trained to-be-trained disease prediction model as a final disease prediction model.

2. The method of claim 1, wherein, The to-be-trained disease prediction model further comprises a controller, before the calling of the feature dynamic selection network layer to perform a multi-scale feature selection and aggregation operation on a fundus feature map corresponding to the sample fundus image to obtain an aggregated feature map, further comprising: calling the controller to filter out features irrelevant to the specified disease category from fundus features of the sample fundus image according to the task encoding information to obtain the fundus feature map.

3. The method of claim 1, wherein, The obtaining of the sample fundus image of the user meeting the model training condition comprises: obtaining a plurality of initial fundus images of the user; calling a preset model to process the initial fundus images to screen out the sample fundus image meeting the model training condition from the initial fundus images, the sample fundus image being a real fundus image category and having an image quality greater than a quality threshold.

4. The method of claim 3, wherein, The calling of the preset model to process the initial fundus images to screen out the sample fundus image meeting the model training condition from the initial fundus images comprises: preprocessing the initial fundus images to generate preprocessed fundus images; calling a fundus image recognition model to process the preprocessed fundus images to obtain fundus image recognition results; determining a standard fundus image in the initial fundus images according to the fundus image recognition results; calling an image quality level classification model to process the preprocessed fundus images to obtain probabilities that the preprocessed fundus images belong to each preset quality level; screening out the sample fundus image meeting the model training condition from the standard fundus image according to the probabilities.

5. The method of claim 1, wherein, After the loss value is within the preset range, the trained to-be-trained disease prediction model is taken as the final disease prediction model. obtaining a target fundus image of a target user meeting a preset disease detection condition; inputting the target fundus image into the disease prediction model; calling the feature dynamic selection network layer to perform a multi-scale feature selection and aggregation operation on a target fundus feature map corresponding to the target fundus image to obtain a target feature map; calling the classification layer to process the target feature map to obtain a prediction result of whether the user has a preset disease.

6. A model training apparatus characterized by comprising: The device comprises: a sample image acquisition module configured to obtain a sample fundus image of a user meeting a model training condition, the sample fundus image being labeled with a real disease category of a preset disease; a sample image input module configured to input the sample fundus image into a disease prediction model to be trained, the disease prediction model to be trained comprising a feature dynamic selection network layer and a classification layer; a feature map acquisition module configured to call the feature dynamic selection network layer to perform a multi-scale feature selection and aggregation operation on a fundus feature map corresponding to the sample fundus image to obtain an aggregated feature map; a prediction category acquisition module configured to call the classification layer to process the aggregated feature map to obtain a predicted disease category of the user; a loss value calculation module configured to calculate a loss value of the disease prediction model to be trained based on the real disease category and the predicted disease category; a prediction model acquisition module configured to, in a case where the loss value is within a preset range, take the trained disease prediction model to be trained as a final disease prediction model; The sample image input module comprises: an encoding information acquisition unit configured to obtain task encoding information corresponding to the sample fundus image, the task encoding information being used to indicate a prediction of a specified disease category; an image encoding input unit configured to input the sample fundus image and the task encoding information into the disease prediction model to be trained; The feature dynamic selection network layer comprises a feature extraction layer, a weight calculation layer, and a feature aggregation layer, The feature map acquisition module comprises: a multi-scale feature acquisition unit configured to call the feature extraction layer to perform a multi-scale feature extraction operation on a fundus feature map corresponding to the sample fundus image to obtain multi-scale fundus features; a feature weight acquisition unit configured to call the weight calculation layer to process the multi-scale fundus features to obtain feature weights corresponding to the multi-scale fundus features; an aggregated feature map acquisition unit configured to call the feature aggregation layer to aggregate the multi-scale fundus features according to the feature weights to obtain the aggregated feature map.

7. An electronic device, comprising: comprise: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being executed by the processor to implement the model training method according to any one of claims 1 to 5.

8. A readable storage medium, characterized by, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute the model training method according to any one of claims 1 to 5.

Images