State prediction method, device, computer equipment and storage medium

By using feature extraction and mixed models to process fundus images and covariate data in the prediction of diabetic retinopathy, the prediction bias caused by censored data is solved and the prediction accuracy is improved.

CN117637144BActive Publication Date: 2024-11-26TSINGHUA UNIVERSITY +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202311597484.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-27
Publication Date
2024-11-26
Estimated Expiration
2043-11-27

AI Technical Summary

Technical Problem

In traditional methods, convolutional neural networks have biases when processing fundus images containing random censored data, increasing the risk of false positive predictions, resulting in a low prediction accuracy of diabetic retinopathy.

Method used

A state prediction method is adopted to obtain fundus image and covariate data, and feature extraction is performed based on a preset feature extraction model to obtain image feature vectors and covariate feature vectors, and use a mixed model to process these feature vectors to satisfy the maximum likelihood function. Conditions, generate target state prediction results, adapt to the presence and absence of censored data.

Benefits of technology

The accuracy of the target state prediction results in the case of censoring of the fundus image is improved, and the accuracy of the prediction results is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117637144B_ABST
    Figure CN117637144B_ABST
Patent Text Reader

Abstract

The present application relates to a state prediction method, device, computer equipment, storage medium and computer program product. The method comprises: obtaining data to be detected; the data to be detected comprises fundus image and covariate data; the covariate data is variable data for assisting in distinguishing target lesions; based on a preset feature extraction model, feature extraction is performed on fundus image and covariate data respectively, and image feature vector, first weight corresponding to image feature vector, covariate feature vector and second weight corresponding to covariate feature vector are obtained; based on the first weight, second weight and preset hybrid model, image feature vector and covariate feature vector are processed to obtain target state prediction result; the hybrid model is constructed based on image feature vector and covariate feature vector that respectively meet the maximum likelihood function condition when there is censored data and when there is no censored data in the data to be detected. The use of this method can improve the prediction accuracy of target state prediction result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a state prediction method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of artificial intelligence technology in the medical field, artificial intelligence plays an increasingly important role in medical diagnosis, screening, prediction and auxiliary management or treatment plans. For example, artificial intelligence technology can be used to detect retinal fundus photos and automatically identify diabetic retinopathy (DR).

[0003] In the traditional method, taking DR lesions as an example, the retinal base photos of individual users within the time of DR onset and disease progression are obtained, and the retinal base photos are used as input data. The retinal base photos are extracted and convolutional calculations are performed on the retinal base photos through deep learning convolutional neural networks to obtain the time of DR onset or disease progression that individual users may have in the future time period, thereby realizing automatic detection of DR. Among them, the existing convolutional neural network uses the mean absolute error loss function or the mean square error loss function in the regression task with a loss function of .

[0004] However, in traditional methods, there may be random missing data in the input data, resulting in lesion discrimination being observed only within an extended time interval (i.e., the time of DR onset or disease progression). In this case, there is a deviation in the calculation of the convolutional neural network, which increases the risk of false positive predictions and, in turn, leads to a low accuracy of the prediction results. Summary of the invention

[0005] Based on this, it is necessary to provide a state prediction method, device, computer equipment, computer readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides a state prediction method, comprising:

[0007] Acquire data to be detected; the data to be detected includes fundus images and covariate data; the covariate data is variable data for assisting in distinguishing target lesions;

[0008] Based on a preset feature extraction model, feature extraction is performed on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector;

[0009] Based on the first weight, the second weight and a preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result; the hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function condition when there are missing data and when there are no missing data in the data to be detected.

[0010] In one of the embodiments, the preset feature extraction model includes a first feature extraction model and a second feature extraction model;

[0011] The method of extracting features from the fundus image and the covariate data based on a preset feature extraction model to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector includes:

[0012] Performing convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector;

[0013] According to the second feature extraction model, convolution calculation is performed on the covariate data to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector.

[0014] In one of the embodiments, the first feature extraction model includes a feature extractor and a multi-layer perceptron;

[0015] The step of performing convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector includes:

[0016] Extracting features from the fundus image according to the feature extractor to obtain a feature map, and performing convolution calculation on the feature map based on an objective function to obtain an image feature vector;

[0017] A first weight corresponding to the image feature vector is determined based on the multilayer perceptron.

[0018] In one embodiment, performing convolution calculation on the covariate data according to the second feature extraction model to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector includes:

[0019] Extracting features from the covariate data according to the second feature extraction model, and performing convolution calculation based on the objective function to obtain a covariate feature vector;

[0020] A second weight corresponding to the covariate feature vector is determined according to the multilayer perceptron of the second feature extraction model.

[0021] In one embodiment, before obtaining the data to be detected, the method further includes:

[0022] Acquire a sample fundus image set;

[0023] Based on a preset loss function, the sample fundus image set and a stochastic gradient descent algorithm, the first feature extraction model is self-supervised trained to obtain a trained first feature extraction model.

[0024] In one embodiment, the self-supervised training of the first feature extraction model based on the preset loss function, the sample fundus image set and the stochastic gradient descent algorithm to obtain the trained first feature extraction model includes:

[0025] Obtaining a feature extractor; the feature extractor is constructed based on a momentum contrast method;

[0026] Constructing a positive sample fundus image and a negative sample fundus image according to the feature extractor and each of the sample fundus images in the sample fundus image set;

[0027] The first feature extraction model is iteratively self-supervised trained based on a stochastic gradient descent algorithm and the positive sample fundus image and the negative sample fundus image to obtain a trained first feature extraction model.

[0028] In one embodiment, the processing of the image feature vector and the covariate feature vector based on the first weight, the second weight and a preset hybrid model to obtain a target state prediction result includes:

[0029] In the case that the fundus image exists in the data to be detected and the covariate data is empty, the second weight is determined to be zero, and a convolution calculation is performed on the image feature vector based on a third state prediction model whose objective function is a survival function and the first weight to obtain a target state prediction result;

[0030] When the covariate data exists in the data to be detected and the fundus image is empty, the first weight is determined to be zero, and a convolution calculation is performed on the covariate data feature vector based on the third state prediction model in which the objective function is the survival function and the second weight to obtain a target state prediction result.

[0031] In a second aspect, the present application also provides a state prediction device, comprising:

[0032] A first acquisition module is used to acquire data to be detected; the data to be detected includes fundus images and covariate data; the covariate data is variable data for assisting in distinguishing target lesions;

[0033] A feature extraction module, used to perform feature extraction on the fundus image and the covariate data based on a preset feature extraction model, to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector;

[0034] A target state prediction module is used to process the image feature vector and the covariate feature vector based on the first weight, the second weight and a preset hybrid model to obtain a target state prediction result; the hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively meet the maximum likelihood function conditions when there are missing data and when there are no missing data in the data to be detected.

[0035] In one of the embodiments, the preset feature extraction model includes a first feature extraction model and a second feature extraction model;

[0036] The feature extraction module is specifically used to perform convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector;

[0037] According to the second feature extraction model, convolution calculation is performed on the covariate data to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector.

[0038] In one of the embodiments, the first feature extraction model includes a feature extractor and a multi-layer perceptron;

[0039] The feature extraction module is specifically used to extract features from the fundus image according to the feature extractor to obtain a feature map, and to perform convolution calculation on the feature map based on an objective function to obtain an image feature vector;

[0040] A first weight corresponding to the image feature vector is determined based on the multilayer perceptron.

[0041] The feature extraction module is specifically used to extract features from the covariate data according to the second feature extraction model, and perform convolution calculation based on the objective function to obtain a covariate feature vector;

[0042] A second weight corresponding to the covariate feature vector is determined according to the multilayer perceptron of the second feature extraction model.

[0043] In one embodiment, before acquiring the data to be detected, the device further includes:

[0044] A second acquisition module is used to acquire a sample fundus image set;

[0045] The training module is used to perform self-supervised training on the first feature extraction model based on a preset loss function, the sample fundus image set and a stochastic gradient descent algorithm to obtain a trained first feature extraction model.

[0046] In one embodiment, the training module is specifically used to obtain a feature extractor; the feature extractor is constructed based on a momentum comparison method;

[0047] Constructing a positive sample fundus image and a negative sample fundus image according to the feature extractor and each of the sample fundus images in the sample fundus image set;

[0048] The first feature extraction model is iteratively self-supervised trained based on a stochastic gradient descent algorithm and the positive sample fundus image and the negative sample fundus image to obtain a trained first feature extraction model.

[0049] In one of the embodiments, the target state prediction module is specifically used to determine the second weight to be zero when the fundus image exists in the data to be detected and the covariate data is empty, and perform convolution calculation on the image feature vector based on the third state prediction model whose objective function is the survival function and the first weight to obtain a target state prediction result;

[0050] When the covariate data exists in the data to be detected and the fundus image is empty, the first weight is determined to be zero, and a convolution calculation is performed on the covariate data feature vector based on the third state prediction model in which the objective function is the survival function and the second weight to obtain a target state prediction result.

[0051] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0052] Acquire data to be detected; the data to be detected includes fundus images and covariate data; the covariate data is variable data for assisting in distinguishing target lesions;

[0053] Based on a preset feature extraction model, feature extraction is performed on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector;

[0054] Based on the first weight, the second weight and a preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result; the hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function condition when there are missing data and when there are no missing data in the data to be detected.

[0055] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0056] Acquire data to be detected; the data to be detected includes fundus images and covariate data; the covariate data is variable data for assisting in distinguishing target lesions;

[0057] Based on a preset feature extraction model, feature extraction is performed on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector;

[0058] Based on the first weight, the second weight and a preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result; the hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function condition when there are missing data and when there are no missing data in the data to be detected.

[0059] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0060] Acquire data to be detected; the data to be detected includes fundus images and covariate data; the covariate data is variable data for assisting in distinguishing target lesions;

[0061] Based on a preset feature extraction model, feature extraction is performed on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector;

[0062] Based on the first weight, the second weight and a preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result; the hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function condition when there are missing data and when there are no missing data in the data to be detected.

[0063] The state prediction method, device, computer equipment, storage medium and computer program product obtain data to be detected; the data to be detected includes fundus image and covariate data; the covariate data is variable data for assisting in distinguishing target lesions; based on a preset feature extraction model, feature extraction is performed on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector and a second weight corresponding to the covariate feature vector; based on the first weight, the second weight and a preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result; the hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively meet the maximum likelihood function condition when there is censored data and when there is no censored data in the data to be detected. With this method, the hybrid model is constructed based on the image feature vector and the covariate feature vector that meet the maximum likelihood function condition when there is censored data and when there is no censored data in the data to be detected. The hybrid model is used to predict the fundus state, which can improve the accuracy of the target state prediction result when there is censored data in the fundus image, thereby improving the accuracy of the prediction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A schematic diagram of a flow chart of a state prediction method in one embodiment;

[0066] Figure 2 A schematic diagram of a flow chart of a feature extraction step of performing feature extraction on fundus images and covariate data respectively using a preset feature extraction model in an embodiment;

[0067] Figure 3 A schematic diagram of a process of extracting features from a fundus image using a first feature extraction model in one embodiment;

[0068] Figure 4 A schematic diagram of a process of extracting features from covariate data using a second feature extraction model in one embodiment;

[0069] Figure 5 is a schematic diagram of a process of training a first feature extraction model in one embodiment;

[0070] Figure 6is a schematic diagram of a process of training a first feature extraction model in another embodiment;

[0071] Figure 7 A schematic diagram of a process of performing state prediction in different scenarios in an embodiment;

[0072] Figure 8 is a structural block diagram of a state prediction device in one embodiment;

[0073] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0075] In one embodiment, Figure 1 As shown, a state prediction method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0076] In an exemplary embodiment, Figure 1 As shown, a state prediction method is provided, including the following steps 102 to 106. Among them:

[0077] Step 102: Obtain data to be detected.

[0078] The data to be detected include fundus images and covariate data.

[0079] Among them, covariate data is variable data that assists in identifying target lesions.

[0080] In an embodiment of the present application, the terminal obtains user input data, including a fundus image and covariate data of the object to be examined, wherein the covariate data may include the age, gender, duration of diabetes, systolic blood pressure, diastolic blood pressure, low-density lipoprotein, high-density lipoprotein, glycated hemoglobin and baseline DR grade of the object to be examined.

[0081] Step 104 , extract features from the fundus image and covariate data based on a preset feature extraction model to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector.

[0082] In the embodiment of the present application, the terminal uses a preset feature extraction model to take the fundus image and covariate data as input respectively, and obtains an image feature vector and a covariate feature vector through calculation and processing by the preset feature extraction model.

[0083] After the image feature vector and the covariate feature vector are generated, the terminal sends the image feature vector and the covariate feature vector to the hybrid model for processing to generate the target state prediction result. The terminal can determine the weights of each parameter in the feature extraction model by maximizing the likelihood function of the supervised annotation data, and optimize the parameters of the model in this way to obtain the corresponding weights.

[0084] In an exemplary embodiment, the object to be inspected is used as an observation object. There may be multiple observation objects, and each object may be expressed as a four-tuple s i = <x i ,t i ,t′ i ,e i >, where x i is a feature vector consisting of image features and / or other covariates collected at the beginning of the study. Symbol e i Indicates whether the record is censored: uncensored records correspond to e i =1, otherwise e i =0. t i represents the last check time before the event of interest was observed, t i ′ represents the time when the event occurred, or if the event was not observed, it is equal to t i .

[0085] Step 106, based on the first weight, the second weight and the preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result.

[0086] The hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function condition when there are censored data and when there are no censored data in the data to be detected.

[0087] In the embodiment of the present application, the survival function of the terminal for each observed object is Modeling is performed, where the mixed model consists of K Weibull distributions. Since the Weibull distribution has support only in the positive real number space, it is suitable for survival regression analysis. The cumulative distribution function (CDF) of the Weibull distribution has a closed solution, so this application uses a gradient-based optimization method for maximum likelihood estimation. The survival function is shown in the following formula (1):

[0088]

[0089] Where K is the length of the characteristic vector representing the Weibull distribution, φ i|x is the parameter set of the Weibull distribution, which represents the parameter set of the ith Weibull distribution and is used to describe the properties of the ith Weibull function, including its shape, scale, and location. i () represents the i-th Weibull function, which contains two parameters β i ,α i , which is a fixed parameter determined based on the target event. For example, K = 512, which means that the final S(t) is composed of 512 Weibull functions, and the proportion of each Weibull function in the final survival function is determined by the mixed distribution weight of the model output in the first step.

[0090] Furthermore, the cumulative distribution function (CDF) of the Weibull distribution is obtained:

[0091]

[0092] The main goal is to accurately estimate the parameters and weight set of the hybrid model given the input covariates. The terminal inputs the first weight and the image feature vector corresponding to the first weight, and the covariate feature vector corresponding to the second weight into the hybrid model for calculation, and calculates the target state prediction result of the observed object through the survival analysis method. The target state prediction result can be a survival function, which characterizes the disease risk, survival period, or other results related to disease symptoms of the observed object.

[0093] In the above state prediction method, the hybrid model is constructed based on the image feature vector and covariate feature vector that meet the maximum likelihood function condition when there are censored data and when there are no censored data in the data to be detected. The hybrid model is used to predict the fundus state, which can improve the accuracy of the target state prediction result when there are censored fundus images, thereby improving the accuracy of the prediction result.

[0094] In an exemplary embodiment, the preset feature extraction model includes a first feature extraction model and a second feature extraction model, such as Figure 2 As shown, step 104 includes steps 202 to 204. Among them:

[0095] Step 202: Perform convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector.

[0096] In an embodiment of the present application, the first feature extraction model is composed of a plurality of convolutional layers, a pooling layer, and a fully connected layer. The first feature extraction model receives an input fundus image and converts it into an image feature vector. Among them, the feature extractor of the first feature extraction model may be ResNet-50, and the first feature extraction model may include a self-attention layer 58 for emphasizing the important parts of the fundus features. Specifically, the self-attention layer may be a standard dot product self-attention layer 59 for calculating the weight of each pixel in the feature map generated by block 3 of ResNet-50. In addition, the first feature extraction model uses a 3-layer multilayer perceptron (MLP, Multilayer perceptron) as a predictor to estimate the weights of a set of fixed-size Weibull distributions of features generated by a pre-trained ResNet-50 model. Among them, the attention layer outputs a feature map of the same size as the input feature map to ensure seamless insertion into ResNet-50. In the interpretation stage, the attention feature map is reshaped into a specific shape and then resized for illustration.

[0097] In an optional embodiment, the image feature vector may need to be normalized or compressed to ensure that it has a good numerical range and statistical characteristics. Processing the normalized image feature vector may generate corresponding first weights, which can be used for subsequent hybrid model inference, combining the information of the feature vector of the covariate and its corresponding weight to obtain the target state prediction result.

[0098] Step 204: Perform convolution calculation on the covariate data according to the second feature extraction model to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector.

[0099] In an embodiment of the present application, the terminal performs convolution calculation on the covariate data according to the second feature extraction model, and can obtain a covariate feature vector and a second weight corresponding to the covariate feature vector. The second feature extraction model can be composed of multiple convolutional layers, pooling layers, and fully connected layers for processing input covariate data. Through this model, the covariate data can be converted into a covariate feature vector.

[0100] In an optional embodiment, similar to the image feature vector, the covariate feature vector may need to be normalized or compressed to ensure that it has a good value range and statistical characteristics. Processing the normalized covariate feature vector may generate a corresponding second weight.

[0101] In this embodiment, the first weight and the second weight can combine the covariate feature vector with the image feature vector as input for hybrid model prediction. By combining the feature vector and weight of the covariate data into the hybrid model, the influence of the covariate on the target state can be more comprehensively considered, thereby improving the prediction accuracy.

[0102] In an exemplary embodiment, the first feature extraction model includes a feature extractor and a multi-layer perceptron, such as Figure 3 As shown, step 202 includes steps 302 to 304. Among them:

[0103] Step 302: extract features from the fundus image using a feature extractor to obtain a feature map, and perform convolution calculation on the feature map based on an objective function to obtain an image feature vector.

[0104] In the embodiment of the present application, the feature extractor can adopt a multi-layer convolutional neural network (CNN), a residual network (ResNet), an auto-encoder, and other models to receive an input fundus image, extract features from it, and obtain a series of feature maps. The feature map can reflect image features of different levels and scales, including local details, overall structure, and other information.

[0105] The terminal can perform convolution calculations on the feature map according to the preset objective function. The objective function is a survival function, and the convolution kernel composed of multiple trainable parameters is used to extract features of a specific type or direction. The convolution process can further process the feature map to generate a new feature map, and can also be used to compress the dimension of the feature map. After multiple convolution operations, a two-dimensional matrix containing various feature information is obtained, namely the image feature vector. Among them, this process involves the setting of multiple hyperparameters and training parameters, including the architecture of the feature extractor, the selection of activation functions, the size and number of convolution kernels, etc.

[0106] Step 304: Determine a first weight corresponding to the image feature vector based on a multi-layer perceptron.

[0107] In the embodiment of the present application, the multilayer perceptron is a commonly used feedforward neural network model, which is composed of multiple hidden layers and an output layer. When the image feature vector is used as input, the terminal can perform weighted summation on the input through the hidden layer nodes of the multilayer perceptron, and perform nonlinear transformation through the activation function. Finally, the nodes of the output layer will produce corresponding prediction results. For the image feature vector, it can be used as the input of the multilayer perceptron, and the objective function can be defined as a problem that needs to be predicted or classified during training. Through back propagation and gradient descent algorithms, the weights of each hidden layer node and output layer node can be automatically calculated, and then the first weight corresponding to the image feature vector is determined.

[0108] In this embodiment, through feature extraction and weight determination, a more representative and informative image feature vector can be obtained, thereby improving the prediction accuracy and generalization ability of the model, and then making full use of the missing images to accurately predict the disease in a short follow-up period, thereby improving the accuracy of the target state prediction results of the observed object.

[0109] In an exemplary embodiment, Figure 4 As shown, step 204 includes steps 402 to 406 .

[0110] in:

[0111] Step 402: extract features from the covariate data according to the second feature extraction model, and perform convolution calculation based on the target function to obtain a covariate feature vector.

[0112] In an embodiment of the present application, the second feature extraction model is a metadata model, and a 3-layer MLP is used in the metadata model. The terminal extracts features from the covariate data according to the second feature extraction model, and performs convolution calculation based on the objective function to obtain a covariate feature vector. Similar to the feature extraction process of fundus images, the feature extraction of covariate data can also use various deep learning models, such as convolutional neural networks (CNN), autoencoders (Autoencoder), recurrent neural networks (RNN), etc. The second feature extraction model extracts features from the input covariate data to obtain a corresponding feature map.

[0113] After obtaining the feature map, the terminal can perform convolution calculations on it based on a preset objective function. The objective function can be a convolution kernel composed of trainable parameters, which is used to extract features of a specific type or direction. This convolution process can perform weighted summation, nonlinear transformation, and other operations on the feature map to obtain a covariate feature vector.

[0114] In an optional embodiment, the metadata model may also take metadata as input and output predicted event occurrence times for individual participants.

[0115] Step 404: Determine a second weight corresponding to the covariate feature vector according to the multilayer perceptron of the second feature extraction model.

[0116] In the embodiment of the present application, the second feature extraction model also includes a 3-layer MLP. The same principle as step 304 is used. The process in which the terminal determines the second weight corresponding to the covariate feature vector through the multilayer perceptron is not repeated in this embodiment.

[0117] In this embodiment, through feature extraction and weight determination, a more representative and informative covariate feature vector can be obtained, thereby improving the prediction accuracy and generalization ability of the model, accurately predicting the disease in a short follow-up period, and improving the accuracy of the target state prediction results of the observed object.

[0118] In an exemplary embodiment, Figure 5As shown, before step 102, the method further includes steps 502 to 504. Among them:

[0119] Step 502: Acquire a sample fundus image set.

[0120] In an embodiment of the present application, the terminal obtains a sample fundus image set, uses a method using momentum contrast (MoCo-v2), and uses self-supervised learning to generate a pre-trained feature extractor. In this process, the feature extractor is trained on a large number of fundus image data sets without manual annotation. MoCo v2 uses a momentum-based contrast learning framework to learn how to create positive and negative image block pairs from the same image during the pre-training process of the fundus model.

[0121] In an optional embodiment, the data enhancement method includes random image compression, random blur, brightness jitter, contrast jitter, random gamma transform, random Gaussian noise and random rotation.

[0122] Step 504 , based on a preset loss function, a sample fundus image set and a stochastic gradient descent algorithm, a self-supervised training is performed on the first feature extraction model to obtain a trained first feature extraction model.

[0123] In an embodiment of the present application, the terminal performs self-supervised training on the first feature extraction model based on a preset loss function, a sample fundus image set, and a stochastic gradient descent algorithm to obtain a trained first feature extraction model. The deep learning model f(·|Θ) is passed to determine the parameter set φ of the mixed distribution, wherein the maximum likelihood estimation method is used to estimate the parameters of the deep learning network. Considering the missing data, the maximum likelihood estimation can be expressed as:

[0124]

[0125] in, represents the sample fundus images and sample covariate data containing censored data, Θ represents the model parameters, X = x i represents the feature vector of the observed object, T is a random time, representing the time when the target event occurs, for example, the time when the observed object develops DR or progresses, T>t i Characterizes that T is in the state of T>t i The probability after that, T <t′ i Indicates that T is at t′ iThe probability before, or the probability of an event occurring before the current time T if no event has been observed. For example, if the subject has 5 follow-up examinations, a total of 5 years, then t1 = 1, t2 = 2, t3 = 3, t4 = 4, t5 = 5. If the subject becomes ill in 3.5 years, then T = 3.5. However, the specific value of T is actually unknown, and we can only conclude that T is between t3 and t4, so T = t3 = 3 years, t' = t4 = 4 years.

[0126] Similarly, for uncensored data, the expression is as follows:

[0127]

[0128]

[0129] in, Characterize the fundus image and covariate data without censored data. The preset loss function is shown in the following formula:

[0130]

[0131] Among them, γ is a hyperparameter used to adjust the weights of censored and uncensored data, and φ is the parameter set of the mixed distribution. Among them, the fundus model (the first feature extraction model), the metadata model (the second feature extraction model) and the mixed model share the same loss function.

[0132] In an optional embodiment, the first feature extraction model, the second feature extraction model and the hybrid model are trained using the same method, and the training process of the second feature extraction model and the hybrid model will not be described in detail in this embodiment.

[0133] In this embodiment, the first feature extraction model is trained through self-supervised training, and modeling is performed separately according to sample fundus images and sample covariate data containing missing data and not containing missing data, which can improve the accuracy of determining the target state prediction results, thereby improving the accuracy of the prediction results.

[0134] In an exemplary embodiment, Figure 6 As shown, step 504 includes steps 602 to 606 .

[0135] in:

[0136] Step 602, obtaining a feature extractor.

[0137] Among them, the feature extractor is constructed based on the momentum comparison method.

[0138] In an embodiment of the present application, the feature extractor uses contrastive learning under a self-supervised learning framework and utilizes the time series information in the data to learn useful feature representations without using any label information.

[0139] The terminal obtains a feature extractor, which constructs a comparison sample according to a data enhancement method, constructs a feature extraction network based on a convolutional neural network (CNN) architecture, and adopts a momentum comparison loss function. The loss function compares the similarity between the feature representations of the input sample and the corresponding comparison sample, so that the features of the same sample are closer and the features of different samples are farther apart.

[0140] Step 604 : construct a positive sample fundus image and a negative sample fundus image according to the feature extractor and each sample fundus image in the sample fundus image set.

[0141] In an embodiment of the present application, in the feature extraction of fundus images, the terminal can use a deep learning model such as a convolutional neural network (CNN) to build a feature extractor. Generally speaking, the input of the feature extractor is a fundus image, and the output is an extracted feature vector. Then, using these feature vectors, training sets and test sets can be constructed based on positive samples and negative samples. Specifically, fundus images based on the same category can be regarded as positive samples. The terminal can calculate the similarity between their feature vectors based on the fundus images of the same category for the training set, validation set, and test set, so as to construct a positive sample library. Common similarity calculation methods include Euclidean distance, cosine similarity, etc.

[0142] Step 606 , iteratively perform self-supervised training on the first feature extraction model based on the stochastic gradient descent algorithm and the positive sample fundus images and the negative sample fundus images to obtain a trained first feature extraction model.

[0143] In an embodiment of the present application, the terminal uses a stochastic gradient descent algorithm for training. Stochastic gradient descent updates the parameters of the model by using the gradient estimate of each sample. During the iterative training process, a portion of samples are extracted from the training set for training in a small batch manner, and the model parameters are updated according to the calculation results of the loss function and the appropriate learning rate, optimizer type and other hyperparameters are selected to improve the effect of the training process and the performance of the model. For example, in each iteration, two 512×512 cropped images are taken for each fundus image, and the terminal uses a k-nearest neighbor (kNN) monitor as a self-supervised evaluation tool, and each epoch (sample fundus image or sample covariate data) is evaluated once. In terms of hyperparameter selection, ResNet-50 is used as an encoder and stochastic gradient descent (SGD, Stochastic Gradient Descent) is used as an optimizer. The input image resolution is 512×512 pixels, the batch size is set to 256, and the MoCo v2 model is trained for 800 epochs. The best hyperparameters were obtained using grid search, which yielded learning rate = 10-3, weight decay = 10-4, SGD momentum = 0.9, and temperature τ = 1.0. The encoder momentum coefficient was m = 0.996 and increased to 1 according to a cosine schedule.

[0144] From the experimental results, the prediction performance of the fundus model pre-trained with MoCo v2 can be evaluated based on the ablation experiment. The results show that incorporating MoCo v2 into the training process can improve the prediction performance of the fundus model.

[0145] In an optional embodiment, the terminal trains the fundus model on the training dataset using a stochastic gradient descent (SGD) optimizer, and uses a grid search to find the optimal values ​​and other hyperparameters. The fundus model is trained by backpropagating the batch error for 50 epochs on 32 images resized to 512×512 pixels, with a learning rate of 10 to the negative fifth power. The data augmentation strategy is exactly the same as during pre-training based on MoCo v2.

[0146] In this embodiment, the first feature extraction model is trained through self-supervised training, and modeling is performed separately according to sample fundus images and sample covariate data containing censored data and excluding censored data, which can improve the accuracy of the target state prediction results, thereby improving the accuracy of the DR prediction results. The average screening interval can be extended from 12 months to 34.7 months, reducing the screening frequency by about 65.38%. At the same time, the system can identify high-risk patients so that they can receive more frequent follow-up and early detection, while low-risk patients can reduce the number of follow-up visits and reduce the screening burden. In addition, the system has a high prediction accuracy and a low risk of delayed detection of vision-threatening DR. Personalized DR screening intervals and interventions can improve efficiency.

[0147] In an exemplary embodiment, Figure 7 As shown, step 106 includes steps 702 to 704 .

[0148] in:

[0149] Step 702, when the data to be detected has a fundus image and the covariate data is empty, the second weight is determined to be zero, and a convolution calculation is performed on the image feature vector based on the third state prediction model whose objective function is the survival function and the first weight to obtain a target state prediction result.

[0150] In the embodiment of the present application, since the data to be detected of the observed object may not contain fundus images and covariate data at the same time, for example, the observed object may lack covariate data during the observation time, the terminal may also predict the target state only through the fundus image to obtain the target state prediction result. Specifically, when the data to be detected has a fundus image and the covariate data is empty, the terminal cannot use the covariate data to train the model, so the terminal can determine the second weight to be zero. Based on the method of the survival function, an objective function can be constructed, which can measure the model's predictive ability for patient survival, and the survival function is used as the objective function and added to the optimization process of the model for training. The third state prediction model constructed by the terminal based on the survival function objective function can be a variety of deep learning models, such as a model based on a convolutional neural network (CNN). The goal of this model is to learn to convert the feature vector into a target state prediction result. The terminal uses the first feature extraction model trained before to extract features from the fundus image in the data to be detected, obtain its feature vector, and input the feature vector into the trained third state prediction model, and use convolution calculation to predict the target state. Optionally, the terminal can use a softmax function to normalize the prediction results.

[0151] Step 704, when there is covariate data in the data to be detected and the fundus image is empty, the first weight is determined to be zero, and a convolution calculation is performed on the covariate data feature vector based on the third state prediction model whose objective function is the survival function and the second weight to obtain a target state prediction result.

[0152] In the embodiment of the present application, for the same reason as step 702, the observed object may lack fundus images during the observation time, resulting in only covariate data in the data to be detected. Therefore, the terminal can predict the target state of the observed object based only on the covariate data to obtain the target state prediction result. The process of the terminal predicting based on the data to be detected containing only covariate data is the same as the process of step 702, and the detailed process of convolution calculation in this embodiment will not be repeated.

[0153] In this embodiment, the target state of the observed object is predicted by using the data to be detected that includes a single data, so that the universality of the state prediction method can be improved.

[0154] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0155] Based on the same inventive concept, the embodiment of the present application also provides a state prediction device for implementing the state prediction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more state prediction device embodiments provided below can refer to the limitations of the state prediction method above, and will not be repeated here.

[0156] In an exemplary embodiment, Figure 8 As shown, a state prediction device 800 is provided, comprising: a first acquisition module 801, a feature extraction module 802 and a target state prediction module 803, wherein:

[0157] The first acquisition module 801 is used to acquire data to be detected; the data to be detected includes fundus images and covariate data; the covariate data is variable data that assists in distinguishing target lesions;

[0158] A feature extraction module 802 is used to extract features from the fundus image and the covariate data based on a preset feature extraction model, and obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector;

[0159] The target state prediction module 803 is used to process the image feature vector and the covariate feature vector based on the first weight, the second weight and the preset hybrid model to obtain the target state prediction result; the hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively meet the maximum likelihood function conditions when there are missing data and when there are no missing data in the data to be detected.

[0160] In one of the embodiments, the preset feature extraction model includes a first feature extraction model and a second feature extraction model;

[0161] The feature extraction module 802 is specifically used to perform convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector;

[0162] According to the second feature extraction model, convolution calculation is performed on the covariate data to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector.

[0163] In one of the embodiments, the first feature extraction model includes a feature extractor and a multi-layer perceptron;

[0164] The feature extraction module 802 is specifically used to extract features from the fundus image according to the feature extractor to obtain a feature map, and to perform convolution calculation on the feature map based on the target function to obtain an image feature vector;

[0165] A first weight corresponding to the image feature vector is determined based on a multi-layer perceptron.

[0166] The feature extraction module is specifically used to extract features from the covariate data according to the second feature extraction model, and perform convolution calculation based on the objective function to obtain a covariate feature vector;

[0167] A second weight corresponding to the covariate feature vector is determined according to the multilayer perceptron of the second feature extraction model.

[0168] In one embodiment, before acquiring the data to be detected, the device 800 further includes:

[0169] A second acquisition module is used to acquire a sample fundus image set;

[0170] The training module is used to perform self-supervised training on the first feature extraction model based on a preset loss function, a sample fundus image set and a stochastic gradient descent algorithm to obtain a trained first feature extraction model.

[0171] In one embodiment, the training module is specifically used to obtain a feature extractor; the feature extractor is constructed based on a momentum contrast method;

[0172] Constructing a positive sample fundus image and a negative sample fundus image according to the feature extractor and each sample fundus image in the sample fundus image set;

[0173] The first feature extraction model is iteratively self-supervised trained based on a stochastic gradient descent algorithm and positive sample fundus images and negative sample fundus images to obtain a trained first feature extraction model.

[0174] In one embodiment, the target state prediction module 803 is specifically used to determine the second weight to be zero when there is a fundus image in the data to be detected and the covariate data is empty, and perform convolution calculation on the image feature vector based on the third state prediction model in which the objective function is the survival function and the first weight to obtain the target state prediction result;

[0175] When there is covariate data in the data to be detected and the fundus image is empty, the first weight is determined to be zero, and a convolution calculation is performed on the covariate data feature vector based on the third state prediction model in which the objective function is the survival function and the second weight to obtain a target state prediction result.

[0176] Each module in the above-mentioned state prediction device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0177] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store fundus images and covariate data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a state prediction method is implemented.

[0178] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0179] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0180] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0181] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0182] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0183] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0184] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0185] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A state prediction method, characterized in that: The method comprises: Acquire data to be detected; the data to be detected includes fundus images and covariate data; the covariate data is variable data for assisting in distinguishing target lesions; Based on a preset feature extraction model, feature extraction is performed on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector; Based on the first weight, the second weight and a preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result; the hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function condition when there is censored data and when there is no censored data in the data to be detected; The object to be inspected is taken as an observation object, and each of the observation objects can be expressed as a four-tuple ,in, is a feature vector including image features and / or other covariates collected at the beginning of the study; the symbol Marks whether the record is deleted: the uncensored record corresponds to ,otherwise ; represents the last check time before the event of interest was observed, represents the time when the event occurred, or if the event was not observed, it is equal to ; When considering missing data, the maximum likelihood estimation can be expressed as: in, Characterize sample fundus images and sample covariate data containing censored data, Characterize the model parameters, represents the feature vector of the observed object, T is the random time, representing the time when the target event occurs, Characterizes T in The probability after that, Indicates that T is in Prior probability, or the probability of an event occurring before the current time T if the event had not been observed; For uncensored data, the expression is as follows: in, Characterize the fundus image and covariate data without missing data; the preset loss function is shown in the following formula: in, is a hyperparameter used to adjust the weights of censored and uncensored data, is the parameter set of the mixed distribution; The processing of the image feature vector and the covariate feature vector based on the first weight, the second weight and a preset hybrid model to obtain a target state prediction result includes: The first weight and the image feature vector corresponding to the first weight, as well as the second weight and the covariate feature vector corresponding to the second weight are respectively input into the mixed model for calculation, and the target state prediction result of the observed object is calculated by the survival analysis method, and the target state prediction result is expressed as a survival function, which represents the disease risk and survival period of the observed object; The survival function for each observation Modeling is performed, wherein the mixed model consists of K Weibull distributions; the cumulative distribution function of the Weibull distribution has a closed solution, and a maximum likelihood estimation is performed based on a gradient optimization method; The survival function is shown in the following formula: Where K is the length of the characteristic vector representing the Weibull distribution, is the parameter set of the Weibull distribution, which represents the parameter set of the j-th Weibull distribution and is used to describe the properties of the j-th Weibull function, including its shape, scale, and location. Characterizes the jth Weibull function, which contains two parameters , the parameters Fixed parameters determined for the target event; The cumulative distribution function of the Weibull distribution is: 。 2. The method according to claim 1, characterized in that The preset feature extraction model includes a first feature extraction model and a second feature extraction model; The method of extracting features from the fundus image and the covariate data based on a preset feature extraction model to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector includes: Performing convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector; According to the second feature extraction model, convolution calculation is performed on the covariate data to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector.

3. The method according to claim 2, characterized in that The first feature extraction model includes a feature extractor and a multi-layer perceptron; The step of performing convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector includes: Extracting features from the fundus image according to the feature extractor to obtain a feature map, and performing convolution calculation on the feature map based on an objective function to obtain an image feature vector; A first weight corresponding to the image feature vector is determined based on the multilayer perceptron.

4. The method according to claim 2, characterized in that: The step of performing convolution calculation on the covariate data according to the second feature extraction model to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector includes: Extracting features from the covariate data according to the second feature extraction model, and performing convolution calculation based on the objective function to obtain a covariate feature vector; A second weight corresponding to the covariate feature vector is determined according to the multilayer perceptron of the second feature extraction model.

5. The method according to claim 1, characterized in that Before obtaining the data to be detected, the method further includes: Acquire a sample fundus image set; Based on a preset loss function, the sample fundus image set and a stochastic gradient descent algorithm, the first feature extraction model is self-supervised trained to obtain a trained first feature extraction model.

6. The method according to claim 5, characterized in that The self-supervised training of the first feature extraction model based on the preset loss function, the sample fundus image set and the stochastic gradient descent algorithm to obtain the trained first feature extraction model includes: Obtaining a feature extractor; the feature extractor is constructed based on a momentum contrast method; Constructing a positive sample fundus image and a negative sample fundus image according to the feature extractor and each of the sample fundus images in the sample fundus image set; The first feature extraction model is iteratively self-supervised trained based on a stochastic gradient descent algorithm and the positive sample fundus image and the negative sample fundus image to obtain a trained first feature extraction model.

7. The method according to claim 1, characterized in that The processing of the image feature vector and the covariate feature vector based on the first weight, the second weight and a preset hybrid model to obtain a target state prediction result includes: In the case that the fundus image exists in the data to be detected and the covariate data is empty, the second weight is determined to be zero, and a convolution calculation is performed on the image feature vector based on a third state prediction model whose objective function is a survival function and the first weight to obtain a target state prediction result; When the covariate data exists in the data to be detected and the fundus image is empty, the first weight is determined to be zero, and a convolution calculation is performed on the covariate data feature vector based on the third state prediction model in which the objective function is the survival function and the second weight to obtain a target state prediction result.

8. A state prediction device, characterized in that: The device comprises: A first acquisition module is used to acquire data to be detected; the data to be detected includes fundus images and covariate data; the covariate data is variable data for assisting in distinguishing target lesions; A feature extraction module, used to perform feature extraction on the fundus image and the covariate data based on a preset feature extraction model, to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector; A target state prediction module, used for processing the image feature vector and the covariate feature vector based on the first weight, the second weight and a preset hybrid model to obtain a target state prediction result; the hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively meet the maximum likelihood function condition when there is censored data and when there is no censored data in the data to be detected; The object to be inspected is taken as an observation object, and each of the observation objects can be expressed as a four-tuple ,in, is a feature vector including image features and / or other covariates collected at the beginning of the study; the symbol Marks whether the record is deleted: the uncensored record corresponds to ,otherwise ; represents the last check time before the event of interest was observed, represents the time when the event occurred, or if the event was not observed, it is equal to ; When considering missing data, the maximum likelihood estimation can be expressed as: in, Characterize sample fundus images and sample covariate data containing censored data, Characterize the model parameters, represents the feature vector of the observed object, T is the random time, representing the time when the target event occurs, Characterizes T in The probability after that, Indicates that T is in Prior probability, or the probability of an event occurring before the current time T if the event had not been observed; For uncensored data, the expression is as follows: in, Characterize the fundus image and covariate data without missing data; the preset loss function is shown in the following formula: in, is a hyperparameter used to adjust the weights of censored and uncensored data, is the parameter set of the mixed distribution; The target state prediction module is specifically used for: The first weight and the image feature vector corresponding to the first weight, as well as the second weight and the covariate feature vector corresponding to the second weight are respectively input into the mixed model for calculation, and the target state prediction result of the observed object is calculated by the survival analysis method, and the target state prediction result is expressed as a survival function, which represents the disease risk and survival period of the observed object; Among them, the survival function for each observation object is Modeling is performed, wherein the mixed model consists of K Weibull distributions; the cumulative distribution function of the Weibull distribution has a closed solution, and a maximum likelihood estimation is performed based on a gradient optimization method; The survival function is shown in the following formula: Where K is the length of the characteristic vector representing the Weibull distribution, is the parameter set of the Weibull distribution, which represents the parameter set of the j-th Weibull distribution and is used to describe the properties of the j-th Weibull function, including its shape, scale, and location. Characterizes the jth Weibull function, which contains two parameters , the parameters Fixed parameters determined for the target event; The cumulative distribution function of the Weibull distribution is: 。 9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Method and system for predicting cancer prognosis risk under high-dimensional censored data

    CN113409946A

  • Hypertension risk prediction method, device, and equipment and medium

    CN113689954A

  • Application of ZBTB20 in hepatocellular carcinoma detection and prognosis evaluation thereof

    CN116047075A

  • Model training method and device, electronic equipment and storage medium

    CN116994101A

  • System to determine inpatient or outpatient care and inform decisions about patient care

    US11114204B1