State prediction method and apparatus, computer device, and storage medium

Abstract

A state prediction method and apparatus, a computer device, a storage medium, and a computer program product. The method comprises: obtaining data for detection, wherein the data for detection comprises a fundus image and covariate data, and the covariate data is variable data for assisting in identifying a target lesion; on the basis of a preset feature extraction model, performing feature extraction on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector; and processing the image feature vector and the covariate feature vector on the basis of the first weight, the second weight, and a preset hybrid model to obtain a target state prediction result, wherein the preset hybrid model is constructed on the basis of the image feature vector and the covariate feature vector that separately satisfy a maximum likelihood function condition and under the conditions where the data for detection has censored data or does not have censored data.

Description

State prediction method, device, computer equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on November 27, 2023, with application number 202311597484.8, and application name “State Prediction Method, Device, Computer Equipment and Storage Medium”, the full text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of artificial intelligence technology, and in particular to a state prediction method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0004] With the development of artificial intelligence technology in the medical field, artificial intelligence plays an increasingly important role in medical diagnosis, screening, prediction and auxiliary management or treatment plans. For example, artificial intelligence technology can be used to detect retinal fundus photos and automatically identify diabetic retinopathy (DR).

[0005] Traditional methods, using DR lesions as an example, obtain retinal fundus photographs within the timeframe of a user's potential DR onset and progression. Using these photographs as input, deep learning convolutional neural networks perform feature extraction and convolution calculations on these photographs to determine the likely future timeframe for DR onset or progression, enabling automatic DR detection. Existing convolutional neural networks typically use the mean absolute error (MAE) or mean squared error (MSE) loss function used in regression tasks.

[0006] However, in traditional methods, there may be randomly deleted data in the input data, resulting in lesion discrimination being observed only within an extended time interval (i.e., the time of DR onset or disease progression). This situation leads to deviations in the calculation of convolutional neural networks, increasing the risk of false positive predictions, and thus resulting in low accuracy of the prediction results.

[0007] Summary of the Invention

[0008] Based on this, it is necessary to provide a state prediction method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0009] In a first aspect, the present application provides a state prediction method, comprising the following steps.

[0010] Acquire data to be detected. The data to be detected includes fundus images and covariate data. The covariate data is variable data that assists in distinguishing target lesions.

[0011] Based on a preset feature extraction model, feature extraction is performed on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector.

[0012] Based on the first weight, the second weight and the preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result, wherein the preset hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function conditions when there are missing data and when there are no missing data in the data to be detected.

[0013] In one embodiment, the preset feature extraction model includes a first feature extraction model and a second feature extraction model.

[0014] The method of performing feature extraction on the fundus image and the covariate data based on a preset feature extraction model to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector and a second weight corresponding to the covariate feature vector includes the following steps.

[0015] According to the first feature extraction model, a convolution calculation is performed on the fundus image to obtain an image feature vector and a first weight corresponding to the image feature vector.

[0016] According to the second feature extraction model, a convolution calculation is performed on the covariate data to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector.

[0017] In one embodiment, the first feature extraction model includes a pre-trained feature extractor and a multi-layer perceptron.

[0018] The method of performing convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector includes the following steps.

[0019] The fundus image is subjected to feature extraction according to the pre-trained feature extractor to obtain a feature map, and the feature map is subjected to convolution calculation based on an objective function to obtain an image feature vector.

[0020] A first weight corresponding to the image feature vector is determined based on the multilayer perceptron.

[0021] In one embodiment, performing convolution calculation on the covariate data according to the second feature extraction model to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector includes the following steps.

[0022] Feature extraction is performed on the covariate data according to the second feature extraction model to obtain a feature map, and convolution calculation is performed on the feature map based on the objective function to obtain a covariate feature vector.

[0023] A second weight corresponding to the covariate eigenvector is determined according to the multilayer perceptron of the second feature extraction model.

[0024] In one embodiment, before acquiring the data to be detected, the method further includes the following steps.

[0025] Acquire a sample fundus image set.

[0026] Based on a preset loss function, the sample fundus image set and a stochastic gradient descent algorithm, the first feature extraction model is self-supervised trained to obtain a trained first feature extraction model.

[0027] In one embodiment, the self-supervised training of the first feature extraction model based on the preset loss function, the sample fundus image set and the stochastic gradient descent algorithm to obtain the trained first feature extraction model includes the following steps.

[0028] A first feature extraction model is obtained, where the first feature extraction model is constructed based on a momentum comparison method.

[0029] A positive sample fundus image and a negative sample fundus image are constructed according to the first feature extraction model and each of the sample fundus images in the sample fundus image set.

[0030] The first feature extraction model is iteratively self-supervised trained based on a stochastic gradient descent algorithm, the positive sample fundus image, and the negative sample fundus image to obtain a trained first feature extraction model.

[0031] In one embodiment, the image feature vector and the covariate feature vector are processed based on the first weight, the second weight and a preset hybrid model to obtain a target state prediction result, including the following steps.

[0032] In the case where the fundus image exists in the data to be detected and the covariate data is empty, the second weight is determined to be zero, and based on the preset hybrid model constructed with the objective function as the survival function and the first weight, a convolution calculation is performed on the image feature vector to obtain a target state prediction result.

[0033] In the case where the covariate data exists in the data to be detected and the fundus image is empty, the first weight is determined to be zero, and the covariate data feature vector is convolutionally calculated based on the preset hybrid model and the second weight constructed based on the objective function to obtain the target state prediction result.

[0034] In one embodiment, the preset mixture model consists of multiple Weibull distributions.

[0035] In one embodiment, after performing feature extraction on the fundus image and the covariate data based on a preset feature extraction model to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector, the method further includes: normalizing or compressing the image feature vector; and / or normalizing or compressing the covariate feature vector.

[0036] In one embodiment, the multi-layer perceptron is a 3-layer perceptron.

[0037] In one embodiment, the size of the positive sample fundus image and the negative sample fundus image is 512×512 pixels. In each iteration, a k-nearest neighbor (kNN) monitor is used as an evaluation tool for self-supervised training.

[0038] In a second aspect, the present application also provides a state prediction device, comprising: a first acquisition module, a feature extraction module and a target state prediction module.

[0039] The first acquisition module is used to acquire data to be detected. The data to be detected includes fundus images and covariate data. The covariate data is variable data that assists in distinguishing target lesions.

[0040] The feature extraction module is used to perform feature extraction on the fundus image and the covariate data based on a preset feature extraction model, obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector and a second weight corresponding to the covariate feature vector.

[0041] The target state prediction module is configured to process the image feature vector and the covariate feature vector based on the first weight, the second weight, and a preset hybrid model to obtain a target state prediction result. The preset hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function conditions when the data to be detected has censored data and when the data to be detected does not have censored data.

[0042] In one embodiment, the preset feature extraction model includes a first feature extraction model and a second feature extraction model.

[0043] The feature extraction module is specifically used to perform convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector; and to perform convolution calculation on the covariate data according to the second feature extraction model to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector.

[0044] In one embodiment, the first feature extraction model includes a pre-trained feature extractor and a multi-layer perceptron.

[0045] The feature extraction module is specifically configured to perform feature extraction on the fundus image according to the feature extractor to obtain a feature map, and perform convolution calculation on the feature map based on an objective function to obtain an image feature vector.

[0046] A first weight corresponding to the image feature vector is determined based on the multilayer perceptron.

[0047] The feature extraction module is specifically used to extract features from the covariate data according to the second feature extraction model, and perform convolution calculation based on the objective function to obtain a covariate feature vector; and determine the second weight corresponding to the covariate feature vector according to the multilayer perceptron of the second feature extraction model.

[0048] In one embodiment, before acquiring the data to be detected, the device further includes: a second acquisition module and a training module.

[0049] The second acquisition module is used to acquire a sample fundus image set.

[0050] The training module is used to perform self-supervised training on the first feature extraction model based on a preset loss function, the sample fundus image set and a stochastic gradient descent algorithm to obtain a trained first feature extraction model.

[0051] In one embodiment, the training module is specifically used to obtain a first feature extraction model, wherein the first feature extraction model is constructed based on a momentum comparison method.

[0052] The training module is further specifically configured to construct a positive sample fundus image and a negative sample fundus image according to the first feature extraction model and each sample fundus image in the sample fundus image set.

[0053] The training module is specifically further used to perform iterative self-supervised training on the first feature extraction model based on a stochastic gradient descent algorithm, positive sample fundus images, and negative sample fundus images to obtain a trained first feature extraction model.

[0054] In one embodiment, the target state prediction module is specifically used to determine the second weight to be zero when the fundus image exists in the data to be detected and the covariate data is empty, and to perform convolution calculation on the image feature vector based on the preset hybrid model constructed with the objective function as the survival function and the first weight to obtain the target state prediction result.

[0055] The target state prediction module 803 is also used to determine the first weight to be zero when the covariate data exists in the data to be detected and the fundus image is empty, and to perform convolution calculation on the covariate data feature vector based on the preset hybrid model and the second weight constructed with the objective function as the survival function to obtain the target state prediction result.

[0056] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program.

[0057] The data to be detected are acquired, wherein the data to be detected include a fundus image and covariate data, wherein the covariate data is variable data for assisting in distinguishing target lesions.

[0058] Based on a preset feature extraction model, feature extraction is performed on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector.

[0059] Based on the first weight, the second weight and the preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result. The preset hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function conditions when there are missing data and when there are no missing data in the data to be detected.

[0060] In a fourth aspect, the present application also provides a computer-readable storage medium having executable instructions stored thereon, which implement the following steps when executed by a processor.

[0061] The data to be detected are acquired, wherein the data to be detected include a fundus image and covariate data, wherein the covariate data is variable data for assisting in distinguishing target lesions.

[0062] Based on a preset feature extraction model, feature extraction is performed on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector.

[0063] Based on the first weight, the second weight and the preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result. The preset hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function conditions when there are missing data and when there are no missing data in the data to be detected.

[0064] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which implements the following steps when executed by a processor.

[0065] The data to be detected are acquired, wherein the data to be detected include a fundus image and covariate data, wherein the covariate data is variable data for assisting in distinguishing target lesions.

[0066] Based on a preset feature extraction model, feature extraction is performed on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector.

[0067] Based on the first weight, the second weight and the preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result. The preset hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function conditions when there are missing data and when there are no missing data in the data to be detected.

[0068] The aforementioned state prediction method, apparatus, computer device, storage medium, and computer program product obtain data to be tested, the data to be tested comprising a fundus image and covariate data, the covariate data being variable data that assists in distinguishing target lesions. Feature extraction is performed on the fundus image and covariate data based on a preset feature extraction model to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector. The image feature vector and the covariate feature vector are then processed based on the first weight, the second weight, and a preset hybrid model to obtain a target state prediction result. The preset hybrid model is constructed based on image feature vectors and covariate feature vectors that satisfy the maximum likelihood function condition, respectively, when the data to be tested has censored data and when it does not. Using this method, the preset hybrid model is constructed based on image feature vectors and covariate feature vectors that satisfy the maximum likelihood function condition, when the data to be tested has censored data and when it does not. Fundus state prediction using the preset hybrid model can improve the accuracy of target state prediction results even when fundus images have censored data, thereby improving the accuracy of the prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] FIG1 is a schematic flow chart of a state prediction method according to an embodiment of the present application;

[0071] FIG2 is a schematic diagram of a process of extracting features from fundus images and covariate data using a preset feature extraction model in one embodiment of the present application;

[0072] FIG3 is a schematic diagram of a process of extracting features from a fundus image by a first feature extraction model in one embodiment of the present application;

[0073] FIG4 is a schematic diagram of a process of extracting features from covariate data by a second feature extraction model in one embodiment of the present application;

[0074] FIG5 is a schematic diagram of a process for training a first feature extraction model in one embodiment of the present application;

[0075] FIG6 is a schematic diagram of a process for training a first feature extraction model in another embodiment of the present application;

[0076] FIG7 is a schematic diagram of a process for performing state prediction in different scenarios in one embodiment of the present application;

[0077] FIG8 is a structural block diagram of a state prediction device according to an embodiment of the present application;

[0078] FIG9 is a diagram showing the internal structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0080] In one embodiment of the present application, as shown in FIG1 , a state prediction method is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server.

[0081] In an exemplary embodiment of the present application, as shown in FIG1 , a state prediction method is provided, including the following steps 102 to 106 .

[0082] Step 102: Acquire the data to be detected, wherein the data to be detected includes fundus images and covariate data, wherein the covariate data is variable data that assists in identifying the target lesion.

[0083] In an embodiment of the present application, the terminal obtains user input data, including fundus images and covariate data of the subject to be examined, wherein the covariate data may include the age, gender, diabetes course, systolic blood pressure, diastolic blood pressure, low-density lipoprotein, high-density lipoprotein, glycated hemoglobin and baseline DR grade of the subject to be examined.

[0084] Step 104 : Based on a preset feature extraction model, feature extraction is performed on the fundus image and covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector.

[0085] In an embodiment of the present application, the terminal uses a preset feature extraction model to take the fundus image and covariate data as inputs respectively, and obtains an image feature vector and a covariate feature vector through calculation and processing by the preset feature extraction model.

[0086] The terminal can determine the weights of each input in the preset feature extraction model by maximizing the likelihood function of the supervised annotation data. This method can be used to optimize the parameters of the model to obtain the first weight corresponding to the image feature vector and the second weight corresponding to the covariate feature vector.

[0087] In an exemplary embodiment, the object to be inspected is used as an observation object. There may be multiple observation objects, and the i-th object can be expressed as a four-tuple s i = <x i ,t i ,t′i,e i >, where x i is a feature vector consisting of image features and / or other covariates collected at the beginning of the study. Symbol e i Indicates whether the record is censored: uncensored records correspond to e i =1, otherwise e i =0. t i represents the last inspection time before the event of interest was observed, t i ′ represents the time when the event occurs. If the event is not observed, then t i 'equal to t i .

[0088] Step 106 : Process the image feature vector and the covariate feature vector based on the first weight, the second weight, and the preset hybrid model to obtain a target state prediction result.

[0089] After the image feature vector and covariate feature vector are generated, the terminal inputs them into a preset hybrid model for processing to generate a target state prediction result. The preset hybrid model is constructed based on the image feature vector and covariate feature vector that respectively meet the maximum likelihood function conditions when the data to be detected has censored data and when the data does not have censored data.

[0090] In the embodiment of the present application, the terminal uses a hybrid model to calculate the survival function of each observation object. Modeling is performed, where the mixed model consists of K Weibull distributions. Since the Weibull distribution has support only in the positive real number space, it is suitable for survival regression analysis. The cumulative distribution function (CDF) of the Weibull distribution has a closed solution, so this application uses a gradient-based optimization method for maximum likelihood estimation. The survival function established using the mixed model is shown in formula (1):

[0091] Among them, K represents that the mixture model consists of K Weibull distributions, which is the length of the characteristic vector of the Weibull distribution; φ i|x represents the parameter set of the i-th Weibull distribution, which is used to describe the properties of the i-th Weibull distribution, including its shape, scale, and location; f i () represents the i-th Weibull function, which contains two function parameters β i ,α i , the function parameters are fixed parameters randomly determined based on the target event. For example, K = 512, which means that the final S(t) is composed of 512 Weibull distributions. The proportion of each Weibull distribution in the final survival function is determined by the weight of each Weibull distribution in the mixture model output by the preset feature extraction model in step S104.

[0092] Furthermore, from the cumulative distribution function (CDF) of the Weibull distribution, we can derive:

[0093] The primary goal is to accurately estimate the parameter set and weight set for each Weibull distribution in a preset hybrid model, given input covariates. The terminal inputs the first weight and the image feature vector corresponding to the first weight, as well as the second weight and the covariate feature vector corresponding to the second weight, into the preset hybrid model for calculation. Using survival analysis, the terminal calculates a target state prediction for the subject. This target state prediction can be the subject's risk of disease, survival period, or other outcomes related to disease symptoms.

[0094] In the above-mentioned state prediction method, the preset hybrid model is constructed based on the image feature vector and covariate feature vector that satisfy the maximum likelihood function conditions when there are missing data or no missing data in the data to be detected. Fundus state prediction is performed by the preset hybrid model. In the case of missing data in the fundus image, the accuracy of the target state prediction result can be improved, thereby improving the accuracy of the prediction result.

[0095] In an exemplary embodiment, the preset feature extraction model includes a first feature extraction model and a second feature extraction model. As shown in FIG. 2 , step 104 includes steps 202 and 204 .

[0096] Step 202: Perform convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector.

[0097] In an embodiment of the present application, the first feature extraction model includes multiple convolutional layers, pooling layers, and fully connected layers. The first feature extraction model receives an input fundus image and converts it into an image feature vector. The first feature extraction model includes a pre-trained feature extractor, ResNet-50. The pre-trained feature extractor of the first feature extraction model may include a self-attention layer 58 for emphasizing important parts of the fundus features. Specifically, the self-attention layer may be a standard dot-product self-attention layer 59 for calculating the weight of each pixel in the feature map generated by block 3 of the feature extractor ResNet-50. In addition, the first feature extraction model also includes a multi-layer perceptron (MLP), which serves as a predictor for estimating the weights of a fixed-size Weibull distribution for the feature vector generated by the pre-trained feature extractor ResNet-50. In one embodiment, the multi-layer perceptron is a three-layer perceptron. The output of the attention layer is a feature map of the same size as the input feature map to ensure seamless insertion into the feature extractor ResNet-50. During the interpretation phase, the attention feature map is reshaped into a specific shape and then resized for illustration.

[0098] In an optional embodiment, the image feature vector may need to be normalized or compressed to ensure that it has a good numerical range and statistical properties. Processing the normalized image feature vector can generate corresponding first weights. These weights can be used for subsequent preset hybrid model inference and combined with the information of the covariate feature vector and its corresponding second weight to obtain the target state prediction result.

[0099] Step 204 : Perform convolution calculation on the covariate data according to the second feature extraction model to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector.

[0100] In an embodiment of the present application, the terminal performs a convolution calculation on the covariate data according to the second feature extraction model to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector. The second feature extraction model may include multiple convolutional layers, pooling layers, and fully connected layers for processing the input covariate data. Through this model, the covariate data can be converted into a covariate feature vector.

[0101] In an optional embodiment, similar to the image feature vector, the covariate feature vector may need to be normalized or compressed to ensure that it has a good numerical range and statistical characteristics. The normalized covariate feature vector is processed to generate a corresponding second weight.

[0102] In this embodiment, the first and second weights can combine the covariate feature vector with the image feature vector, which are used as input for a preset hybrid model to predict the target state. By combining the feature vector of the covariate data and the second weight into the preset hybrid model, the impact of the covariate on the target state can be more comprehensively considered, thereby improving the accuracy of the target state prediction.

[0103] In an exemplary embodiment, the first feature extraction model includes a pre-trained feature extractor and a multi-layer perceptron. As shown in FIG3 , step 202 includes step 302 and step 304 .

[0104] Step 302 : extract features from the fundus image using a pre-trained feature extractor to obtain a feature map, and perform convolution calculation on the feature map based on an objective function to obtain an image feature vector.

[0105] In embodiments of the present application, the pre-trained feature extractor can employ a multi-layer convolutional neural network (CNN), a residual network (ResNet), an auto-encoder, or other models, and can be used to receive an input fundus image and extract features from it to generate a series of feature maps. These feature maps can reflect image features at different levels and scales, including information such as local details and overall structure.

[0106] The terminal can perform convolution calculations on the feature map based on a preset objective function. The objective function can be a survival function, and a convolution kernel composed of multiple trainable parameters is used to extract features of a specific type or direction. The convolution process can further process the feature map to generate a new feature map, and can also be used to compress the dimension of the feature map. After multiple convolution operations, a two-dimensional matrix containing various feature information is obtained, namely the image feature vector. The convolution process involves the setting of multiple hyperparameters and trainable parameters, including the architecture of the feature extractor, the choice of activation function, and the size and number of convolution kernels.

[0107] Step 304: Determine a first weight corresponding to the image feature vector based on a multi-layer perceptron.

[0108] In the embodiment of the present application, the multilayer perceptron is a commonly used feedforward neural network model, which consists of multiple hidden layers and an output layer. When the image feature vector is used as input, the terminal can perform weighted summation on the input through the hidden layer nodes of the multilayer perceptron, and perform nonlinear transformation through the activation function. Ultimately, the nodes of the output layer will produce corresponding prediction results. For the image feature vector, it can be used as the input of the multilayer perceptron, and the objective function can be defined as a problem that needs to be predicted or classified during training. Through backpropagation and gradient descent algorithms, the weights of each hidden layer node and output layer node can be automatically calculated, and then the first weight corresponding to the image feature vector can be determined.

[0109] In this embodiment, through feature extraction and weight determination, a more representative and informative image feature vector can be obtained, thereby improving the prediction accuracy and generalization ability of the model, and then making full use of the missing images to accurately predict the disease in a short follow-up period, thereby improving the accuracy of the target state prediction results of the observed object.

[0110] In an exemplary embodiment, as shown in FIG. 4 , step 204 includes step 402 and step 404 .

[0111] Step 402: extract features from the covariate data according to the second feature extraction model to obtain a feature graph, and perform convolution calculation on the feature graph based on the objective function to obtain a covariate feature vector.

[0112] In an embodiment of the present application, the second feature extraction model includes a metadata model. The terminal performs feature extraction on the covariate data according to the second feature extraction model, and performs convolution calculation based on the objective function to obtain a covariate feature vector. Similar to the feature extraction process of fundus images, the feature extraction of covariate data can also use various deep learning network models, such as convolutional neural networks (CNN), autoencoders (Autoencoder), recurrent neural networks (RNN), etc. The second feature extraction model performs feature extraction on the input covariate data to obtain a corresponding feature map.

[0113] After obtaining the feature map, the terminal can perform convolution calculations on the feature map based on a preset objective function. The objective function can be a survival function, and a convolution kernel composed of multiple trainable parameters is used to extract features of a specific type or direction. This convolution process can perform weighted summation and nonlinear transformations on the feature map to obtain a covariate feature vector.

[0114] In an optional embodiment, the metadata model may also take metadata as input and output predicted event occurrence times for individual participants.

[0115] Step 404: Determine a second weight corresponding to the covariate eigenvector according to the multilayer perceptron of the second feature extraction model.

[0116] In the embodiment of the present application, the second feature extraction model further includes a multi-layer MLP. In one embodiment, the multi-layer MLP is a three-layer MLP. The process of the terminal determining the second weight corresponding to the covariate eigenvector using the multi-layer perceptron is similar to that in step 304 and will not be further described in this embodiment.

[0117] In this embodiment, through covariate feature extraction and second weight determination, a more representative and informative covariate feature vector can be obtained, thereby improving the prediction accuracy and generalization ability of the model, accurately predicting the disease in a short follow-up period, and improving the accuracy of the target state prediction results of the observed object.

[0118] In an exemplary embodiment, as shown in FIG5 , before step 102 , the method further includes the step of training a first feature extraction model, specifically including steps 502 and 504 .

[0119] Step 502: Acquire a sample fundus image set.

[0120] In an embodiment of the present application, the terminal obtains a sample fundus image set, and the sample fundus image set is used to train a first feature extraction model.

[0121] Step 504 : Based on a preset loss function, a sample fundus image set, and a stochastic gradient descent algorithm, the first feature extraction model is self-supervised trained to obtain a trained first feature extraction model.

[0122] In an embodiment of the present application, the terminal performs self-supervised training on the first feature extraction model based on a preset loss function, a sample fundus image set and a stochastic gradient descent algorithm to obtain a trained first feature extraction model. In an embodiment of the present application, a momentum contrast (MoCo-v2) method is used to generate a trained first feature extraction model using self-supervised learning. During the training process, the first feature extraction model is trained based on a large number of fundus image sets without the need for manual annotation. MoCo-v2 adopts a momentum-based contrast learning framework to learn how to create positive and negative image block pairs from the same image during the pre-training process of the first feature extraction model (fundus model). The feature vector X of the observed object is input and passed through the deep learning network model f(·|Θ) to determine the parameter set φ of the mixed distribution, wherein, in order to estimate the parameters of the deep learning network model, the maximum likelihood estimation method is adopted. Considering the missing data, the maximum likelihood estimation can be expressed as:

[0123] in, Represents the sample fundus images and sample covariate data containing censored data, Θ represents the parameters of the deep learning network model, X = x i represents the feature vector of the observed object, T is a random time, representing the time when the target event occurs, for example, the time when the observed object develops DR or the disease progresses, P(T>t i ) represents T at t i The probability after that, P(T <t′ i ) indicates that T is at t′ i The probability before, or the probability of an event occurring before the current time T if no event has been observed. For example, if the subject has five follow-up examinations for a total of five years, then t1 = 1, t2 = 2, t3 = 3, t4 = 4, and t5 = 5. If the subject becomes ill in year 3.5, then T = 3.5. However, the specific value of T is actually unknown, and we can only conclude that T is between t3 and t4, so T = t3 = 3 years, and t' = t4 = 4 years.

[0124] Similarly, for uncensored data, the maximum likelihood estimate is expressed as follows:

[0125] in, Represent the fundus image and covariate data without censored data. The preset loss function is shown in formula (5):

[0126] Where γ is a hyperparameter used to adjust the weights of censored and uncensored data, and φ is the parameter set of the mixture distribution. The first feature extraction model (fundus model), the second feature extraction model (including the metadata model), and the mixture model have the same loss function.

[0127] In an optional embodiment, the first feature extraction model, the second feature extraction model and the hybrid model are trained using the same method, and the training process of the second feature extraction model and the hybrid model will not be described in detail in this embodiment.

[0128] In this embodiment, the first feature extraction model is trained through self-supervised training, and modeling is performed separately based on sample fundus images and sample covariate data containing and not containing censored data, which can improve the accuracy of the target state prediction results and thereby improve the accuracy of the prediction results.

[0129] In an exemplary embodiment, as shown in FIG6 , step 504 includes steps 602 to 606 .

[0130] Step 602: Obtain a first feature extraction model.

[0131] Among them, the first feature extraction model is constructed based on the momentum comparison method.

[0132] In an embodiment of the present application, the first feature extraction model uses contrastive learning under a self-supervised learning framework and utilizes the time series information in the data to learn useful feature representations without using any label information.

[0133] The terminal obtains a first feature extraction model, which constructs comparison samples based on a data augmentation method, builds a feature extractor based on a convolutional neural network (CNN) architecture, and uses a momentum contrast loss function. This loss function compares the similarity between the feature representations of the input sample and the corresponding comparison sample, making the features of the same sample closer and the features of different samples farther apart.

[0134] In an optional embodiment, the data enhancement method includes random image compression, random blur, brightness dithering, contrast dithering, random gamma transformation, random Gaussian noise and random rotation.

[0135] Step 604 : constructing a positive sample fundus image and a negative sample fundus image according to the first feature extraction model and each sample fundus image in the sample fundus image set.

[0136] In an embodiment of the present application, in the feature extraction of fundus images, the terminal can use a deep learning model such as a convolutional neural network (CNN) to build a feature extractor. Generally speaking, the input of the feature extractor is the fundus image, and the output is the extracted feature vector. Then, the terminal can use these feature vectors to construct a training set and a test set based on positive samples and negative samples. Specifically, fundus images based on the same category can be regarded as positive samples. The terminal can calculate the similarity between feature vectors based on fundus images of the same category for the training set and the test set to construct a positive sample library. Common similarity calculation methods include Euclidean distance, cosine similarity, etc.

[0137] Step 606 : Based on the stochastic gradient descent algorithm, the positive sample fundus images, and the negative sample fundus images, the first feature extraction model is iteratively self-supervised trained to obtain a trained first feature extraction model.

[0138] In an embodiment of the present application, the terminal uses a stochastic gradient descent algorithm for training. Stochastic gradient descent updates the parameters of the model by using the gradient estimate of each sample. During the iterative training process, a portion of samples are extracted from the training set in a small batch manner for training, and the model parameters are updated according to the calculation results of the loss function and the appropriate learning rate, optimizer type and other hyperparameters are selected to improve the effect of the training process and the performance of the model. For example, in each iteration, two 512×512 pixel crops are taken for each fundus image, and the terminal uses a k-nearest neighbor (kNN) monitor as an evaluation tool for self-supervised training, and each epoch (for sample fundus images or sample covariate data) is evaluated once. In one embodiment of the present application, for the selection of hyperparameters, ResNet-50 is used as a feature extractor and stochastic gradient descent (SGD) is used as an optimizer. The input image resolution is 512×512 pixels, the batch size is set to 256, and the MoCo v2 model is trained for 800 epochs. In one embodiment of the present application, the optimal hyperparameters were obtained using grid search, and the learning rate was obtained as 10 -3 , weight decay = 10 -4 , SGD momentum = 0.9, temperature τ = 1.0. The feature extractor momentum coefficient is m = 0.996 and increases to 1 according to the cosine schedule.

[0139] Based on experimental results, ablation experiments can be used to evaluate the predictive performance of the first feature extraction model (fundus model) pre-trained with MoCo v2. The results show that incorporating MoCo v2 into the training process can improve the predictive performance of the first feature extraction model (fundus model).

[0140] In an optional embodiment, the terminal uses a stochastic gradient descent (SGD) optimizer to train the first feature extraction model (fundus model) on the training dataset and uses a grid search to find the optimal value and other hyperparameters. The first feature extraction model (fundus model) is trained by batch error back propagation for 50 epochs on 32 images resized to 512×512 pixels, with a learning rate of 10 -5 The data augmentation strategy is exactly the same as during pre-training on MoCo v2.

[0141] In this embodiment, the first feature extraction model is trained through self-supervised training, and modeling is performed separately based on sample fundus images and sample covariate data containing and excluding censored data. This can improve the accuracy of the target state prediction results, thereby improving the accuracy of the DR prediction results, and thus extending the average screening interval from 12 months to 34.7 months, reducing the screening frequency by approximately 65.38%. At the same time, the system can identify high-risk patients so that they can receive more frequent follow-up and early detection, while low-risk patients can reduce the number of follow-up visits and reduce the screening burden. In addition, the system has high prediction accuracy and a low risk of delayed detection of vision-threatening DR. Personalized DR screening intervals and interventions can improve efficiency.

[0142] In an exemplary embodiment, as shown in FIG. 7 , step 106 includes step 702 and step 704 .

[0143] In step 702, when there is a fundus image in the data to be detected and the covariate data is empty, the second weight is determined to be zero, and a convolution calculation is performed on the image feature vector based on the preset hybrid model constructed with the objective function as the survival function and the first weight to obtain a target state prediction result.

[0144] In an embodiment of the present application, since the data to be detected of the observed object may not contain both fundus images and covariate data, for example, the observed object may lack covariate data during the observation time, the terminal may also predict the target state only through the fundus image to obtain the target state prediction result. Specifically, when the data to be detected contains a fundus image and the covariate data is empty, the terminal cannot use the covariate data to train the model, so the terminal can determine the second weight to be zero. Based on the survival function method, an objective function can be constructed, which can measure the model's predictive ability for patient survival. The survival function is used as the objective function and added to the optimization process of the preset hybrid model for training. The preset hybrid model constructed by the terminal based on the survival function as the objective function can be a variety of deep learning models, such as a model based on a convolutional neural network (CNN). The goal of the preset hybrid model is to learn to convert the feature vector into the target state prediction result. The terminal uses the previously trained first feature extraction model to extract features from the fundus image in the data to be detected to obtain a feature vector, and inputs the feature vector into the trained preset hybrid model to predict the target state using convolution calculation. Optionally, the terminal can use a softmax function to normalize the prediction results.

[0145] In step 704, when there is covariate data in the data to be detected and the fundus image is empty, the first weight is determined to be zero, and based on the preset hybrid model constructed with the objective function as the survival function and the second weight, a convolution calculation is performed on the covariate data feature vector to obtain the target state prediction result.

[0146] In this embodiment of the present application, for the same reason as step 702, the subject may lack fundus images during the observation time, resulting in only covariate data in the data to be detected. Therefore, the terminal can predict the target state of the subject based solely on the covariate data to obtain a target state prediction result. The process of the terminal performing prediction based on the data to be detected containing only covariate data is consistent with the process in step 702, and the detailed process of the convolution calculation in this embodiment will not be repeated.

[0147] In this embodiment, the target state of the observed object is predicted by using the data to be detected containing a single data, which can improve the universality of the state prediction method.

[0148] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0149] Based on the same inventive concept, embodiments of the present application also provide a state prediction device for implementing the aforementioned state prediction method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more state prediction device embodiments provided below can be found in the above-described limitations of the state prediction method and will not be further elaborated here.

[0150] In an exemplary embodiment, as shown in FIG8 , a state prediction device 800 is provided, including: a first acquisition module 801 , a feature extraction module 802 and a target state prediction module 803 .

[0151] The first acquisition module 801 is used to acquire data to be detected. The data to be detected includes fundus images and covariate data, where the covariate data is variable data that assists in identifying target lesions.

[0152] The feature extraction module 802 is used to perform feature extraction on the fundus image and covariate data based on a preset feature extraction model to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector.

[0153] Target state prediction module 803 is configured to process the image feature vector and the covariate feature vector based on the first weight, the second weight, and a preset hybrid model to obtain a target state prediction result. The hybrid model is constructed based on the image feature vector and the covariate feature vector that satisfy the maximum likelihood function conditions, both in the presence and absence of censored data.

[0154] In one embodiment, the preset feature extraction model includes a first feature extraction model and a second feature extraction model.

[0155] The feature extraction module 802 is specifically used to perform convolution calculation on the fundus image according to the first feature extraction model to obtain the image feature vector and the first weight corresponding to the image feature vector; and to perform convolution calculation on the covariate data according to the second feature extraction model to obtain the covariate feature vector and the second weight corresponding to the covariate feature vector.

[0156] In one embodiment, the first feature extraction model includes a pre-trained feature extractor and a multi-layer perceptron;

[0157] The feature extraction module 802 is specifically used to extract features from the fundus image according to the pre-trained feature extractor to obtain a feature map, and perform convolution calculation on the feature map based on the objective function to obtain an image feature vector; and determine the first weight corresponding to the image feature vector based on a multi-layer perceptron.

[0158] The feature extraction module 802 is specifically used to extract features from the covariate data according to the second feature extraction model to obtain a feature graph, and perform convolution calculation on the feature graph based on the objective function to obtain a covariate feature vector; and determine the second weight corresponding to the covariate feature vector according to the multilayer perceptron of the second feature extraction model.

[0159] In one embodiment, before acquiring the data to be detected, the apparatus 800 further includes: a second acquisition module and a training module.

[0160] The second acquisition module is used to acquire a sample fundus image set.

[0161] The training module is used to perform self-supervised training on the first feature extraction model based on a preset loss function, a sample fundus image set and a stochastic gradient descent algorithm to obtain a trained first feature extraction model.

[0162] In one embodiment, the training module is specifically used to obtain a first feature extraction model, wherein the first feature extraction model is constructed based on a momentum comparison method.

[0163] The training module is further specifically configured to construct a positive sample fundus image and a negative sample fundus image according to the first feature extraction model and each sample fundus image in the sample fundus image set.

[0164] The training module is specifically further used to perform iterative self-supervised training on the first feature extraction model based on a stochastic gradient descent algorithm, positive sample fundus images, and negative sample fundus images to obtain a trained first feature extraction model.

[0165] In one embodiment, the target state prediction module 803 is specifically used to determine the second weight to be zero when there is a fundus image in the data to be detected and the covariate data is empty, and to perform convolution calculation on the image feature vector based on the preset hybrid model constructed with the objective function as the survival function and the first weight to obtain the target state prediction result.

[0166] The target state prediction module 803 is also used to determine the first weight to be zero when there is covariate data in the data to be detected and the fundus image is empty, and to perform convolution calculation on the covariate data feature vector based on the preset hybrid model and the second weight constructed with the objective function as the survival function to obtain the target state prediction result.

[0167] Each module in the above-mentioned state prediction device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0168] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 9. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, executable instructions and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store fundus images and covariate data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the executable instructions are executed by the processor, a state prediction method is implemented.

[0169] Those skilled in the art will understand that the structure shown in Figure 9 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0170] In an exemplary embodiment of the present application, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0171] In one embodiment of the present application, a computer-readable storage medium is provided, on which executable instructions are stored. When the executable instructions are executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0172] In one embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0174] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through executable instructions. The executable instructions can be stored in a non-volatile computer-readable storage medium. When the executable instructions are executed, the processes of the embodiments of the above-mentioned methods can be implemented. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0175] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A state prediction method, comprising: Acquire data to be detected, wherein the data to be detected includes a fundus image and covariate data, wherein the covariate data is variable data for assisting in distinguishing target lesions; Based on a preset feature extraction model, feature extraction is performed on the fundus image and the covariate data respectively to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector; and Based on the first weight, the second weight and a preset hybrid model, the image feature vector and the covariate feature vector are processed to obtain a target state prediction result, wherein the preset hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function conditions when there are missing data or when there are no missing data in the data to be detected.

2. The method according to claim 1, characterized in that The preset feature extraction model includes a first feature extraction model and a second feature extraction model; The method of extracting features from the fundus image and the covariate data based on a preset feature extraction model to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector includes: performing convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector; and According to the second feature extraction model, convolution calculation is performed on the covariate data to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector.

3. The method according to claim 2, characterized in that The first feature extraction model includes a pre-trained feature extractor and a multi-layer perceptron; The step of performing convolution calculation on the fundus image according to the first feature extraction model to obtain an image feature vector and a first weight corresponding to the image feature vector includes: Extracting features from the fundus image according to the pre-trained feature extractor to obtain a feature map, and performing convolution calculation on the feature map based on an objective function to obtain an image feature vector; and A first weight corresponding to the image feature vector is determined based on the multilayer perceptron.

4. The method according to claim 2 or 3, characterized in that: The step of performing convolution calculation on the covariate data according to the second feature extraction model to obtain a covariate feature vector and a second weight corresponding to the covariate feature vector includes: The covariate data is subjected to feature extraction according to the second feature extraction model to obtain a feature graph, and based on the target The function performs convolution on the feature map to obtain the covariate feature vector; and A second weight corresponding to the covariate feature vector is determined according to the multilayer perceptron of the second feature extraction model.

5. The method according to any one of claims 1 to 4, characterized in that: Before obtaining the data to be detected, the method further includes: acquiring a sample fundus image set; and Based on a preset loss function, the sample fundus image set and a stochastic gradient descent algorithm, the first feature extraction model is self-supervised trained to obtain a trained first feature extraction model.

6. The method according to claim 5, characterized in that The self-supervised training of the first feature extraction model based on the preset loss function, the sample fundus image set and the stochastic gradient descent algorithm to obtain the trained first feature extraction model includes: Acquire a first feature extraction model; the first feature extraction model is constructed based on a momentum comparison method; constructing a positive sample fundus image and a negative sample fundus image according to the first feature extraction model and each of the sample fundus images in the sample fundus image set; and Based on the stochastic gradient descent algorithm, the positive sample fundus image and the negative sample fundus image, the first feature extraction model is iteratively self-supervised trained to obtain a trained first feature extraction model.

7. The method according to any one of claims 1 to 6, characterized in that: The processing of the image feature vector and the covariate feature vector based on the first weight, the second weight and the preset hybrid model to obtain a target state prediction result includes: For the case where the fundus image exists in the data to be detected and the covariate data is empty, the second weight is determined to be zero, and based on the preset hybrid model constructed with the objective function being the survival function and the first weight, a convolution calculation is performed on the image feature vector to obtain a target state prediction result; and In the case where the covariate data exists in the data to be detected and the fundus image is empty, the first weight is determined to be zero, and based on the preset mixed model constructed with the objective function as the survival function and the second weight, a convolution calculation is performed on the covariate data feature vector to obtain a target state prediction result.

8. The method according to any one of claims 1 to 7, characterized in that: The preset mixed model consists of multiple Weibull distributions.

9. The method according to any one of claims 1 to 8, characterized in that: After extracting features from the fundus image and the covariate data based on the preset feature extraction model to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector, the method further includes: Normalizing or compressing the image feature vector; and / or The covariate eigenvector is normalized or compressed.

10. The method according to claim 3, characterized in that The multi-layer perceptron is a 3-layer perceptron.

11. The method according to claim 6, characterized in that: The size of the positive sample fundus image and the negative sample fundus image is 512×512 pixels; and In each iteration, a k-nearest neighbor (kNN) monitor is used as an evaluation tool for self-supervised training.

12. A state prediction device, characterized in that: The device comprises: A first acquisition module is used to acquire data to be detected, wherein the data to be detected includes fundus images and covariate data, and the covariate data is variable data for assisting in distinguishing target lesions; A feature extraction module, used to perform feature extraction on the fundus image and the covariate data based on a preset feature extraction model, to obtain an image feature vector, a first weight corresponding to the image feature vector, a covariate feature vector, and a second weight corresponding to the covariate feature vector; A target state prediction module is used to process the image feature vector and the covariate feature vector based on the first weight, the second weight and a preset hybrid model to obtain a target state prediction result, wherein the preset hybrid model is constructed based on the image feature vector and the covariate feature vector that respectively satisfy the maximum likelihood function conditions when there are missing data or when there are no missing data in the data to be detected.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A non-volatile computer-readable storage medium having executable instructions stored thereon, characterized in that: When the executable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images