A risk assessment method, device, storage medium and electronic device

Abstract

The present disclosure discloses a risk assessment method, apparatus, storage medium, and electronic device, relating to the technical field of neural networks. The risk assessment method includes: generating M sets of region-of-interest image sequences based on M sets of initial image sequences of a target user, where the modalities of the M sets of initial image sequences are different from each other, and M is a positive integer greater than or equal to 2; using a survival analysis model, based on the M sets of region-of-interest image sequences and the clinical information data of the target user, determining the survival risk data of the target user, where the survival analysis model is constructed based on a deep learning framework and outputs risk assessment data by splicing an image feature vector and a clinical information feature vector. The present disclosure performs a survival risk assessment on the target user through the image features and clinical information features of the patient, obtains a more accurate assessment result, assists the doctor in diagnosing and treating the target user, and thus helps the doctor improve the survival rate of the target user.

Description

Technical Field

[0001] The present disclosure relates to the technical field of neural networks, and particularly to a risk assessment method, apparatus, storage medium, and electronic device. Background Art

[0002] Nowadays, malignant tumors are one of the diseases with the highest incidence rates globally. As the most common intracranial malignant tumor, glioma is still mainly treated by surgery at present. However, glioma is difficult to cure completely, the disease progresses relatively fast, and it is prone to recurrence after surgery, resulting in a reduced survival probability of glioma patients. Therefore, in order to be able to treat in a timely manner, prevent tumor growth, and delay recurrence, survival risk analysis and assessment are particularly important. However, due to different types of malignant tumors and different degrees of disease development, despite the rapid progress of tumor treatment levels, the prognosis of high-grade gliomas is still poor. Doctors need to treat and diagnose different patients based on their own experience, and doctors need to spend a lot of time accumulating experience. Therefore, there is an urgent need for a survival risk analysis that can help doctors be faster and more accurate, so as to help doctors better diagnose and predict recurrence of glioma patients. Summary of the Invention

[0003] In view of this, the present disclosure provides a risk assessment method, apparatus, storage medium, and electronic device, which can quickly and highly accurately perform risk assessment on glioma patients, so as to solve the problem of poor prognosis of glioma patients, and can help with diagnosis and more accurately predict recurrence.

[0004] In a first aspect, an embodiment of the present disclosure provides a risk assessment method, which includes: generating M sets of region of interest (ROI) image sequences based on M sets of initial image sequences of a target user, where the modalities of the M sets of initial image sequences are different from each other, and M is a positive integer greater than or equal to 2; using a survival analysis model, based on the M sets of ROI image sequences and the clinical information data of the target user, determining the survival risk data of the target user, where the survival analysis model is constructed based on a deep learning framework, and outputs risk assessment data by splicing an image feature vector and a clinical information feature vector.

[0005] In combination with the first aspect, in some implementation manners of the first aspect, using a survival analysis model, based on the M sets of ROI image sequences and the clinical information data of the target user, determining the survival risk data of the target user includes: using the survival analysis model to perform feature extraction on the M sets of ROI image sequences to obtain an image feature vector; using the survival analysis model to perform feature extraction on the clinical information data to obtain a clinical information feature vector; splicing the image feature vector and the clinical information feature vector to obtain a splicing vector that can represent the target user; and based on the splicing vector, obtaining the survival risk data of the target user.

[0006] In combination with the first aspect, in certain implementations of the first aspect, the survival analysis model includes a deep convolutional neural network encoder, a long short-term memory network module, a multi-layer perceptron encoder, and a Cox fully-connected layer. Among them, using the survival analysis model, feature extraction is performed on M sets of region of interest image sequences to obtain image feature vectors, including: using the deep convolutional neural network encoder to perform feature extraction on each of the M sets of region of interest image sequences to obtain M initial feature vectors; using the long short-term memory network module to fuse the M initial feature vectors to obtain image feature vectors; among them, using the survival analysis model, feature extraction is performed on clinical information data to obtain clinical information feature vectors, including: using the multi-layer perceptron encoder to perform feature extraction on clinical information data to obtain clinical information feature vectors; among them, based on the concatenated vector, survival risk data of the target user is obtained, including: using the Cox fully-connected layer to process the concatenated vector to obtain the survival risk data of the target user.

[0007] In combination with the first aspect, in certain implementations of the first aspect, based on M sets of initial image sequences of the target user, M sets of region of interest image sequences are generated, including: performing region of interest calibration on the M sets of initial image sequences of the target user to obtain initial rectangular frames of regions of interest corresponding to each of the M sets of initial image sequences; for each initial rectangular frame of the region of interest, expanding each side of the initial rectangular frame of the region of interest away from the center of the region of interest by at least one unit pixel distance to obtain an extended rectangular frame of the region of interest corresponding to the initial rectangular frame of the region of interest; based on the extended rectangular frames of the regions of interest corresponding to each of the M sets of initial image sequences, segmenting each of the M sets of initial image sequences to obtain M sets of region of interest image sequences.

[0008] In combination with the first aspect, in certain implementations of the first aspect, before using the survival analysis model to determine the survival risk data of the target user based on M sets of region of interest image sequences and the clinical information data of the target user, it further includes: generating P sets of region of interest image sample sequences based on P sets of initial image sample sequences, where the P sets of region of interest image sample sequences correspond to N sample users, N is a positive integer less than P, and P is a positive integer greater than or equal to 2; based on the P sets of region of interest image sample sequences and the clinical information data of each of the N sample users, generating sample data sets corresponding to each of the N sample users with the sample users as units, where the region of interest image sample sequences included in the sample data sets are arranged in descending order of survival time; for each sample data set, using the model to be trained to generate image feature vectors and clinical information feature vectors corresponding to the sample data set, and adjusting the model parameters of the model to be trained based on the image feature vectors and clinical information feature vectors corresponding to the sample data set until the model to be trained reaches a preset convergence condition to obtain the survival analysis model.

[0009] In combination with the first aspect, in some implementations of the first aspect, after generating the sample data sets corresponding to each of the N sample users, it further includes: partitioning the sample data sets corresponding to each of the N sample users to obtain a training data set and a validation data set, where the quantity ratio of the training data set to the validation data set is 3:1; and for each sample data set, using the model to be trained to generate an image feature vector and a clinical information feature vector corresponding to the sample data set, and adjusting the model parameters of the model to be trained based on the image feature vector and the clinical information feature vector corresponding to the sample data set until the model to be trained reaches a preset convergence condition, to obtain a survival analysis model, including: for each training data set, using the model to be trained to generate an image feature vector and a clinical information feature vector corresponding to the training data set, and adjusting the model parameters of the model to be trained based on the image feature vector and the clinical information feature vector corresponding to the training data set; using the validation data set to evaluate the model to be trained with adjusted parameters by means of C-Index to obtain an evaluation result; if the evaluation result meets the preset convergence condition, obtaining the survival analysis model.

[0010] In combination with the first aspect, in some implementations of the first aspect, the target user is a tumor patient, and the tumor patient includes a glioma patient; the initial image sequence includes the image sequence of the part where the patient's lesion is located; the region of interest is the lesion region of the patient.

[0011] In a second aspect, an embodiment of the present disclosure discloses a risk assessment device, which includes: a generation module, configured to generate M sets of region-of-interest image sequences based on M sets of initial image sequences of a target user, where the modalities of each of the M sets of initial image sequences are different from each other, and M is a positive integer greater than or equal to 2; a determination module, configured to use the survival analysis model to determine the survival risk data of the target user based on the M sets of region-of-interest image sequences and the clinical information data of the target user, where the survival analysis model is constructed based on a deep learning framework and outputs the survival risk data by splicing the image feature vector and the clinical information feature vector.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor, where the processor is configured to execute the method mentioned in the first aspect above.

[0013] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program for executing the method mentioned in the first aspect above.

[0014] The risk assessment method provided by the embodiments of the present disclosure, through different modalities of image data and clinical data, splices the characteristics of the image data and clinical data, trains a survival analysis model based on the spliced vector, and evaluates the survival analysis model using the C-Index through continuous adjustment of the feature vector. Finally, an optimal survival analysis model is obtained to perform risk assessment on the sample data of glioma patients, which can improve the clinical diagnosis and treatment effect of glioma patients. At the same time, it can more accurately obtain relevant survival information such as the disease development and recurrence probability of glioma patients, and improve the prognosis level of glioma patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present disclosure in more detail with reference to the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure, and do not constitute a limitation to the present disclosure.

[0016] Figure 1 The following shows a schematic diagram of an application scenario provided by an embodiment of the present disclosure.

[0017] Figure 2 The following shows a schematic diagram of the flow of a risk assessment method provided by an embodiment of the present disclosure.

[0018] Figure 3 The following shows a schematic diagram of the process of determining the survival risk data of a target user based on M sets of region-of-interest image sequences and the clinical information data of the target user using a survival analysis model provided by an embodiment of the present disclosure.

[0019] Figure 4 The following shows a schematic diagram of the process of obtaining an image feature vector and a clinical information feature vector using a survival analysis model provided by an embodiment of the present disclosure.

[0020] Figure 5 The following shows a schematic diagram of the process of performing survival analysis using a survival analysis model provided by an embodiment of the present disclosure.

[0021] Figure 6 The following shows a schematic diagram of the process of generating M sets of region-of-interest image sequences based on M sets of initial image sequences of a target user provided by an embodiment of the present disclosure.

[0022] Figure 7 The following shows a schematic diagram of the flow of a risk assessment method provided by another embodiment of the present disclosure.

[0023] Figure 8 The following shows a schematic diagram of the flow of a risk assessment method provided by another embodiment of the present disclosure.

[0024] Figure 9The figure shows a schematic diagram of a risk assessment device provided by an embodiment of the present disclosure.

[0025] Figure 10 The figure shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0026] Next, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments.

[0027] In recent years, malignant tumors have become one of the diseases with the highest incidence rates globally and have become diseases that seriously threaten people's lives and health. For different types of malignant tumors, the survival rates of patients are also different. Usually, the main method for treating malignant tumors is surgical treatment. As a common intracranial malignant tumor, gliomas have a high recurrence rate after surgery, and there are various different types of gliomas. The treatments and prognosis effects of different types of gliomas are also different. Although the level of tumor diagnosis and treatment has been continuously improved in recent years, the prognosis of high-grade gliomas is still poor. Existing prognosis analysis technologies still cannot help doctors quickly and accurately evaluate the survival probability of glioma patients, cannot help doctors provide more accurate development results for the diagnosis and treatment of glioma patients, and cannot help doctors improve the survival rate of glioma patients. Therefore, it can be seen that how to quickly and accurately evaluate the survival probability of gliomas is an urgent problem to be solved.

[0028] Next, a simple introduction to the application scenario of an embodiment of the present disclosure will be given in conjunction with Figure 1 a simple introduction to the application scenario of an embodiment of the present disclosure will be given.

[0029] Figure 1 The figure shows a schematic diagram of the application scenario of an embodiment of the present disclosure. As Figure 1 shown, this scenario is a scenario for risk scoring of glioma patients. Specifically, the risk scoring scenario includes a server 110, a user terminal 120 and a data storage device 130 that are respectively communicatively connected to the server 110. The server 110 is used to execute the risk scoring method mentioned in the embodiments of the present disclosure.

[0030] Exemplarily, in the actual application process, the user uses the user terminal 120 to send a risk scoring instruction for glioma patient A to the server 110. After receiving the instruction, the server 110 retrieves the patient data of glioma patient A (including M sets of image sequences of regions of interest and clinical information data) from the data storage device 130, generates a risk scoring result for glioma patient A based on the M sets of image sequences of regions of interest and clinical information data through a survival analysis model, and then sends the risk scoring result to the user terminal 120 so that the user terminal 120 can present the result to the user.

[0031] Exemplarily, the aforementioned user terminal 120 includes, but is not limited to, computer terminals such as desktop computers and laptop computers, and mobile terminals such as tablet computers and mobile phones. The data stored in the aforementioned data storage device 130 includes, but is not limited to, all the imaging data and clinical data saved by the medical institution during the treatment of patient A, and the relevant lesion area imaging data and clinical information data of patient A input by the user.

[0032] The following will Figures 2 to 7 give a brief introduction to the risk assessment method of the present disclosure.

[0033] Figure 2 Shown is a schematic flowchart of a risk assessment method provided by an embodiment of the present disclosure. As Figure 2 shown, the risk assessment method provided by the embodiment of the present disclosure includes the following steps.

[0034] Step S210, based on M sets of initial image sequences of the target user, generate M sets of region-of-interest image sequences.

[0035] Exemplarily, select glioma patients as the target users, and the region-of-interest image sequences include the lesion area image sequences of gliomas. That is to say, according to the magnetic resonance imaging (MRI) sequence data of the lesion location of the target user, such as the brain imaging sequences of brain glioma patients, select Flair sequence images, T1 sequence images, T1CE sequence images, and T2 sequence images, and generate M sets of region-of-interest image sequences with the lesion area as the region of interest, so that the risk assessment results are more accurate, the workload of the survival analysis model is less, and the risk assessment speed is improved. It should be understood that the M sets of initial image sequences of the target user include, but are not limited to, Flair sequence images, T1 sequence images, T1CE sequence images, and T2 sequence images. Each set of initial image sequences may include multiple image data of the same sequence, that is, multiple imaging data in different modalities, or multiple imaging data in one modality. It should be understood that the target users include, but are not limited to, tumor patients such as glioma patients and nasopharyngeal carcinoma patients.

[0036] Step S220, using the survival analysis model, based on the M sets of region-of-interest image sequences and the clinical information data of the target user, determine the survival risk data of the target user.

[0037] Exemplarily, using the above M sets of region-of-interest image sequences and the clinical information data of glioma patients, after taking the above lesion area as the region of interest, input it into the constructed and trained survival analysis model. The survival analysis model is modeled using the PyTorch framework. By inputting the MRI image sequences and clinical information data, obtain the lesion area image feature vector and clinical information, splice the lesion area image feature vector and clinical information, and output the risk assessment data.

[0038] Exemplarily, before generating M sets of region-of-interest image sequences based on the target user's M sets of initial image sequences, the target user's initial data (including the initial image sequence and corresponding clinical information data) should also be cleaned and preprocessed. Data cleaning includes, but is not limited to, removing image data of non-lesion sites, removing impact data with poor imaging quality, and removing patient data where image sequence data and corresponding clinical data do not match; data preprocessing includes, but is not limited to, drawing an initial border on the image data, with the initial border range just encompassing the lesion area, filling missing values and normalizing the clinical information data; it should be understood that in order to ensure the accuracy of the survival analysis model used, the above-mentioned data cleaning and preprocessing are also required when training the survival analysis model.

[0039] Since the survival analysis model used is built based on a deep learning framework, it can have higher accuracy than traditional algorithms. At the same time, by splicing image feature vectors and clinical information feature vectors, it can combine the lesion area and clinical information. It can be used in the case of one modality data or multiple modal data, and has application accessibility. In addition, the embodiment of the present disclosure can use the survival analysis model to improve the accuracy and robustness of risk assessment results.

[0040] Figure 3 The figure shows a flow chart of determining the survival risk data of a target user using a survival analysis model according to an embodiment of the present disclosure based on M sets of image sequences of regions of interest and clinical information data of the target user. Figure 2 Based on the embodiment shown Figure 3 The embodiment shown is described below in detail. Figure 3 The embodiment shown and Figure 2 The differences and similarities between the illustrated embodiments are not described in detail.

[0041] like Figure 3 As shown, the present disclosure provides a flow chart for determining the survival risk data of a target user using a survival analysis model based on M sets of region of interest image sequences and clinical information data of the target user, including the following steps.

[0042] Step S310 : Using the survival analysis model, feature extraction is performed on M sets of region of interest image sequences to obtain image feature vectors.

[0043] Exemplarily, the above-mentioned pre-built and trained survival analysis model is used to obtain corresponding feature vectors under different modalities through a deep convolutional network model, and the corresponding feature vectors under different modalities are fused to obtain the image feature vector of the target user.

[0044] Step S320: Using a survival analysis model, extract features from the clinical information data to obtain a clinical information feature vector.

[0045] Exemplarily, using a pre-constructed and trained survival analysis model, the survival analysis model processes the clinical information through multiple fully connected layers to obtain a clinical information feature vector corresponding to the target image feature vector.

[0046] Step S330: Concatenate the image feature vector and the clinical information feature vector to obtain a concatenated vector that can represent the target user.

[0047] Exemplarily, concatenating the image feature vector of the target user and the corresponding clinical information feature vector, the obtained concatenated vector can directly judge the factors affecting the survival probability of the target user, improve the accuracy of the output risk assessment result, and play a positive role in diagnosis and treatment.

[0048] Step S340: Based on the concatenated vector, obtain the survival risk data of the target user.

[0049] Since the used survival analysis model processes the imaging information and the clinical information separately, the computational complexity of the model is reduced, the computational amount of the survival analysis model is reduced, and the output speed of the risk assessment result is improved. Therefore, the embodiments of the present disclosure can quickly and efficiently obtain the survival probability result of the target user.

[0050] Figure 4 Shown is a schematic flowchart of obtaining an image feature vector and a clinical information feature vector using a survival analysis model provided by an embodiment of the present disclosure. As Figure 4 shown, in an embodiment of the present disclosure, obtaining an image feature vector and a clinical information feature vector using a survival analysis model includes the following steps.

[0051] Step S410: Using a deep convolutional neural network encoder, extract features from M sets of region of interest image sequences respectively to obtain M initial feature vectors.

[0052] Exemplarily, inputting the above M sets of lesion region image sequences into the survival analysis model, the image information is processed by a deep convolutional neural network Resnet encoder to obtain the corresponding M initial feature vectors. It should be understood that the used convolutional neural network includes but is not limited to the Resnet deep convolutional neural network and the Denset deep convolutional neural network.

[0053] Step S420: Using a Long Short Term Memory (LSTM) module, fuse the M initial feature vectors to obtain an image feature vector.

[0054] Exemplarily, the M initial feature vectors obtained above, the feature vectors of different modalities, are fused through a long short-term memory network module to obtain a single fused image feature vector.

[0055] Step S430: Use a multi-layer perceptron encoder to extract features from the clinical information data to obtain a clinical information feature vector.

[0056] Exemplarily, while the above-mentioned processing of image information to obtain image features, the survival analysis model processes the clinical information, and through a multi-layer perceptron encoder, extracts features from the clinical information data.

[0057] Step S440: Use a Cox fully connected layer to process the concatenated vector to obtain the survival risk data of the target user.

[0058] Exemplarily, the survival analysis model concatenates the obtained image information feature vector and clinical information feature vector to obtain a concatenated vector, and the concatenated vector passes through a Cox fully connected layer to output the survival risk data of the target user.

[0059] Exemplarily, the survival analysis model includes a deep convolutional neural network encoder, a long short-term memory network (LSTM) module, a multi-layer perceptron encoder, and a Cox fully connected layer.

[0060] Since the survival analysis model processes image information and clinical information and fuses the feature vectors in multiple modalities, it improves the accuracy of the output result of the survival analysis model, that is, improves the accuracy of the risk assessment result. Therefore, the embodiments of the present disclosure can accurately obtain the survival probability result of the target user.

[0061] Figure 5 The following shows a schematic flowchart of the survival analysis performed by the survival analysis model provided by an embodiment of the present disclosure. As Figure 5 shown, in an embodiment of the present disclosure, the process of the survival analysis model performing survival analysis includes the following steps.

[0062] Step S510: Input user sample data, and randomly divide each sample data into N different small sample data according to the user as a unit.

[0063] Exemplarily, the input patient sample data is randomly divided into Batch0, Batch1,..., Batch k ,..., Batch n , N different Batch sample data.

[0064] Step S520: Sort the data within each small sample in descending order according to the survival time.

[0065] Exemplarily, sort each of the above Batch samples in descending order of survival time within the sample.

[0066] Step S530: Pass the image information in each small sample through the deep convolutional neural network in sequence to obtain feature vectors corresponding to different modalities.

[0067] Exemplarily, pass the image information in each of the above Batch samples through the deep convolutional neural network in sequence to obtain corresponding feature vectors F0, F1, F2, ..., F n , where the feature vector F0 corresponds to the image of modality 0, the feature vector F1 corresponds to the image of modality 1, and so on, and the feature vector F n corresponds to the image of modality N.

[0068] Step S540: Input the feature vectors of different modalities obtained for each small sample into the LSTM network module to fuse the feature vectors of multiple modalities and obtain an image feature vector.

[0069] Exemplarily, input the feature vectors F0, F1, F2, ..., F of different modalities obtained for each Batch sample n , into the LSTM network module, and fuse F0, F1, F2, ..., F n , the N feature vectors to obtain an image feature vector F.

[0070] Step S550: Pass the clinical information data in each small sample through the multi-layer perceptron encoder to obtain corresponding clinical information feature vectors.

[0071] Exemplarily, while processing the image features, process the clinical information data. Input the clinical information of the same Batch sample corresponding to the above image data into the multi-layer perceptron encoder for processing to obtain the clinical information feature vector F' of each Batch sample.

[0072] Step S560: Use the data of the same small sample to construct a Cox loss function for model supervised learning.

[0073] Exemplarily, while extracting the feature vectors, use the data of the same Batch sample to construct a Cox loss function for model supervised learning.

[0074] Step S570: Concatenate the image feature vector and the clinical information feature vector to obtain a feature vector representing the sample, and output a risk score through the Cox fully connected layer.

[0075] Exemplarily, concatenate the obtained image feature vector F and the clinical information feature vector to obtain a concatenated vector, and the concatenated vector outputs a risk score through the Cox fully connected layer.

[0076] Figure 6 The following is a schematic flow chart of generating M sets of region-of-interest image sequences based on M sets of initial image sequences of a target user provided by an embodiment of the present disclosure. As Figure 6 shown, the specific steps of generating M sets of region-of-interest image sequences based on M sets of initial image sequences of a target user are as follows.

[0077] Step S610: Perform region-of-interest calibration on M sets of initial image sequences of the target user to obtain initial rectangular frames of regions of interest corresponding to each of the M sets of initial image sequences.

[0078] Exemplarily, when performing region-of-interest calibration on M sets of initial image sequences of the target user, use a rectangular frame to just enclose the region of interest, and obtain initial rectangular frames of regions of interest corresponding to each of the M sets of initial image sequences

[0079] Step S620: For each initial rectangular frame of the region of interest, expand each side of the initial rectangular frame of the region of interest away from the center of the region of interest by at least one unit pixel distance to obtain an expanded rectangular frame of the region of interest corresponding to the initial rectangular frame of the region of interest.

[0080] Exemplarily, for each initial rectangular frame of the region of interest, that is, the rectangular frame that just encloses the region of interest, expand each side of the rectangular frame away from the center of the region of interest by a distance of one unit pixel to obtain an expanded rectangular frame corresponding to the initial frame of the region of interest.

[0081] Step S630: Based on the expanded rectangular frames of regions of interest corresponding to each of the M sets of initial image sequences, respectively segment the M sets of initial image sequences to obtain M sets of region-of-interest image sequences.

[0082] Exemplarily, based on the expanded rectangular frames of regions of interest corresponding to each of the M sets of initial image sequences, obtain M sets of region-of-interest image sequences, that is, image sequences that can include all the boundaries of the tumor.

[0083] Using the selected expanded rectangular frame to process the image sequence can completely contain the tumor boundary and reduce the inconsistent situation existing in the current technology by manually delineating the region. And according to clinical knowledge, fully considering that the type of tumor is related to the microenvironment where the tumor is located, the accuracy and efficiency of the survival analysis model algorithm are improved. Therefore, the embodiment of the present disclosure can obtain the survival probability result of the target user with higher accuracy.

[0084] Figure 7 The following is a schematic flow chart of a risk assessment method provided by another embodiment of the present disclosure. On the basis of the Figure 2 embodiment shown, the Figure 6 embodiment shown is extended, and the following will focus on describing Figure 7The embodiment shown and Figure 2 the differences between the embodiments shown will not be elaborated, and the similarities will not be repeated.

[0085] As Figure 7 shown, in another embodiment of the present disclosure, before determining the survival risk data of the target user based on the M sets of region of interest image sequences and the clinical information data of the target user using the survival analysis model, the following steps are further included.

[0086] Step S211: Generate P sets of region of interest image sample sequences based on P sets of initial image sample sequences.

[0087] Step S212: Based on the P sets of region of interest image sample sequences and the clinical information data of each of the N sample users, generate sample data sets corresponding to each of the N sample users in units of the sample users.

[0088] Exemplarily, based on the P sets of region of interest image sample sequences and the clinical information data of each of the N sample users, in units of the sample users, compare the P sets of region of interest image sample sequences with the clinical information data of the N sample users to generate sample data sets corresponding to each of the N sample users.

[0089] Step S213: For each sample data set, use the model to be trained to generate an image feature vector and a clinical information feature vector corresponding to the sample data set, and adjust the model parameters of the model to be trained based on the image feature vector and the clinical information feature vector corresponding to the sample data set until the model to be trained reaches the preset convergence condition, and obtain the survival analysis model.

[0090] Exemplarily, the model parameters of the training model are selected as the neuron weights and biases of the deep convolutional neural network, the multi-layer perceptron, and the Cox fully connected layer in the deep convolutional neural network encoder, and the neuron weights and biases together constitute the model parameters of the training model. Through calculation for the parameter set, the preset convergence condition threshold is 0.9, and if it is lower than the value of 0.9, continue training until the preset convergence condition is reached to obtain the survival analysis model.

[0091] The survival analysis model obtained through training and reaching the preset convergence condition is the optimal survival analysis model with the best effect, which can improve the accuracy of the survival analysis model, thereby improving the accuracy of the survival risk data, being more conducive to the diagnosis and subsequent treatment of the target user, and improving the prognosis level.

[0092] Figure 8 Shown is a schematic flowchart of a risk assessment method provided by another embodiment of the present disclosure, which extends from the Figure 7 embodiment shown, and Figure 8 the embodiment shown is extended. The following focuses on Figure 8 the differences between the embodiment shown and Figure 7Differences between the illustrated embodiments will not be elaborated, and the same parts will not be repeated.

[0093] As Figure 8 shown, in another embodiment of the present disclosure, after generating the sample data sets corresponding to each of the N sample users, the following steps are further included.

[0094] Step S2131: Divide the corresponding sample data set to obtain a training data set and a validation data set, where the quantity ratio of the training data set to the validation data set is 3:1.

[0095] Exemplarily, the corresponding sample data sets are combined and randomly divided according to the ratio of 3:1 to obtain a training data set and a validation data set, that is, the sample data set is randomly divided, and the ratio of the number of sample data sets included in the training set data to the number of sample data sets included in the validation set is 3:1.

[0096] Step S2132: For each training data set, use the model to be trained to generate the image feature vectors and clinical information feature vectors corresponding to the training data set, and adjust the model parameters of the model to be trained based on the image feature vectors and clinical information feature vectors corresponding to the training data set.

[0097] Exemplarily, for the sample data sets corresponding to each sample user, use the model to be trained to generate the image feature vectors and clinical information feature vectors corresponding to each sample customer in the training set. According to the concatenated feature vectors obtained from the training set, obtain the model parameters of the training model, retain the model parameters of the model to be trained, and adjust them according to the preset conditions.

[0098] Step S2133: Use the validation data set to evaluate the model to be trained with adjusted parameters by using C-Index to obtain an evaluation result; if the evaluation result meets the preset convergence condition, obtain a survival analysis model.

[0099] Exemplarily, according to the validation data set, evaluate the parameters of the model to be trained by using C-Index. According to the calculated corresponding C-Index value, evaluate the model to be trained. If the obtained C-Index value is lower than 0.90, it does not meet the preset convergence condition, and then continue the iterative training of the model until finally obtaining an optimal model with a high accuracy with a C-Index value higher than 0.90. Among them, C-Index is between 0.50 and 1.00. The range with a C-Index between 0.50 and 0.70 is a range of lower accuracy, the range with a C-Index between 0.71 and 0.90 is a range of medium accuracy, and the range with a C-Index higher than 0.91 to 1.00 is a range of high accuracy.

[0100] Iteratively optimize the model to be trained through the C-Index, select a relatively high threshold, which can improve the accuracy of the survival analysis model, further improve the accuracy of the risk assessment method, obtain more accurate output survival risk data, improve the prognosis level of glioma patients, and provide a reliable basis for diagnosing the condition.

[0101] As described above in conjunction with Figures 2 to 8 , the method embodiments of the present disclosure have been described in detail. Next, in conjunction with Figure 9 and Figure 10 , the device embodiments of the present disclosure will be described in detail. In addition, it should be understood that the descriptions of the method embodiments correspond to those of the device embodiments. Therefore, for parts not described in detail, reference may be made to the previous method embodiments.

[0102] Figure 9 The following shows a schematic structural diagram of a risk assessment device provided by an embodiment of the present disclosure. As shown in Figure 9 , the risk assessment device provided by the embodiment of the present disclosure includes a generation module 910 and a determination module 920.

[0103] Specifically, the generation module 910 is configured to generate M sets of region-of-interest image sequences based on M sets of initial image sequences of a target user, where the modalities of the M sets of initial image sequences are different from each other, and M is a positive integer greater than or equal to 2. The determination module 920 is configured to use a survival analysis model to determine the survival risk data of the target user based on the M sets of region-of-interest image sequences and the clinical information data of the target user, where the survival analysis model is constructed based on a deep learning framework and outputs the survival risk data by splicing an image feature vector and a clinical information feature vector.

[0104] In some embodiments, the generation module 910 is further configured to perform region-of-interest calibration on the M sets of initial image sequences of the target user to obtain initial rectangular frames of regions of interest corresponding to the M sets of initial image sequences; for each initial rectangular frame of a region of interest, expand each side of the initial rectangular frame of the region of interest away from the center of the region of interest by at least one unit pixel distance to obtain an extended rectangular frame of the region of interest corresponding to the initial rectangular frame of the region of interest; and based on the extended rectangular frames of the regions of interest corresponding to the M sets of initial image sequences, respectively segment the M sets of initial image sequences to obtain M sets of region-of-interest image sequences.

[0105] In some embodiments, the determination module 920 is further configured to use the survival analysis model to extract features from the M sets of region-of-interest image sequences to obtain an image feature vector; use the survival analysis model to extract features from the clinical information data to obtain a clinical information feature vector; splice the image feature vector and the clinical information feature vector to obtain a splicing vector that can represent the target user; and based on the splicing vector, obtain the survival risk data of the target user.

[0106] In some embodiments, the determining module 920 is further configured to use a survival analysis model to extract features from M sets of image sequences of regions of interest, and obtain image feature vectors, including: using a deep convolutional neural network encoder to extract features from each of the M sets of image sequences of regions of interest to obtain M initial feature vectors; using a long short-term memory network module to fuse the M initial feature vectors to obtain image feature vectors; wherein, using the survival analysis model to extract features from clinical information data to obtain clinical information feature vectors, including: using a multi-layer perceptron encoder to extract features from clinical information data to obtain clinical information feature vectors; wherein, based on the concatenated vector, the survival risk data of the target user is obtained, including: using a Cox fully connected layer to process the concatenated vector to obtain the survival risk data of the target user.

[0107] In some embodiments, the determining module 920 is further configured to generate P sets of image sample sequences of regions of interest based on P sets of initial image sample sequences, wherein the P sets of image sample sequences of regions of interest correspond to N sample users, N is a positive integer less than P, and P is a positive integer greater than or equal to 2; based on the P sets of image sample sequences of regions of interest and the respective clinical information data of the N sample users, taking the sample users as units, generate respective sample data sets for the N sample users, wherein the image sample sequences included in the sample data sets are arranged in descending order of survival time; for each sample data set, use the model to be trained to generate the corresponding image feature vectors and clinical information feature vectors of the sample data set, and adjust the model parameters of the model to be trained based on the corresponding image feature vectors and clinical information feature vectors of the sample data set until the model to be trained reaches a preset convergence condition to obtain a survival analysis model.

[0108] In some embodiments, the determining module 920 is further configured to partition the respective sample data sets of the N sample users to obtain a training data set and a validation data set, wherein the quantity ratio of the training data set to the validation data set is 3:1; and, for each sample data set, use the model to be trained to generate the corresponding image feature vectors and clinical information feature vectors of the sample data set, and adjust the model parameters of the model to be trained based on the corresponding image feature vectors and clinical information feature vectors of the sample data set until the model to be trained reaches a preset convergence condition to obtain a survival analysis model, including: for each training data set, use the model to be trained to generate the corresponding image feature vectors and clinical information feature vectors of the training data set, and adjust the model parameters of the model to be trained based on the corresponding image feature vectors and clinical information feature vectors of the training data set; use the validation data set to evaluate the model to be trained with adjusted parameters using C-Index to obtain an evaluation result; if the evaluation result meets the preset convergence condition, obtain a survival analysis model.

[0109] Figure 10 The following is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. Figure 10 The illustrated electronic device 1000 (the electronic device 1000 may specifically be a computer device) includes a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004. Among them, the memory 1001, the processor 1002, and the communication interface 1003 are communicatively connected to each other through the bus 1004.

[0110] The memory 1001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1001 may store a program. When the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to execute each step of the model construction method of the embodiment of the present disclosure.

[0111] The processor 1002 may be a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, and is used to execute relevant programs to implement the functions required to be executed by each unit in the model construction device of the embodiment of the present disclosure.

[0112] The processor 1002 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the model construction method of the present disclosure can be completed by the integrated logic circuit of the hardware in the processor 1002 or the instructions in the form of software. The above-mentioned processor 1002 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. This storage medium is located in the memory 1001, and the processor 1002 reads the information in the memory 1001 and combines its hardware to complete the functions required to be executed by the units included in the model construction device of the embodiments of the present disclosure, or executes the model construction method of the method embodiments of the present disclosure.

[0113] The communication interface 1003 uses a transceiver device such as, but not limited to, a transceiver to implement the communication between the electronic device 1000 and other devices or communication networks. For example, the functional magnetic resonance imaging data signal can be obtained through the communication interface 1003.

[0114] The bus 1004 can include a path for transmitting information between various components of the electronic device 1000 (for example, the memory 1001, the processor 1002, the communication interface 1003).

[0115] It should be noted that although Figure 10 the illustrated electronic device 1000 only shows the memory, the processor, and the communication interface, in the specific implementation process, those skilled in the art should understand that the electronic device 1000 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the electronic device 1000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the electronic device 1000 may also only include the devices necessary for implementing the embodiments of the present disclosure, and do not necessarily include Figure 9 all the devices shown in

[0116] Those of ordinary skill in the art will realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this disclosure.

[0117] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0118] In several embodiments provided by this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be electrical, mechanical, or other forms.

[0119] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, the functional units in each embodiment of this disclosure can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0121] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories, random access memories, magnetic disks, or optical discs.

[0122] As described above, the above are only specific implementation manners of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present disclosure can easily think of changes or substitutions, which should all be covered by the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A risk assessment method, characterized in that, Including: Generating M sets of region-of-interest image sequences based on M sets of initial image sequences of a target user, where the modalities of the M sets of initial image sequences are different from each other, and M is a positive integer greater than or equal to 2; Using a survival analysis model, based on the M sets of region-of-interest image sequences and the clinical information data of the target user, determining the survival risk data of the target user, where the survival analysis model is constructed based on a deep learning framework and outputs risk assessment data by splicing an image feature vector and a clinical information feature vector; Among them, the using the survival analysis model to determine the survival risk data of the target user based on the M sets of region-of-interest image sequences and the clinical information data of the target user includes: Using the survival analysis model to extract features from the M sets of region-of-interest image sequences to obtain the image feature vector; Using the survival analysis model to extract features from the clinical information data to obtain the clinical information feature vector; Splicing the image feature vector and the clinical information feature vector to obtain a splicing vector that can represent the target user; Based on the splicing vector, obtaining the survival risk data of the target user.

2. The method according to claim 1, characterized in that, The survival analysis model includes a deep convolutional neural network encoder, a long short-term memory network module, a multi-layer perceptron encoder, and a Cox fully connected layer. Among them, the using the survival analysis model to extract features from the M sets of region-of-interest image sequences to obtain the image feature vector includes: Using the deep convolutional neural network encoder to extract features from the M sets of region-of-interest image sequences respectively to obtain M initial feature vectors; Using the long short-term memory network module to fuse the M initial feature vectors to obtain the image feature vector; Among them, the using the survival analysis model to extract features from the clinical information data to obtain the clinical information feature vector includes: Using the multi-layer perceptron encoder to extract features from the clinical information data to obtain the clinical information feature vector; Among them, the obtaining the survival risk data of the target user based on the splicing vector includes: Using the Cox fully connected layer to process the splicing vector to obtain the survival risk data of the target user.

3. The method according to claim 1 or 2, characterized in that, The generating M sets of region-of-interest image sequences based on M sets of initial image sequences of a target user includes: Performing region-of-interest calibration on the M sets of initial image sequences of the target user to obtain the initial rectangular frames of the regions of interest corresponding to the M sets of initial image sequences; For each initial rectangular frame of the region of interest, expanding each side of the initial rectangular frame of the region of interest away from the center of the region of interest by at least one unit pixel distance to obtain an extended rectangular frame of the region of interest corresponding to the initial rectangular frame of the region of interest; Based on the extended rectangular frames of the regions of interest corresponding to the M sets of initial image sequences, respectively segmenting the M sets of initial image sequences to obtain the M sets of region-of-interest image sequences.

4. The method according to claim 1 or 2, characterized in that, Before determining the survival risk data of the target user based on the M sets of region-of-interest image sequences and the clinical information data of the target user using the survival analysis model, the following steps are further included: Based on P sets of initial image sample sequences, generate P sets of region-of-interest image sample sequences, where the P sets of region-of-interest image sample sequences correspond to N sample users, N is a positive integer less than P, and P is a positive integer greater than or equal to 2; Based on the P sets of region-of-interest image sample sequences and the respective clinical information data of the N sample users, generate respective sample data sets for the N sample users in units of the sample users, where the region-of-interest image sample sequences included in the sample data sets are arranged in descending order of survival time; For each of the sample data sets, use the model to be trained to generate the image feature vectors and clinical information feature vectors corresponding to the sample data set, and adjust the model parameters of the model to be trained based on the image feature vectors and clinical information feature vectors corresponding to the sample data set until the model to be trained reaches a preset convergence condition to obtain the survival analysis model.

5. The method according to claim 4, wherein After generating the respective sample data sets for the N sample users, the following steps are further included: Perform data partitioning on the respective sample data sets for the N sample users to obtain a training data set and a validation data set; And the step of, for each of the sample data sets, using the model to be trained to generate the image feature vectors and clinical information feature vectors corresponding to the sample data set, and adjusting the model parameters of the model to be trained based on the image feature vectors and clinical information feature vectors corresponding to the sample data set until the model to be trained reaches a preset convergence condition to obtain the survival analysis model includes: For each of the training data sets, use the model to be trained to generate the image feature vectors and clinical information feature vectors corresponding to the training data set, and adjust the model parameters of the model to be trained based on the image feature vectors and clinical information feature vectors corresponding to the training data set; Use the validation data set to evaluate the model to be trained with adjusted parameters using C-Index to obtain an evaluation result; If the evaluation result meets the preset convergence condition, obtain the survival analysis model.

6. The method according to any one of claims 1 or 2, characterized in that The target user includes glioma patients; and / or, the region-of-interest image sequence includes the lesion region image sequence of the glioma.

7. A risk assessment device, characterized in that, Including: A generation module, configured to generate M sets of region-of-interest image sequences based on M sets of initial image sequences of the target user, where the modalities of the M sets of initial image sequences are different from each other, and M is a positive integer greater than or equal to 2; A determination module, configured to use the survival analysis model to determine the survival risk data of the target user based on the M sets of region-of-interest image sequences and the clinical information data of the target user, where the survival analysis model is constructed based on a deep learning framework, and outputs the survival risk data by splicing the image feature vectors and the clinical information feature vectors; Among them, the determination module is further configured to use the survival analysis model to extract features from the M sets of region-of-interest image sequences to obtain the image feature vectors; use the survival analysis model to extract features from the clinical information data to obtain the clinical information feature vectors; splice the image feature vectors and the clinical information feature vectors to obtain a splicing vector that can represent the target user; and obtain the survival risk data of the target user based on the splicing vector.

8. An electronic device, characterized in that, Including: A processor; A memory for storing executable instructions of the processor, wherein the processor is configured to execute the risk assessment method according to any one of claims 1 to 6 above.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is configured to execute the risk assessment method according to any one of claims 1 to 6 above.

Images