Multi-mode colorectal cancer pre-diagnosis information processing method, system, medium and equipment

Abstract

The invention relates to a multi-modal colorectal cancer pre-diagnosis information processing method and system, a medium and equipment, and the method comprises the steps: converting image information into a first modal feature, converting first clinical information into a second modal feature, converting family medical history information into a third modal feature, carrying out the feature fusion, and inputting the features into a deep learning neural network model. The finally output first pre-diagnosis report can provide disease risk level information, disease stage feature prediction information, disease cancerization probability information and disease treatment plan information of the current user, and a plurality of information sources such as personal information, image information, first clinical symptoms and family medical history information of the patient are integrated by applying a multi-modal technology; according to the method, comprehensive analysis of the colorectal disease of the current user is improved, so that a doctor can obtain more comprehensive disease assessment, more accurate risk grade division, disease stage prediction and personalized treatment scheme making are further realized, and the efficiency and the precision degree of the diagnosis and treatment process are improved.

Description

Technical Field

[0001] The present application relates to the field of colorectal cancer, and specifically to a multimodal colorectal cancer pre-diagnosis information processing method, system, medium and equipment. Background Art

[0002] Colorectal cancer is a malignant tumor that occurs in the colon and rectum. It usually originates from the abnormal proliferation and differentiation of the intestinal mucosal epithelial cells and gradually develops into a tumor. Colorectal cancer includes colon cancer and rectal cancer. Colon cancer mainly occurs in different parts of the colon, such as the ascending colon, transverse colon, descending colon and sigmoid colon, and rectal cancer occurs in the rectum segment between the anus and the sigmoid colon. Most colorectal cancers are caused by the adenoma-malignancy sequence of intestinal epithelial cells. Colorectal cancer is also associated with a genetic history. Diagnostic methods include colonoscopy, imaging examinations and clinical symptoms, combined with doctors to conduct specific disease assessment, prediction and diagnosis.

[0003] As a malignant tumor, the causes, probability of canceration and diagnosis and treatment methods of colorectal cancer will vary due to the specific circumstances of each patient. The information provided by the patient during the medical treatment process is also one-sided, which makes it impossible for the doctor to know the current patient's more comprehensive colorectal cancer causes, which in turn affects the process of formulating diagnosis and treatment methods. Summary of the invention

[0004] In view of the above problems, the present application provides a multimodal colorectal cancer pre-diagnosis information processing method, system, medium and equipment, which solves the problem that the information provided by patients in the early diagnosis stage of colorectal cancer is not comprehensive enough, which affects the formulation of subsequent diagnosis and treatment methods.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a multimodal colorectal cancer prognostic information processing method, comprising:

[0006] Collecting the user's medical information, which includes personal information, imaging information, primary clinical information, and family medical history information. Personal information includes the current user's name, permanent residence information, and user ID number. Imaging information includes multiple pathological images of the current user. Primary clinical information includes clinical symptoms, duration of the disease, and onset point of the current medical stage. Family medical history information includes secondary clinical information of family members who are related to the current user under the current disease.

[0007] Inputting the image information into the UNET algorithm model to obtain a first modality feature, wherein the first modality feature includes at least one of the lesion area, tumor size, tumor distribution information, and the proportion of cancerous cells in the tumor;

[0008] Extract keywords from the first clinical information to obtain multiple semantic keywords, input the multiple semantic keywords into a semantic dictionary, and construct a semantic feature vector of the current first clinical information, which is recorded as a second modal feature;

[0009] The family medical history information is input into the random forest algorithm model to obtain the third modality feature, which includes the specific disease characteristics of the current user under the current disease;

[0010] Inputting the first modal feature, the second modal feature, and the third modal feature into a feature fusion algorithm model for feature fusion to obtain a first fusion feature;

[0011] Inputting the first fusion feature into the trained neural network model to obtain a first pre-diagnosis result, the first pre-diagnosis result including the disease risk level information of the current user, the stage feature prediction information of the disease, the canceration probability information of the disease, and the treatment plan information of the disease;

[0012] The first preliminary diagnosis result is converted into a first preliminary diagnosis report and displayed.

[0013] Furthermore, the image information is input into the UNET algorithm model, and the first modal features obtained include:

[0014] Annotating the image information using an annotator to obtain a plurality of first image information;

[0015] Preprocessing the first image information one by one to obtain a plurality of second image information, wherein the preprocessing includes at least one of image enhancement, image cutting, and image normalization processing;

[0016] Arranging the plurality of second image information in an acquisition order, and acquiring slice position information between two adjacent second image information, wherein the slice position information includes slice thickness and slice spacing;

[0017] Stacking a plurality of second image information arranged in acquisition sequence along a preset direction according to slice position information to generate a first three-dimensional image matrix;

[0018] Input the first three-dimensional image matrix into the trained UNET algorithm model to obtain a second three-dimensional image matrix;

[0019] The second three-dimensional image matrix is ​​mapped to the plurality of second image information to obtain the first modal feature.

[0020] Furthermore, the first three-dimensional image matrix includes a plurality of first voxel vectors, the second three-dimensional image matrix includes a plurality of second voxel vectors, and the first three-dimensional image matrix is ​​input into the trained UNET algorithm model to obtain the second three-dimensional image matrix including:

[0021] Encoding the first three-dimensional image matrix layer by layer to a first preset threshold value to obtain first local feature information, where the first local feature information includes a local feature extracted from each first voxel vector in the layer-by-layer encoding process;

[0022] Decoding the first local feature information layer by layer to obtain second local feature information, where the second local feature information includes a local feature of each second voxel vector, and the local feature of the second voxel vector is a local feature of each first voxel vector fused layer by layer during the decoding process;

[0023] Convolving the second local feature information to obtain a first primary three-dimensional image matrix;

[0024] Performing threshold processing on the first primary three-dimensional image matrix to obtain a first segmentation mask;

[0025] Determining whether the first segmentation mask satisfies a second preset threshold;

[0026] If yes, outputting the first primary three-dimensional image matrix as the second three-dimensional image matrix;

[0027] If not, the first segmentation mask is modified to obtain a second segmentation mask;

[0028] Calculate the loss function of the UNET algorithm model in the layer-by-layer decoding process according to the second segmentation mask;

[0029] Perform gradient feedback according to the loss function to update the model parameters used by the UNET algorithm model in the layer-by-layer decoding process;

[0030] The first local feature information is decoded layer by layer in the updated UNET algorithm model to obtain third local feature information, where the third local feature information includes the local features of each first voxel vector fused in the decoding process;

[0031] Convolving the third local feature information to obtain a second primary three-dimensional image matrix;

[0032] performing threshold processing on the second primary three-dimensional image matrix to obtain a third segmentation mask;

[0033] Determining whether the third segmentation mask meets a second preset threshold;

[0034] If so, the second primary three-dimensional image matrix is ​​output as the second three-dimensional image matrix.

[0035] Furthermore, keywords are extracted from the first clinical information to obtain multiple semantic keywords including:

[0036] Performing data cleaning on the first clinical information to obtain a first information text;

[0037] Segmenting the first information text into sentences to obtain a plurality of to-be-processed sentence segments, each of which has a plurality of words;

[0038] Input the current sentence segments to be processed into the BERT variant model one by one to obtain the first embedding vector information of each sentence segment to be processed, wherein the first embedding vector information includes the first word embedding vector of each vocabulary in the same sentence segment to be processed;

[0039] Input the first embedding vector information into the TF-IDF model to obtain the weight value corresponding to the first word embedding vector of the same sentence to be processed;

[0040] Arrange multiple first word embedding vectors in the first information text from left to right according to the size of the weight value, select the words corresponding to a preset number of first word embedding vectors from the left as semantic keywords, and record the first word embedding vectors corresponding to the semantic keywords as second word embedding vectors.

[0041] Furthermore, multiple semantic keywords are input into the semantic dictionary to construct a semantic feature vector of the current first clinical information, which is recorded as the second modal feature including:

[0042] Obtaining a semantic dictionary, the semantic dictionary is obtained by training based on sample clinical information, and the semantic dictionary includes multiple clinical terms;

[0043] Match semantic keywords with clinical terms one by one;

[0044] Fill the second word embedding vector associated with the successfully matched semantic keyword into the semantic vector corresponding to the clinical term and record it as the first semantic vector;

[0045] Set the semantic vectors corresponding to the unmatched clinical terms to 0;

[0046] Perform feature concatenation on multiple first semantic vectors to obtain a semantic feature vector represented by a matrix, which is recorded as a second modal feature;

[0047] Matching semantic keywords with clinical terms one by one includes:

[0048] Get the second word embedding vector of each semantic keyword;

[0049] Obtain the semantic vector of each clinical term;

[0050] Calculate the cosine similarity between the second word embedding vector and the current semantic vector one by one;

[0051] Determine whether the cosine similarity exceeds the semantic similarity threshold;

[0052] If yes, the cosine similarity is recorded as the first cosine similarity;

[0053] Determine whether the number of the first cosine similarity is greater than 1;

[0054] If the number of first cosine similarities is greater than 1, multiple first cosine similarities are arranged in descending order according to the similarity values, and the semantic keyword corresponding to the second word embedding vector with the highest first cosine similarity is selected to match with the current clinical term;

[0055] If the number of the first cosine similarity is 1, the semantic keyword associated with the first cosine similarity is matched with the current clinical term;

[0056] If not, it means that the current clinical term matching is unsuccessful.

[0057] Furthermore, the family medical history information is input into the random forest algorithm model, and the third modal features obtained include:

[0058] The family tree information is constructed based on the family members in the family medical history information as nodes and the kinship relationship as the hierarchical basis. The family members with the second clinical information are assigned a value of 1, and the family members without the second clinical information are assigned a value of 0;

[0059] The family tree information is used as input features, and the semantic keywords are used as variable labels to input into the random forest algorithm model, and the influencing features of each family member under the current disease are obtained, which are recorded as the first influencing features;

[0060] The importance of the first influencing feature is calculated one by one and recorded as the first importance, which is expressed by formula (1). Formula (1) is as follows:

[0061] ;

[0062] In formula (1), For the The first importance of the first influencing feature, For the tree Use in When features are split Impurity reduction, is the total number of trees in the random forest algorithm model, is the tree in the random forest algorithm model;

[0063] The multiple first importances are normalized to obtain multiple second importances, which are expressed by formula (2). Formula (2) is as follows:

[0064] ;

[0065] In formula (2), For the The second importance of the first influencing feature, N is the number of nodes of the family;

[0066] The plurality of second importances are represented by numerical sequences to obtain the third modal features.

[0067] Furthermore, the first modal feature, the second modal feature and the third modal feature are input into the feature fusion algorithm model for feature fusion, and the first fusion feature obtained includes:

[0068] The first modal feature, the second modal feature and the third modal feature are concatenated to obtain a first concatenated feature, and the first concatenated feature is expressed by formula (3). Formula (3) is as follows:

[0069] ;

[0070] In formula (3), is the first modal feature, is the second modal feature, is the third modal feature, is the first splicing feature;

[0071] The first concatenated feature is input into the attention mechanism algorithm model to obtain the query vector, key vector and value vector, which are expressed by formula (4). Formula (4) is as follows:

[0072]

[0073] In formula (4), is the query vector, is the key vector, is a value vector, is the weight matrix of the query vector, is the weight matrix of the key vector, is the weight matrix of the value vector;

[0074] The attention weight is calculated according to the query vector, key vector and value vector to obtain the attention weight matrix, which is expressed by formula (5). Formula (5) is as follows:

[0075] ;

[0076] In formula (5), for The transposed matrix of is the dimension of the key vector, is the attention of the first modal feature to itself, is the attention of the first modal feature to the second modal feature, is the attention degree of the first modal feature to the third modal feature, is the attention of the second modal feature to the first modal feature, is the attention of the second modal feature to itself, is the attention degree of the second modal feature to the third modal feature, is the attention of the third modal feature to the first modal feature, is the attention of the third modal feature to the second modal feature, is the third modal feature’s attention to itself;

[0077] The value vector is weighted and summed according to the attention weight matrix to obtain the attention fusion feature. The attention fusion feature is expressed by formula (6), which is as follows:

[0078] ;

[0079] In formula (6), is the attention fusion feature, is the weighted sum function;

[0080] The attention fusion feature is fused with the first concatenation feature to obtain the first fusion feature. The first fusion feature is expressed by formula (7). Formula (7) is as follows:

[0081] ;

[0082] In formula (7), is the first fusion feature.

[0083] In a second aspect, the present invention further provides a multimodal colorectal cancer prognostic information processing system, which is applicable to the prognostic information processing method described in the first aspect, and the prognostic information processing system includes an information acquisition module, a first feature conversion module, a second feature conversion module, a third feature conversion module, a feature fusion module, and a prognostic report generation module;

[0084] The information collection module is used to collect the user's medical information, which includes personal information, imaging information, first clinical information and family medical history information. The personal information includes the current user's name, permanent residence information, and user ID number. The imaging information includes multiple pathological images of the current user. The first clinical information includes clinical symptoms, duration of the disease, and onset node of the current medical stage. The family medical history information includes the second clinical information of the family members who are related to the current user under the current disease; the first feature conversion module is used to input the imaging information into the UNET algorithm model to obtain the first modal feature, and the first modal feature includes at least one of the lesion area, tumor size, tumor distribution information, and the proportion of tumor cancerous cells; the second feature conversion module is used to extract keywords from the first clinical information to obtain multiple semantic keywords, and convert multiple semantic keywords into Keywords are input into the semantic dictionary to construct the semantic feature vector of the current first clinical information, which is recorded as the second modal feature; the third feature conversion module is used to input the family medical history information into the random forest algorithm model to obtain the third modal feature, which includes the specific onset characteristics of the current user under the current disease; the feature fusion module is used to input the first modal feature, the second modal feature and the third modal feature into the feature fusion algorithm model for feature fusion to obtain the first fusion feature; the pre-diagnosis report generation module is used to input the first fusion feature into the trained neural network model to obtain the first pre-diagnosis result, which includes the current user's disease risk level information, the stage feature prediction information of the disease, the probability of canceration of the disease and the treatment plan information of the disease, and convert the first pre-diagnosis result into a first pre-diagnosis report and display it.

[0085] In a third aspect, the present invention further provides a computer-readable storage medium storing computer program instructions, which implement the method described in the first aspect when executed by a processor.

[0086] In a fourth aspect, the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0087] Different from the existing technology, in the above technical scheme, by converting the imaging information into the first modal feature, the first clinical information into the second modal feature, and the family medical history information into the third modal feature, the first modal feature, the second modal feature and the third modal feature are then feature fused and input into the deep learning neural network model. The first preliminary diagnosis report finally output can provide the current user's disease risk level information, disease stage feature prediction information, disease cancer probability information and disease treatment plan information. The entire technical scheme applies the principle of multimodal technology to integrate the patient's personal information, imaging information, first clinical symptoms and family medical history information and other information sources, so as to improve the comprehensive analysis of the current user's colorectal disease, enable doctors to obtain a more comprehensive disease assessment, and then achieve more accurate risk level classification, disease stage prediction and personalized treatment plan formulation, thereby improving the efficiency and accuracy of the diagnosis and treatment process.

[0088] The above-mentioned records related to the invention content are only an overview of the technical solution of the present application. In order to enable ordinary technicians in the field to more clearly understand the technical solution of the present application, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purpose and other purposes, features and advantages of the present application easier to understand, the following is an explanation in combination with the specific implementation mode and drawings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present invention and other related contents, and shall not be considered as limiting the present application.

[0090] In the drawings of the specification:

[0091] Figure 1 It is a schematic diagram of steps S101 to S107 of the multimodal colorectal cancer pre-diagnosis information processing method described in the specific implementation method;

[0092] Figure 2 It is a schematic diagram of a multimodal colorectal cancer prognosis information processing system according to a specific implementation method;

[0093] Figure 3 It is a schematic diagram of steps S201 to S206 of the multimodal colorectal cancer pre-diagnosis information processing method described in the specific implementation method;

[0094] Figure 4 It is a schematic diagram of steps S301 to S305 of the multimodal colorectal cancer pre-diagnosis information processing method described in the specific implementation method;

[0095] Figure 5This is a schematic diagram of steps S401 to S405 of the multimodal colorectal cancer prognosis information processing method described in the specific implementation method.

[0096] The reference numerals in the above drawings are described as follows:

[0097] 1. Multimodal colorectal cancer prognostic information processing system;

[0098] 11. Information collection module;

[0099] 12. The first feature conversion module;

[0100] 13. Second feature conversion module;

[0101] 14. The third feature conversion module;

[0102] 15. Feature fusion module;

[0103] 16. Pre-diagnosis report generation module. DETAILED DESCRIPTION

[0104] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0105] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0106] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0107] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.

[0108] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0109] Without further limitations, in this application, the words "include", "comprises", "has" or other similar open-ended expressions used in sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0110] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.

[0111] See also Figure 1 In a first aspect, this embodiment provides a multimodal colorectal cancer prognostic information processing method, comprising:

[0112] S101, collecting the user's medical information, the medical information including personal information, imaging information, first clinical information and family medical history information, the personal information including the current user's name, permanent residence information, and user ID number, the imaging information including multiple pathological images of the current user, the first clinical information including clinical symptoms, duration of the disease course and onset point of the current medical stage, and the family medical history information including second clinical information of the family members who are related to the current user under the current disease;

[0113] S102, inputting the image information into the UNET algorithm model to obtain a first modal feature, where the first modal feature includes at least one of the lesion area, tumor size, tumor distribution information, and the proportion of cancerous cells in the tumor;

[0114] S103, extracting keywords from the first clinical information to obtain multiple semantic keywords, inputting the multiple semantic keywords into a semantic dictionary, constructing a semantic feature vector of the current first clinical information, and recording it as a second modal feature;

[0115] S104, inputting the family medical history information into the random forest algorithm model to obtain a third modal feature, where the third modal feature includes specific disease characteristics of the current user under the current disease;

[0116] S105, inputting the first modal feature, the second modal feature, and the third modal feature into a feature fusion algorithm model for feature fusion to obtain a first fusion feature;

[0117] S106, inputting the first fusion feature into the trained neural network model to obtain a first pre-diagnosis result, the first pre-diagnosis result including the disease risk level information of the current user, the stage feature prediction information of the disease, the canceration probability information of the disease, and the treatment plan information of the disease;

[0118] S107: Convert the first pre-diagnosis result into a first pre-diagnosis report and display it.

[0119] In step S101, the medical information includes personal information, imaging information, first clinical information and family medical history information, wherein personal information can be understood as the personal identity information of the current patient, including name, permanent residence, ID number, and may also include contact number, eating habits, etc. Personal information mainly plays an identification role in this step. The historical medical records of the current user can be extracted in the database through personal information, and the historical medical records of relatives associated with the current user can also be extracted through personal information to facilitate subsequent data analysis. The imaging information includes multiple pathological images taken by the current user during the medical consultation. Specifically, the imaging information can be a CT image or an MRI image, wherein a CT image is preferably used. The imaging information is multiple image information collected by the user in the disease area during the current medical consultation, including the tumor area, and may also include a local tumor area. Preferably, the imaging information is a plurality of continuous CT slice images, so as to facilitate a more comprehensive three-dimensional modeling and analysis of the user's disease area in the future. The first clinical information includes the clinical symptoms, duration of the disease course and the onset node of the current treatment stage. The clinical symptoms can be recorded and generated by the doctor after oral description, or by the doctor performing the corresponding simple pressing and observation on the spot. The duration of the disease course and the onset node mainly rely on the direct oral description of the user. The doctor can make a textual description of the clinical symptoms according to the on-site examination to obtain the first clinical information. The family medical history information includes the second clinical information of the family members who are related to the current user under the current disease. It should be noted that the relationship with the current user can be understood as a relationship with the current user that is associated with genetic information, and the second clinical information is clinical information of the same category as the current disease, that is, the user's current first clinical information is the information content in the field of colorectal disease, and the second clinical information is the information content in the field of colorectal cancer disease of the family members who are related to the current user. This embodiment introduces family medical history information, which is more in line with the characteristics of colorectal cancer being affected by certain genetic factors, so that the subsequent feature fusion is more comprehensive and closer to the actual situation of the current user.

[0120] In step S102, the UNET algorithm model is used to extract and fuse features of multiple pathological images in the image information to obtain the first modal feature. The specific steps can be referred to in the following description. The UNET algorithm model can perform detailed segmentation on each pathological image, thereby removing the features of the tumor area or suspected tumor area from the pathological image. The first modal feature obtained can reflect the association between the tumor area or suspected tumor area and the organ position and organ area of ​​the current user in the form of a vector, which is convenient for subsequent feature fusion. Specifically, the first modal feature includes at least one of the lesion area, tumor size, tumor distribution information, and the proportion of tumor cancer cells; the lesion area is also the area where the user suffers from repeated attacks of the disease. The tumor size and tumor distribution information can be determined according to the actual situation. For example, some users have evolved from intestinal polyps to tumors, then the first modal feature includes the tumor size and tumor distribution information. The tumor distribution information is the distribution position of the tumor in the colon and rectum. The proportion of tumor cancer cells can be understood as a prediction of the evolution stage from benign tumors to malignant tumors, which is also determined according to the actual situation. If the user is currently in the stage of ordinary enteritis, there may be a situation where the proportion of tumor cancer cells is 0. This embodiment does not limit this.

[0121] In step S103, the first clinical information is subjected to keyword extraction to obtain semantic keywords, where the semantic keywords are keywords containing semantic information associated with the current disease. In addition, the present embodiment also introduces a semantic dictionary as a basis for constructing a semantic feature vector, wherein the semantic dictionary may be generated in advance through big data information. The specific steps may be: obtaining sample clinical information of the disease field to which colorectal cancer belongs, and performing similarity classification on multiple sample keywords in the sample clinical information to obtain multiple clinical terms, each clinical term including multiple synonyms, hyponyms, and hyponyms, and different clinical terms have corresponding associations with the current disease field, and the associations may be obtained by converting the occurrence probability of the clinical terms in the current disease field. For example, the user's clinical symptoms include "toothache", but in the semantic dictionary of colorectal cancer, the clinical terms corresponding to "toothache" have a low association. If 1-10 is used as the unit of association, the association of "toothache" is 0.5, because this symptom will only occur when the cancer cells of colorectal cancer metastasize to the oral cavity in the later stage, and when this symptom occurs, there must be other clinical terms with higher associations, such as "intestinal bleeding". By constructing a semantic dictionary using sample clinical information from big data, the resulting semantic dictionary can cover all clinical terms in the current disease field, and each clinical term has a corresponding degree of relevance to facilitate subsequent analysis.

[0122] Constructing the semantic feature vector of the current first clinical information can be understood as follows: matching semantic keywords in the semantic dictionary in sequence, and when the semantic keyword is matched successfully, replacing the semantic keyword with a clinical entry, and so on, to form a semantic feature vector corresponding to the first clinical information of the current user, which is recorded as the second modal feature.

[0123] In step S104, a random forest algorithm model is used to perform probability analysis and prediction on the cancer of the current user based on the family history information. Specifically, the random forest algorithm model improves the accuracy and stability of the prediction by constructing multiple decision trees and combining their results. Further steps can be understood by referring to the following description. Random forest can effectively handle complex feature relationships, provide interpretable results, and help identify important genetic factors. Through the random forest algorithm model, the influence of genetic factors between different kinship relationships can be clarified. For example, when the family members within the three direct generations of the user are all diagnosed with the disease feature A associated with colorectal cancer, the probability of the disease feature A occurring in the current user is relatively high, and then a disease probability range is formed for whether the user suffers from colorectal cancer. When some of the family members outside the three direct generations of the user are diagnosed with the disease feature B associated with colorectal cancer, the probability of the disease feature B occurring in the current user is relatively low, and the probability of colorectal cancer of the current user has certain fluctuations. The third modal feature obtained by the random forest algorithm model includes the specific disease characteristics of the current user under the current disease. It can be understood that the third modal feature is a vector, that is, the disease characteristics include disease symptoms and disease probability.

[0124] In step S105, after obtaining the first modal feature, the second modal feature and the third modal feature, a feature fusion algorithm model is used to perform feature fusion to obtain a first fused feature. Specifically, the feature fusion algorithm model can be obtained based on a nonlinear activation function in a feature fusion network, can also be obtained based on an attention mechanism fusion, can also be obtained based on a domain adaptation algorithm, and the feature fusion algorithm shown in this embodiment can also be understood as a simple feature fusion logic such as direct concatenation and direct dot multiplication of multiple features, and this embodiment does not limit this. Preferably, this embodiment selects a model corresponding to the attention mechanism as a feature fusion algorithm model, which can achieve the optimal fusion effect of fusing the first modal feature, the second modal feature and the third modal feature in the intermediate stage.

[0125] In step S106, the first fusion feature is input into the trained neural network model to obtain the first pre-diagnosis result. It should be noted that the neural network model can use a convolutional neural network model, and the training process of the neural network model can be specifically: constructing a basic neural network model, obtaining multiple sample fusion features in the sample database to train the basic neural network model, and comparing the accuracy of the output result with the sample pre-diagnosis result. When the accuracy of the output result and the sample pre-diagnosis result exceeds the preset accuracy threshold, it indicates that the current neural network model training is completed. Optionally, the accuracy threshold can be set to 90% to improve the pre-diagnosis accuracy of the model.

[0126] In this embodiment, the first pre-diagnosis result obtained includes the user's disease risk level information, disease stage feature prediction information, disease cancer probability information and disease treatment plan information; wherein, the disease risk level information can be understood as the treatment level classification of the disease field where colorectal cancer is located, for example, in the stage without cancer, it can be understood as the cancer risk level, and in the stage with cancer, it can be understood as the cancer diffusion level, such as early, middle and late stages. The treatment level classification facilitates doctors to select more appropriate treatment plans for the user's level. For example, when there is no cancer but the cancer risk level is intermediate, doctors are more inclined to adjust living habits such as eating habits and work and rest rules to achieve the preventive effect on users. For example, in the early stage of cancer, doctors are more inclined to use chemotherapy, resection and other methods to eliminate cancer cells. The disease stage feature prediction information includes the evolution of the disease in the next stage generated based on the existing medical information, the specific characteristics of the disease that may appear, etc. The stage feature prediction information can provide references for doctors and users, so that doctors can make corresponding solutions in advance.

[0127] In this embodiment, the cancer probability information of the disease is more suitable for users who have colorectal diseases but have not yet developed cancer. When a certain area in the colorectum forms a lesion due to repeated disease, the cells in the area have a certain probability of cancer. Through the cancer probability information of the disease, doctors can have a more accurate understanding of the user's current situation, and can also provide warnings to users, mobilizing the user's enthusiasm and cooperation for treatment. In this embodiment, there can be multiple treatment plans or one treatment plan, which can be analyzed according to actual conditions.

[0128] In step S107, the first pre-diagnosis result is converted into a first pre-diagnosis report, which is displayed in the form of a report to the user and the doctor. When the user is still in the waiting stage, the first pre-diagnosis report can help the user understand his or her own condition in advance and improve the user's own enthusiasm and cooperation for treatment during the consultation. The first pre-diagnosis report also helps the doctor to efficiently understand the user's disease status and current clinical condition, thereby improving the efficiency of the consultation.

[0129] This embodiment converts the imaging information into a first modal feature, the first clinical information into a second modal feature, and the family medical history information into a third modal feature, and then fuses the first modal feature, the second modal feature and the third modal feature and inputs them into a deep learning neural network model. The first preliminary diagnosis report finally outputted can provide the current user's disease risk level information, disease stage feature prediction information, disease cancer probability information and disease treatment plan information. The entire technical solution uses the principle of multimodal technology to integrate multiple information sources such as the patient's personal information, imaging information, first clinical symptoms and family medical history information, so as to improve the comprehensive analysis of the current user's colorectal disease, enable doctors to obtain a more comprehensive disease assessment, and further achieve more accurate risk level classification, disease stage prediction and personalized treatment plan formulation, thereby improving the efficiency and accuracy of the diagnosis and treatment process.

[0130] For the corresponding Figure 2 In the second aspect, the present embodiment further provides a multimodal colorectal cancer prognostic information processing system 1, which is applicable to the prognostic information processing method described in the first aspect, and the prognostic information processing system includes an information acquisition module 11, a first feature conversion module 12, a second feature conversion module 13, a third feature conversion module 14, a feature fusion module 15 and a prognostic report generation module 16;

[0131] The information collection module 11 is used to collect the user's medical information, which includes personal information, imaging information, first clinical information and family medical history information. The personal information includes the current user's name, permanent residence information, and user ID number. The imaging information includes multiple pathological images of the current user. The first clinical information includes clinical symptoms, duration of the disease course, and onset node of the current medical stage. The family medical history information includes the second clinical information of the family members who are related to the current user under the current disease; the first feature conversion module 12 is used to input the imaging information into the UNET algorithm model to obtain the first modal feature, and the first modal feature includes at least one of the lesion area, tumor size, tumor distribution information, and the proportion of tumor cancerous cells; the second feature conversion module 13 is used to extract keywords from the first clinical information to obtain multiple semantic keywords, and convert multiple semantic keywords into Keywords are input into the semantic dictionary to construct the semantic feature vector of the current first clinical information, which is recorded as the second modal feature; the third feature conversion module 14 is used to input the family medical history information into the random forest algorithm model to obtain the third modal feature, which includes the specific onset characteristics of the current user under the current disease; the feature fusion module 15 is used to input the first modal feature, the second modal feature and the third modal feature into the feature fusion algorithm model for feature fusion to obtain the first fusion feature; the pre-diagnosis report generation module 16 is used to input the first fusion feature into the trained neural network model to obtain the first pre-diagnosis result, which includes the current user's disease risk level information, the stage feature prediction information of the disease, the probability of canceration of the disease and the treatment plan information of the disease, and convert the first pre-diagnosis result into a first pre-diagnosis report and display it.

[0132] The execution steps in the above embodiment can be understood by referring to the method described in the first aspect above, and no unnecessary details are given here.

[0133] This embodiment applies the principle of multimodal technology to integrate multiple information sources such as the patient's personal information, imaging information, first clinical symptoms and family medical history information, so as to improve the comprehensive analysis of the current user's colorectal disease, enable doctors to obtain a more comprehensive disease assessment, and further achieve more accurate risk level classification, disease stage prediction and personalized treatment plan formulation, thereby improving the efficiency and accuracy of the diagnosis and treatment process.

[0134] See also Figure 3 In some embodiments, the image information is input into the UNET algorithm model to obtain the first modality feature including:

[0135] S201, annotating the image information using an annotator to obtain a plurality of first image information;

[0136] S202, preprocessing the first image information one by one to obtain a plurality of second image information, wherein the preprocessing includes at least one of image enhancement, image cutting, and image normalization processing;

[0137] S203, arranging the plurality of second image information in an acquisition order, and obtaining slice position information between two adjacent second image information, where the slice position information includes slice thickness and slice spacing;

[0138] S204, stacking the plurality of second image information arranged in acquisition order along a preset direction according to the slice position information to generate a first three-dimensional image matrix;

[0139] S205, inputting the first three-dimensional image matrix into the trained UNET algorithm model to obtain a second three-dimensional image matrix;

[0140] S206: Map the second three-dimensional image matrix to the plurality of second image information to obtain a first modal feature.

[0141] In step S201, the image information is annotated using a labeler. Optionally, the labeler can be a Labelbox labeler, an ITK-SNAP labeler, a 3D Slicer labeler, etc. The labeler can realize autonomous labeling of the image information through sample training. The doctor can also manually label the image information through the labeler. Preferably, after the image information is autonomously annotated by the labeler, the doctor manually adjusts the labeling result, which can ensure the accuracy of the labeling while saving the doctor's labeling workload. Labeling the image information using a labeler can be specifically understood as labeling and boxing the tumor area, suspected tumor area, lesion area, and lesion area contained in the image information, and recording the annotated pathological image as the first image information.

[0142] In step S202, the first image information is preprocessed one by one, and image enhancement may include adjusting the brightness, contrast, color, etc. of the original image to improve the image quality and enhance the visibility of the lesion area. Specifically, filtering algorithms such as Gaussian filtering and median filtering can be used to remove noise and artifacts in the image to further optimize the image details; sharpening algorithms are applied to highlight the edge information of the image, which is conducive to subsequent image segmentation and feature extraction. Image cutting is specifically, according to anatomical knowledge, the region of interest (ROI) of the image is automatically or manually located and cut, and irrelevant background areas are removed to ensure that the cut image only contains the colorectal part, which is conducive to subsequent feature analysis. Image normalization is to standardize the scale of the cut two-dimensional image and adjust it to the same size. For example, it can be grayscale value normalization: the image pixel value is mapped to a standard numerical range, such as [0, 1] or [-1, 1], so that the image scale and brightness differences generated under different scanning devices and conditions can be eliminated. This step can make the subsequent UNET segmentation process more accurate and reduce the impact of image noise.

[0143] In step S203, the plurality of second image information are arranged in the order of acquisition of the pathological image, and there is a certain slice thickness and slice spacing between two adjacent second image information, so as to form a continuous CT image group.

[0144] Further, in step S204, the plurality of second image information arranged in the acquisition sequence are stacked along a preset direction according to the slice position information. It should be noted that the preset direction can be understood as the scanning direction of the user's second image information, and after stacking, they can be stitched together to form a first three-dimensional image matrix. Specifically, the missing values ​​in the first three-dimensional image matrix can be supplemented between two adjacent second image information by linear interpolation to achieve a smooth transition of the three-dimensional volume data between the plurality of second image information.

[0145] In step S205, the first three-dimensional image matrix is ​​input into the trained UNET algorithm model to obtain a second three-dimensional image matrix; the UNET algorithm model is a deep learning algorithm widely used in medical image segmentation, which is composed of a convolutional neural network and an upsampling layer, and can effectively extract and fuse the multi-scale features of the image. Further, the UNET model consists of an encoder (contraction path) and a decoder (expansion path). The encoder uses a series of convolution and pooling operations to extract multi-scale feature representations, while the decoder fuses these features through upsampling and jump connections to achieve fine pixel-level segmentation. Before using the UNET algorithm model, the UNET algorithm model needs to be trained. A large number of sample three-dimensional image matrices can be used for end-to-end supervised learning training. The training goal is to minimize the segmentation loss function of the model. For example, Dice loss or cross entropy loss can be used to improve the accuracy of model segmentation recognition.

[0146] The first three-dimensional image matrix is ​​input into the trained UNET algorithm model to obtain the second three-dimensional image matrix. It should be noted that the first three-dimensional image matrix contains the numerical values ​​corresponding to multiple pixel points, which represent the grayscale information of the pixel points. Similarly, after the first three-dimensional image matrix is ​​converted into the second three-dimensional image matrix, the numerical values ​​contained in the second three-dimensional image matrix are the confidence that each pixel point belongs to the area where the colorectal disease is located. In addition, since the first three-dimensional image matrix and the second three-dimensional image matrix include the three-dimensional volume data of the entire colorectal area, the pixel points corresponding to the numerical values ​​contained in the second three-dimensional image matrix are pixel points in the three-dimensional sense.

[0147] Further, in step S206, the second three-dimensional image matrix is ​​mapped to the plurality of second image information, that is, the three-dimensional volume data is remapped to the second image information to realize the numerical mapping relationship from three-dimensional to two-dimensional, and this method can obtain more refined boundary features of the colorectal disease area and the colorectal area. After mapping the second three-dimensional image matrix to the plurality of second image information, the characteristic information containing the colorectal disease area, such as tumor volume, tumor location, and tumor shape, is combined to form a characteristic vector. Preferably, the features and mapping relationships of the same attributes are combined to form a vector of one dimension, and the features and mapping relationships of multiple attributes are combined to form a multidimensional vector, that is, the first modal feature is a multidimensional vector representation of the colorectal disease area.

[0148] This embodiment uses the UNET algorithm to accurately segment the original image information, and can extract rich feature information of the colorectal area. The two-dimensional CT / MRI images are stacked into a three-dimensional image matrix, making full use of the three-dimensional spatial structure information, and then the three-dimensional image matrix is ​​input into the trained UNET model to obtain a three-dimensional output matrix representing the probability of the colorectal region. After that, this three-dimensional output matrix is ​​mapped with the original two-dimensional CT / MRI image to obtain a series of vector representations containing features such as tumor size, location, and shape, i.e., the first modal feature. This three-dimensional to two-dimensional feature mapping can make full use of the rich three-dimensional features extracted by UNET while maintaining compatibility with the two-dimensional medical imaging information commonly used in clinical applications. More accurate description of the lesion information in the colorectal area provides high-quality input data for subsequent multimodal feature fusion and disease prediction.

[0149] Furthermore, in some embodiments, the first three-dimensional image matrix includes a plurality of first voxel vectors, the second three-dimensional image matrix includes a plurality of second voxel vectors, and the first three-dimensional image matrix is ​​input into the trained UNET algorithm model to obtain the second three-dimensional image matrix including:

[0150] Encoding the first three-dimensional image matrix layer by layer to a first preset threshold value to obtain first local feature information, where the first local feature information includes a local feature extracted from each first voxel vector in the layer-by-layer encoding process;

[0151] Decoding the first local feature information layer by layer to obtain second local feature information, where the second local feature information includes a local feature of each second voxel vector, and the local feature of the second voxel vector is a local feature of each first voxel vector fused layer by layer during the decoding process;

[0152] Convolving the second local feature information to obtain a first primary three-dimensional image matrix;

[0153] Performing threshold processing on the first primary three-dimensional image matrix to obtain a first segmentation mask;

[0154] Determining whether the first segmentation mask satisfies a second preset threshold;

[0155] If yes, outputting the first primary three-dimensional image matrix as the second three-dimensional image matrix;

[0156] If not, the first segmentation mask is modified to obtain a second segmentation mask;

[0157] Calculate the loss function of the UNET algorithm model in the layer-by-layer decoding process according to the second segmentation mask;

[0158] Perform gradient feedback according to the loss function to update the model parameters used by the UNET algorithm model in the layer-by-layer decoding process;

[0159] The first local feature information is decoded layer by layer in the updated UNET algorithm model to obtain third local feature information, where the third local feature information includes the local features of each first voxel vector fused in the decoding process;

[0160] Convolving the third local feature information to obtain a second primary three-dimensional image matrix;

[0161] performing threshold processing on the second primary three-dimensional image matrix to obtain a third segmentation mask;

[0162] Determining whether the third segmentation mask meets a second preset threshold;

[0163] If so, the second primary three-dimensional image matrix is ​​output as the second three-dimensional image matrix.

[0164] In this embodiment, the first three-dimensional image matrix includes a plurality of first voxel vectors, and the first voxel vector refers to the feature representation of each voxel in the first three-dimensional image matrix input to the UNET algorithm model, and the first voxel vector generally includes the gray value or other feature information of the corresponding voxel in the original image. The second three-dimensional image matrix includes a plurality of second voxel vectors, and it can be understood that after being processed by the UNET algorithm model, the second voxel vector is the feature representation of the model output, and generally includes the feature information obtained after the encoding and decoding process, and the second voxel vector reflects the UNET algorithm model's understanding of the input data and the extracted features, including local spatial relationships, etc.

[0165] In this embodiment, the first three-dimensional image matrix is ​​encoded layer by layer until a first preset threshold value, where the first preset threshold value can be understood as the number of layers of layer-by-layer encoding. The UNET algorithm model will eventually reach a bottleneck layer after layer-by-layer encoding, which is the limit of layer-by-layer encoding. This bottleneck layer is set to the first preset threshold value. During the layer-by-layer encoding process, the local features of each layer will be segmented and retained to form the first local feature information, that is, the first local feature information includes the local features extracted by each first voxel vector during the layer-by-layer encoding process.

[0166] The first local feature information is decoded layer by layer again. Layer by layer decoding is the reverse step of layer by layer encoding. If layer by layer encoding is understood as downsampling, then layer by layer decoding can be understood as upsampling. During the layer by layer decoding process, the local features of each layer will be fused to finally form the second local feature information. That is, the second local feature information includes the local features of the first voxel vector fused layer by layer during the decoding process. For the convenience of expression, the fused first voxel vector is recorded as the second voxel vector.

[0167] Finally, the second local feature information is convolved to form a first primary three-dimensional image matrix. The first primary three-dimensional image matrix is ​​thresholded to generate a segmentation mask. Optionally, the segmentation mask is recorded as a first segmentation mask, and the first segmentation mask is a binary image. Further, the first segmentation mask is identified, and if the first segmentation mask meets the second preset threshold, the first primary three-dimensional image matrix is ​​output as the second three-dimensional image matrix. In this process, the second preset threshold can be understood as a criterion for judging the accuracy of model segmentation. When the first segmentation mask meets the second preset threshold, it means that the model segmentation accuracy of the current first primary three-dimensional image matrix meets the conditions, that is, the error of the output first primary three-dimensional image matrix is ​​small, and it can be directly used as the second three-dimensional image matrix for subsequent mapping steps.

[0168] If the first segmentation mask does not meet the second preset threshold, the first segmentation mask needs to be corrected to obtain the second segmentation mask, and then the loss function in the UNET algorithm model is calculated based on the second segmentation mask. The loss function affects the model parameters used by the UNET algorithm model in the layer-by-layer decoding process. This step can further correct the upsampling accuracy of the UNET algorithm model to improve the feature fusion effect of the UNET algorithm model in the decoding process.

[0169] Furthermore, the layer-by-layer decoding process of the first local feature information is repeated in the updated UNET algorithm model to finally obtain the third local feature information, and the third local feature information is convolved to obtain the second primary three-dimensional image matrix. The second primary three-dimensional image matrix is ​​also threshold processed to obtain the third segmentation mask, and then it is determined whether the third segmentation mask meets the second preset threshold. If so, the second primary three-dimensional image matrix is ​​output as the second three-dimensional image matrix.

[0170] It should be noted that, in this process, if the third segmentation mask still does not meet the second preset threshold, the aforementioned steps can be repeated to modify the parameters in the UNET algorithm model until the latest segmentation mask meets the second preset threshold.

[0171] This embodiment provides the processing steps of the UNET algorithm model between the first three-dimensional image matrix and the second three-dimensional image matrix, so that the local features contained in the second three-dimensional image matrix are more accurate and comprehensive, so as to improve the accuracy of the first modal features.

[0172] See also Figure 4 In some embodiments, keywords are extracted from the first clinical information to obtain multiple semantic keywords including:

[0173] S301, performing data cleaning on the first clinical information to obtain a first information text;

[0174] S302, segmenting the first information text into sentences to obtain a plurality of to-be-processed sentence segments, each of which has a plurality of words;

[0175] S303, inputting the current sentence segments to be processed into the BERT variant model one by one, obtaining the first embedding vector information of each sentence segment to be processed, wherein the first embedding vector information includes the first word embedding vector of each vocabulary in the same sentence segment to be processed;

[0176] S304, inputting the first embedding vector information into the TF-IDF model to obtain a weight value corresponding to the first word embedding vector of the same sentence to be processed;

[0177] S305. Arrange the multiple first word embedding vectors in the first information text from left to right according to the size of the weight value, select the words corresponding to a preset number of first word embedding vectors from the left as semantic keywords, and record the first word embedding vectors corresponding to the semantic keywords as second word embedding vectors.

[0178] In step S301, the first clinical information is cleaned to remove useless characters in the first clinical information. Since the first clinical information is manually input or presented in the form of notes by the doctor, there are some useless characters in the first clinical information, such as modal particles, unnecessary punctuation marks, etc. The cleaned first clinical information is recorded as the first information text.

[0179] In step S302, the first information text is sentence segmented, that is, the first information text is segmented sentence by sentence to obtain a plurality of to-be-processed segments, each of which describes at least one clinical symptom. For example, "diarrhea 5 times, bloody stools 2 times, accompanied by pain" can be divided into three to-be-processed segments, namely "diarrhea 5 times", "bloody stools 2 times", and "accompanied by pain", and each to-be-processed segment has multiple words.

[0180] In step S303, the current sentence segments to be processed are input into the BERT variant model one by one, and the BERT variant model is preferably the ClinicalBERT model. The BERT variant model converts the text information contained in the sentence segments to be processed into a high-dimensional vector, and stores the features of the semantic information and the context connection relationship for each word in the high-dimensional vector, and finally obtains multiple first embedding vector information.

[0181] In step S304, the TF-IDF model is a text feature weighted algorithm model. Specifically, the TF-IDF model calculates the frequency of each word in the first information text, and combines the frequency of the word in the entire first information text set to obtain the weight value of each word. The weight value obtained by TF-IDF reflects the importance of each word in the current sentence to be processed. The higher the weight value, the more important the word is in the current sentence to be processed, and the greater the semantic contribution to the sentence to be processed.

[0182] In step S305, multiple first word embedding vectors in the first information text are arranged in order from left to right according to the weight values, and then a preset number of first word embedding vectors are selected from the left, that is, the words corresponding to the first word embedding vectors with the greatest semantic contribution to the first information text are selected as semantic keywords. The preset number can be set according to actual needs, and this embodiment does not limit this. For ease of expression, this embodiment also records the first word embedding vector corresponding to the semantic keyword as the second word embedding vector.

[0183] For further information, see Figure 5 In some embodiments, after obtaining the semantic keywords and the corresponding second word embedding vectors, multiple semantic keywords are input into the semantic dictionary to construct the semantic feature vector of the current first clinical information, which is recorded as the second modality feature including:

[0184] S401, obtaining a semantic dictionary, where the semantic dictionary is obtained by training based on sample clinical information, and the semantic dictionary includes multiple clinical terms;

[0185] S402, matching semantic keywords with clinical terms one by one;

[0186] S403, filling the second word embedding vector associated with the successfully matched semantic keyword into the semantic vector corresponding to the clinical term, and recording it as the first semantic vector;

[0187] S404, setting the semantic vector corresponding to the unmatched clinical terms to 0;

[0188] S405, performing feature concatenation on the plurality of first semantic vectors to obtain a semantic feature vector represented by a matrix, which is recorded as a second modal feature;

[0189] In step S401, the semantic dictionary can capture professional terms and concepts in the clinical field and the semantic relationships between them. Optionally, the semantic dictionary is trained based on sample clinical information. For ease of expression, the professional terms in the semantic dictionary are recorded as clinical entries.

[0190] In step S402, the semantic keywords are matched with the clinical terms. It is understandable that this matching process includes matching of semantic similarity. Preferably, the cosine similarity algorithm described below is used to calculate the semantic similarity between multiple semantic keywords and multiple clinical terms.

[0191] In step S403, the second word embedding vector corresponding to the successfully matched semantic keyword is filled into the semantic vector corresponding to the clinical term, and the clinical term matched with the semantic keyword is fused into a first semantic vector, reflecting the semantic features of the clinical term after the fusion of the semantic keyword.

[0192] In step S404, if there are clinical terms that have not been successfully matched, the semantic vector corresponding to the clinical terms that have not been successfully matched is set to 0 to indicate that the current patient does not have such a symptom.

[0193] In step S405, multiple first semantic vectors are concatenated to finally obtain a semantic feature vector represented by a matrix, which is recorded as a second modal feature.

[0194] Furthermore, matching semantic keywords with clinical terms one by one includes:

[0195] Get the second word embedding vector of each semantic keyword;

[0196] Obtain the semantic vector of each clinical term;

[0197] Calculate the cosine similarity between the second word embedding vector and the current semantic vector one by one;

[0198] Determine whether the cosine similarity exceeds the semantic similarity threshold;

[0199] If yes, the cosine similarity is recorded as the first cosine similarity;

[0200] Determine whether the number of the first cosine similarity is greater than 1;

[0201] If the number of first cosine similarities is greater than 1, multiple first cosine similarities are arranged in descending order according to the similarity values, and the semantic keyword corresponding to the second word embedding vector with the highest first cosine similarity is selected to match with the current clinical term;

[0202] If the number of the first cosine similarity is 1, the semantic keyword associated with the first cosine similarity is matched with the current clinical term;

[0203] If not, it means that the current clinical term matching is unsuccessful.

[0204] In this embodiment, the clinical terms have a semantic vector in the semantic dictionary, which exists in the training stage of the semantic dictionary and is used to characterize the degree of influence of the current clinical terms in this disease. The second word embedding vector of each semantic keyword is obtained, and the cosine similarity between the second word embedding vector and the semantic vector is calculated. Specifically, when there are multiple semantic keywords and multiple clinical terms, it is necessary to calculate the cosine similarity between the multiple semantic keywords and each clinical term. Optionally, in order to reduce subsequent computing power, when a semantic keyword successfully matches a clinical term, the successfully matched semantic keyword does not participate in the matching process of the remaining clinical terms. This method can improve the matching efficiency of the entire semantic keyword and clinical terms.

[0205] Furthermore, in this embodiment, a semantic similarity threshold is also set, and the semantic similarity threshold can be adjusted according to actual needs. The semantic similarity threshold indicates that the current semantic keyword can reach the similarity standard with the clinical entry. For example, the semantic similarity between blood in stool and blood in stool meets the semantic similarity threshold, while the semantic similarity between abdominal pain and blood in stool does not reach the standard set by the semantic similarity threshold. That is, the semantic similarity threshold belongs to the lower limit standard for judging clinical entries and semantic keywords.

[0206] If the cosine similarity calculated between the semantic keyword and the clinical term does not exceed the semantic similarity threshold, it means that the current semantic keyword and the clinical term are not similar or the similarity is low, that is, the current semantic keyword and the clinical term are not successfully matched.

[0207] If the cosine similarity calculated between the semantic keyword and the clinical term exceeds the semantic similarity threshold, it means that the current semantic keyword has a certain correlation with this clinical term. Then the calculation and judgment of the cosine similarity between the next semantic keyword and the current clinical term will continue until the remaining unmatched semantic keywords are judged. The cosine similarity corresponding to the semantic keyword similar to the current clinical term is recorded as the first cosine similarity.

[0208] It is further determined whether the number of first cosine similarities is greater than 1. If it is greater than 1, it means that there are multiple first cosine similarities, that is, there are multiple semantic keywords that are similar to the current clinical term. Then, the multiple first cosine similarities are arranged in descending order according to the similarity value, and the semantic keyword corresponding to the second word embedding vector with the highest first cosine similarity is selected to match the current clinical term, which means that the current clinical term is successfully matched.

[0209] If the number of the first cosine similarity is 1, it means that there is currently only one semantic keyword that is similar to the clinical entry, and then this semantic keyword is matched with the current clinical entry.

[0210] This embodiment constructs a rich semantic feature vector by combining semantic keywords with a pre-trained semantic dictionary. This feature vector can better capture the semantic information of clinical texts and provide valuable input for subsequent disease prediction or other medical decisions.

[0211] In some embodiments, the family medical history information is input into the random forest algorithm model to obtain the third modality feature including:

[0212] The family tree information is constructed based on the family members in the family medical history information as nodes and the kinship relationship as the hierarchical basis. The family members with the second clinical information are assigned a value of 1, and the family members without the second clinical information are assigned a value of 0;

[0213] The family tree information is used as input features, and the semantic keywords are used as variable labels to input into the random forest algorithm model, and the influencing features of each family member under the current disease are obtained, which are recorded as the first influencing features;

[0214] The importance of the first influencing feature is calculated one by one and recorded as the first importance, which is expressed by formula (1). Formula (1) is as follows:

[0215] ;

[0216] In formula (1), For the The first importance of the first influencing feature, For the tree Use in When features are split Impurity reduction, is the total number of trees in the random forest algorithm model, is the tree in the random forest algorithm model;

[0217] The multiple first importances are normalized to obtain multiple second importances, which are expressed by formula (2). Formula (2) is as follows:

[0218] ;

[0219] In formula (2), For the The second importance of the first influencing feature, N is the number of nodes of the family;

[0220] The plurality of second importances are represented by numerical sequences to obtain the third modal features.

[0221] In this embodiment, family medical history information can be obtained by constructing a medical consultation database and associating the medical consultation database with population information. Family medical history information can also be provided directly by the user. Optionally, the user can provide family information within three generations of direct blood relatives, and then the family medical history information of the user is improved and supplemented through the medical consultation database.

[0222] The family members in the family medical history information are used as nodes in the family tree information, and the kinship relationship is used as the hierarchical basis. For example, the father is one level higher than the user, and the grandfather is one level higher than the father, and so on. The family members with the second clinical information are assigned a value of 1, and the family members without the second clinical information are assigned a value of 0, so that the family tracking of colorectal diseases can be realized.

[0223] Furthermore, the family tree information is used as input features, and the semantic keywords are used as variable labels to input into the random forest algorithm model, and the influence features of each family member under the current disease are obtained, which are recorded as the first influence features. The random forest algorithm model is an integrated learning algorithm that trains multiple decision tree models for prediction and classification. The semantic keywords can reflect the important concepts and feature information related to colorectal diseases. Through the random forest algorithm model, the influence degree of each family member in the family tree information on the colorectal disease is obtained. These influence degrees are recorded as the first influence features, reflecting the probability of each family member's influence on the disease suffered by the current user.

[0224] Furthermore, the importance of the first influencing feature is calculated. For the sake of distinction, the importance of the first influencing feature calculated for the first time is recorded as the first importance. For details, please refer to formula (1). Formula (1) calculates the average importance of each first influencing feature in the random forest algorithm model.

[0225] After obtaining the first importance, the first importance is normalized, and the difference between the obtained second importance and the first importance is that the cumulative sum of multiple second importances is 1, that is, the second importance is the relative importance of each first influencing feature in the random forest algorithm model.

[0226] In this embodiment, the calculated multiple second importances are represented by numerical sequences to form a third modal feature for subsequent modal fusion steps.

[0227] This embodiment quantifies the influence of each family member on the colorectal disease of the current user and converts it into a feature representation that can be used in a machine learning model. The formation of the third modality feature facilitates subsequent multimodal fusion and pre-diagnosis steps based on multimodal fusion.

[0228] In some embodiments, the first modal feature, the second modal feature, and the third modal feature are input into a feature fusion algorithm model for feature fusion, and the first fusion feature is obtained, including:

[0229] The first modal feature, the second modal feature and the third modal feature are concatenated to obtain a first concatenated feature, and the first concatenated feature is expressed by formula (3). Formula (3) is as follows:

[0230] ;

[0231] In formula (3), is the first modal feature, is the second modal feature, is the third modal feature, is the first splicing feature;

[0232] The first concatenated feature is input into the attention mechanism algorithm model to obtain the query vector, key vector and value vector, which are expressed by formula (4). Formula (4) is as follows:

[0233]

[0234] In formula (4), is the query vector, is the key vector, is a value vector, is the weight matrix of the query vector, is the weight matrix of the key vector, is the weight matrix of the value vector;

[0235] The attention weight is calculated according to the query vector, key vector and value vector to obtain the attention weight matrix, which is expressed by formula (5). Formula (5) is as follows:

[0236] ;

[0237] In formula (5), for The transposed matrix of is the dimension of the key vector, is the attention of the first modal feature to itself, is the attention of the first modal feature to the second modal feature, is the attention degree of the first modal feature to the third modal feature, is the attention of the second modal feature to the first modal feature, is the attention of the second modal feature to itself, is the attention degree of the second modal feature to the third modal feature, is the attention of the third modal feature to the first modal feature, is the attention of the third modal feature to the second modal feature, is the third modal feature’s attention to itself;

[0238] The value vector is weighted and summed according to the attention weight matrix to obtain the attention fusion feature. The attention fusion feature is expressed by formula (6), which is as follows:

[0239] ;

[0240] In formula (6), is the attention fusion feature, is the weighted sum function;

[0241] The attention fusion feature is fused with the first concatenation feature to obtain the first fusion feature. The first fusion feature is expressed by formula (7). Formula (7) is as follows:

[0242] ;

[0243] In formula (7), is the first fusion feature.

[0244] In this embodiment, the first modal feature, the second modal feature and the third modal feature are first spliced ​​to form a first spliced ​​feature, and then the first spliced ​​feature is introduced into the attention mechanism algorithm model. Through the attention mechanism, the correlation and interaction between the three modal features can be learned. The query vector, the key vector and the value vector contain the interaction information between the three modal features, which helps to better integrate different features. Furthermore, the attention weight matrix is ​​calculated based on the query vector, the key vector and the value vector, and then the attention fusion feature is obtained. Finally, the attention fusion feature is fused with the first spliced ​​feature to obtain a first fusion feature with rich feature content, which is convenient for improving the accuracy of the subsequent first pre-diagnosis result.

[0245] In a third aspect, this embodiment further provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0246] The computer program involved in this embodiment can be stored in a computer device readable storage medium, which includes but is not limited to a disk, a tape, a magnetic card, a floppy disk, a flash memory, an optical disk, an optical card, a read-only memory (ROM), a random access memory (RAM), an erasable programmable ROM (EPROM) and an electrically erasable programmable ROM (EEPROM), etc., and also includes other biological, physical or chemical structures that can achieve the same or equivalent functions as the storage media listed above, such as DNA, RNA, protein and other units with information storage capabilities. In a specific embodiment, the storage medium involved can be one of the above-mentioned media types, or a combination of the above-mentioned media types. In different embodiments, the computer program involved in the embodiment can be stored in a single medium in a centralized manner, or it can be stored in multiple media in a distributed manner. The memory containing the computer device readable storage medium can be a non-volatile memory or a random access memory. These computer device readable storage media can be built into the device, or they can be connected to the device involved in the embodiment as an external device or a part of an external device. In some embodiments, a memory with a computer device readable storage medium is deployed locally; in other embodiments, a solution of deploying the memory away from the processor may also be adopted, such as a network attached memory accessed via an RF circuit or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), etc., or a suitable combination thereof, as long as the computer device can access the memory. In addition, the computer program involved in the embodiment may be stored in plaintext / ciphertext form, or may be designed as training data, which may be integrated and reorganized through model training and implicitly stored in the parameter state of a deep neural network or other machine learning model.

[0247] In a fourth aspect, this embodiment further provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0248] The processor described in this embodiment can be implemented by hardware, firmware, software or a combination thereof, and can use a circuit, a single or multiple application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), a programmable logic device (Programmable Logic Device, PLD), a field programmable gate array (Field Programmable Gate Array, FPGA), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, at least one of a microprocessor, and also includes other physical, biological or chemical structures that can achieve similar or equivalent functions to the processors listed above, such as biological neurons, quantum computing units, DNA computing units, etc., so that the processor can execute some or all of the steps in the computer program or method involved in the various embodiments of the present application, or any combination of the steps mentioned therein.

[0249] In the above technical scheme, by converting the imaging information into a first modal feature, the first clinical information into a second modal feature, and the family medical history information into a third modal feature, the first modal feature, the second modal feature and the third modal feature are feature fused and input into a deep learning neural network model. The first preliminary diagnosis report finally outputted can provide the current user's disease risk level information, disease stage feature prediction information, disease cancer probability information and disease treatment plan information. The entire technical scheme applies the principle of multimodal technology to integrate multiple information sources such as the patient's personal information, imaging information, first clinical symptoms and family medical history information, so as to improve the comprehensive analysis of the current user's colorectal disease, enable doctors to obtain a more comprehensive disease assessment, and then achieve more accurate risk level classification, disease stage prediction and personalized treatment plan formulation, thereby improving the efficiency and accuracy of the diagnosis and treatment process.

[0250] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concept of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. A multimodal colorectal cancer prognostic information processing method, characterized in that: include: Collecting the user's medical information, the medical information includes personal information, imaging information, first clinical information and family medical history information, the personal information includes the current user's name, permanent residence information, and user ID number, the imaging information includes multiple pathological images of the current user, the first clinical information includes clinical symptoms, duration of the disease course and onset node of the current medical stage, and the family medical history information includes second clinical information of family members who are related to the current user under the current disease; Inputting the image information into a UNET algorithm model to obtain a first modal feature, wherein the first modal feature includes at least one of a lesion area, a tumor size, a tumor distribution information, and a percentage of cancerous cells in the tumor; Extracting keywords from the first clinical information to obtain a plurality of semantic keywords, inputting the plurality of semantic keywords into a semantic dictionary, and constructing a semantic feature vector of the current first clinical information, recorded as a second modal feature; Inputting the family medical history information into a random forest algorithm model to obtain a third modality feature, wherein the third modality feature includes specific disease characteristics of the current user under the current disease; Inputting the first modal feature, the second modal feature and the third modal feature into a feature fusion algorithm model for feature fusion to obtain a first fusion feature; Inputting the first fusion feature into the trained neural network model to obtain a first pre-diagnosis result, the first pre-diagnosis result including the disease risk level information of the current user, the stage feature prediction information of the disease, the canceration probability information of the disease, and the treatment plan information of the disease; The first pre-diagnosis result is converted into a first pre-diagnosis report and displayed.

2. The multimodal colorectal cancer prognostic information processing method according to claim 1, characterized in that: The image information is input into the UNET algorithm model to obtain the first modal features including: Annotating the image information using an annotator to obtain a plurality of first image information; Preprocessing the first image information one by one to obtain a plurality of second image information, wherein the preprocessing includes at least one of image enhancement, image cutting, and image normalization processing; Arranging the plurality of second image information in a collection order, and acquiring slice position information between two adjacent second image information, wherein the slice position information includes slice thickness and slice spacing; Stacking the plurality of sequentially acquired second image information along a preset direction according to the slice position information to generate a first three-dimensional image matrix; Inputting the first three-dimensional image matrix into the trained UNET algorithm model to obtain a second three-dimensional image matrix; The second three-dimensional image matrix is ​​mapped to the plurality of second image information to obtain the first modal feature.

3. The multimodal colorectal cancer prognostic information processing method according to claim 2, characterized in that: The first three-dimensional image matrix includes a plurality of first voxel vectors, and the second three-dimensional image matrix includes a plurality of second voxel vectors. The first three-dimensional image matrix is ​​input into the trained UNET algorithm model to obtain the second three-dimensional image matrix including: Encoding the first three-dimensional image matrix layer by layer to a first preset threshold value to obtain first local feature information, where the first local feature information includes a local feature extracted from each of the first voxel vectors in the layer-by-layer encoding process; Decoding the first local feature information layer by layer to obtain second local feature information, where the second local feature information includes a local feature of each second voxel vector, where the local feature of the second voxel vector is a local feature of each first voxel vector fused layer by layer during the decoding process; Convolving the second local feature information to obtain a first primary three-dimensional image matrix; performing threshold processing on the first primary three-dimensional image matrix to obtain a first segmentation mask; Determining whether the first segmentation mask meets a second preset threshold; If yes, outputting the first primary three-dimensional image matrix as a second three-dimensional image matrix; If not, modifying the first segmentation mask to obtain a second segmentation mask; Calculating a loss function of the UNET algorithm model in a layer-by-layer decoding process according to the second segmentation mask; Performing gradient feedback according to the loss function to update the model parameters used by the UNET algorithm model in the layer-by-layer decoding process; Decoding the first local feature information layer by layer in the updated UNET algorithm model to obtain third local feature information, wherein the third local feature information includes the local features of each of the first voxel vectors fused during the decoding process; Convolving the third local feature information to obtain a second primary three-dimensional image matrix; performing threshold processing on the second primary three-dimensional image matrix to obtain a third segmentation mask; Determining whether the third segmentation mask meets a second preset threshold; If so, the second primary three-dimensional image matrix is ​​output as a second three-dimensional image matrix.

4. The multimodal colorectal cancer prognostic information processing method according to claim 1, characterized in that: The first clinical information is subjected to keyword extraction to obtain multiple semantic keywords including: Performing data cleaning on the first clinical information to obtain a first information text; Segmenting the first information text into sentences to obtain a plurality of to-be-processed sentence segments, each of which has a plurality of words; Input the current to-be-processed segments into the BERT variant model one by one to obtain first embedding vector information of each to-be-processed segment, wherein the first embedding vector information includes a first word embedding vector of each vocabulary in the same to-be-processed segment; Inputting the first embedding vector information into the TF-IDF model to obtain a weight value corresponding to the first word embedding vector of the same sentence to be processed; Arrange the multiple first word embedding vectors in the first information text from left to right according to the size of the weight value, select a preset number of words corresponding to the first word embedding vectors from the left as the semantic keywords, and record the first word embedding vectors corresponding to the semantic keywords as second word embedding vectors.

5. The multimodal colorectal cancer prognostic information processing method according to claim 4, characterized in that: Inputting the plurality of semantic keywords into the semantic dictionary to construct the semantic feature vector of the current first clinical information, which is recorded as the second modal feature, includes: Acquire a semantic dictionary, wherein the semantic dictionary is obtained by training based on sample clinical information, and the semantic dictionary includes a plurality of clinical terms; Matching the semantic keywords with the clinical terms one by one; Filling the second word embedding vector associated with the successfully matched semantic keyword into the semantic vector corresponding to the clinical term and recording it as the first semantic vector; The semantic vector corresponding to the clinical term that has not been successfully matched is set to 0; Performing feature concatenation on the plurality of the first semantic vectors to obtain a semantic feature vector represented by a matrix, which is recorded as a second modal feature; Matching the semantic keywords with the clinical terms one by one includes: Obtaining a second word embedding vector for each of the semantic keywords; Obtaining a semantic vector of each of the clinical terms; Calculate the cosine similarity between the second word embedding vector and the current semantic vector one by one; Determining whether the cosine similarity exceeds a semantic similarity threshold; If yes, the cosine similarity is recorded as the first cosine similarity; Determine whether the number of the first cosine similarities is greater than 1; If the number of the first cosine similarities is greater than 1, arranging the plurality of the first cosine similarities in descending order according to the similarity values, and selecting the semantic keyword corresponding to the second word embedding vector with the highest first cosine similarity to match with the current clinical term; If the number of the first cosine similarity is 1, matching the semantic keyword associated with the first cosine similarity with the current clinical term; If not, it means that the current clinical term matching is unsuccessful.

6. The multimodal colorectal cancer prognostic information processing method according to claim 1, characterized in that: The family medical history information is input into the random forest algorithm model, and the third modal features obtained include: Taking the family members in the family medical history information as nodes and the kinship as the hierarchical basis to construct family tree information, assigning a value of 1 to the family members with the second clinical information and assigning a value of 0 to the family members without the second clinical information; The family tree information is used as an input feature, and the semantic keywords are used as variable labels to input into the random forest algorithm model, so as to obtain the influencing feature of each family member under the current disease, which is recorded as the first influencing feature; The importance of the first influencing features is calculated one by one and recorded as the first importance, which is expressed by formula (1). The formula (1) is as follows: ; In formula (1), For the The first importance of the first influencing feature, For the tree Use in When features are split Impurity reduction, is the total number of trees in the random forest algorithm model, is the tree in the random forest algorithm model; The plurality of first importances are normalized to obtain a plurality of second importances, which are expressed by formula (2). The formula (2) is as follows: ; In formula (2), For the The second importance of the first influencing feature, N is the number of nodes of the family; The third modal feature is obtained by representing a plurality of the second importances with a numerical sequence.

7. The multimodal colorectal cancer prognostic information processing method according to claim 1, characterized in that: The first modal feature, the second modal feature and the third modal feature are input into the feature fusion algorithm model for feature fusion, and the first fusion feature obtained includes: The first modal feature, the second modal feature and the third modal feature are concatenated to obtain a first concatenated feature, and the first concatenated feature is expressed by formula (3), which is as follows: ； In formula (3), is the first modal feature, is the second modal feature, is the third modal feature, is the first splicing feature; The first concatenated feature is input into the attention mechanism algorithm model to obtain a query vector, a key vector and a value vector, which are expressed by formula (4). Formula (4) is as follows: In formula (4), is the query vector, is the key vector, is a value vector, is the weight matrix of the query vector, is the weight matrix of the key vector, is the weight matrix of the value vector; The attention weight is calculated according to the query vector, the key vector and the value vector to obtain an attention weight matrix, which is expressed by formula (5). The formula (5) is as follows: ； In formula (5), for The transposed matrix of is the dimension of the key vector, is the attention of the first modal feature to itself, is the attention of the first modal feature to the second modal feature, is the attention degree of the first modal feature to the third modal feature, is the attention of the second modal feature to the first modal feature, is the attention of the second modal feature to itself, is the attention degree of the second modal feature to the third modal feature, is the attention of the third modal feature to the first modal feature, is the attention of the third modal feature to the second modal feature, is the third modal feature’s attention to itself; The value vector is weighted and summed according to the attention weight matrix to obtain an attention fusion feature, and the attention fusion feature is expressed by formula (6), which is as follows: ； In formula (6), is the attention fusion feature, is the weighted sum function; The attention fusion feature is fused with the first concatenation feature to obtain a first fusion feature, and the first fusion feature is expressed by formula (7), which is as follows: ； In formula (7), is the first fusion feature.

8. A multimodal colorectal cancer prognostic information processing system, characterized in that: The method for processing prognostic information according to any one of claims 1 to 7 is applicable, wherein the prognostic information processing system comprises an information acquisition module, a first feature conversion module, a second feature conversion module, a third feature conversion module, a feature fusion module and a prognostic report generation module; The information collection module is used to collect the user's medical information, which includes personal information, imaging information, first clinical information and family medical history information. The personal information includes the current user's name, permanent residence information, and user ID number. The imaging information includes multiple pathological images of the current user. The first clinical information includes clinical symptoms, duration of the disease course, and onset node of the current medical stage. The family medical history information includes second clinical information of family members who are related to the current user under the current disease. The first feature conversion module is used to input the image information into the UNET algorithm model to obtain a first modal feature, wherein the first modal feature includes at least one of the lesion area, tumor size, tumor distribution information, and the proportion of tumor cancerous cells; The second feature conversion module is used to extract keywords from the first clinical information to obtain multiple semantic keywords, input the multiple semantic keywords into a semantic dictionary, and construct a semantic feature vector of the current first clinical information, which is recorded as a second modal feature; The third feature conversion module is used to input the family medical history information into the random forest algorithm model to obtain a third modal feature, wherein the third modal feature includes a specific onset feature of the current user under the current disease; The feature fusion module is used to input the first modal feature, the second modal feature and the third modal feature into the feature fusion algorithm model to perform feature fusion to obtain a first fusion feature; The pre-diagnosis report generation module is used to input the first fusion feature into the trained neural network model to obtain a first pre-diagnosis result, which includes the current user's disease risk level information, disease stage feature prediction information, disease cancer probability information and disease treatment plan information, and convert the first pre-diagnosis result into a first pre-diagnosis report and display it.

9. A computer-readable storage medium storing computer program instructions, characterized in that: The computer program instructions implement the method of any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 7.

Images