Intelligent processing method and system for multi-modal biological medicine data

Abstract

The invention discloses an intelligent processing method and system for multi-modal biological medicine data. The method comprises the following steps: acquiring a tumor tissue microscopic image and clinical characteristics of a diagnosed patient object; performing semantic coding on the clinical features to obtain clinical feature semantic understanding feature vectors; performing feature extraction on the tumor tissue microscopic image through a tumor tissue morphological feature extractor based on a deep neural network model to obtain a tumor tissue microscopic feature map; performing target significance analysis on the tumor tissue microscopic feature map to obtain a sequence of target manifestation tumor tissue microscopic local feature vectors; performing class attention interaction fusion processing on the sequence of the clinical feature semantic understanding feature vector and the target explicit tumor tissue microscopic local feature vector to obtain a multi-modal medical semantic fusion feature; and determining a tumor type label based on the multi-modal medical semantic fusion features. Therefore, the tumor type can be detected and judged, and powerful support is provided for the aspects of early diagnosis, personalized treatment and the like of tumors.

Description

Technical Field

[0001] The present application relates to the field of intelligent data processing technology, and more specifically, to an intelligent processing method and system for multimodal biomedical data. Background Art

[0002] Cancer is a disease that seriously endangers human health, with high morbidity and mortality rates. The diagnosis and treatment of tumors require comprehensive consideration of a variety of biomedical data, such as the morphological characteristics of tumor tissues and clinical information of patients. However, there are complex correlations and heterogeneity between these data, and how to effectively use these data to classify and diagnose tumor types is a challenging problem. At present, traditional tumor classification methods are mainly based on manual experience or single-modality data, lacking a deep understanding and fusion of multimodal data, resulting in unsatisfactory detection quality for tumor types.

[0003] Therefore, an optimized intelligent processing solution for multimodal biomedical data is desired. Summary of the invention

[0004] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides an intelligent processing method and system for multimodal biomedical data, which obtains tumor tissue microscopic images and clinical characteristics of diagnosed patient objects; semantically encodes clinical characteristics to obtain clinical characteristic semantic understanding feature vectors; extracts features from tumor tissue microscopic images through a tumor tissue morphology feature extractor based on a deep neural network model to obtain a tumor tissue microscopic feature map; performs target significance analysis on the tumor tissue microscopic feature map to obtain a sequence of target-manifested tumor tissue microscopic local feature vectors; performs attention-like interactive fusion processing on the sequence of clinical characteristic semantic understanding feature vectors and target-manifested tumor tissue microscopic local feature vectors to obtain multimodal medical semantic fusion features; and determines the tumor type label based on the multimodal medical semantic fusion features. In this way, the tumor type can be detected and judged, providing strong support for early diagnosis and personalized treatment of tumors.

[0005] In a first aspect, a method for intelligently processing multimodal biomedical data is provided, comprising:

[0006] Obtaining tumor tissue microscopic images and clinical characteristics of the diagnosed patient subject;

[0007] semantically encoding the clinical features to obtain a clinical feature semantic understanding feature vector;

[0008] Extracting features from the tumor tissue microscopic image using a tumor tissue morphology feature extractor based on a deep neural network model to obtain a tumor tissue microscopic feature map;

[0009] Performing target significance analysis on the tumor tissue microscopic feature map to obtain a sequence of target significant tumor tissue microscopic local feature vectors;

[0010] Performing attention-like interactive fusion processing on the sequence of the clinical feature semantic understanding feature vector and the target visualized tumor tissue microscopic local feature vector to obtain a multimodal medical semantic fusion feature;

[0011] Based on the multimodal medical semantic fusion features, a tumor type label is determined.

[0012] In a second aspect, a multimodal biomedical data intelligent processing system is provided, which includes:

[0013] A diagnosed patient information acquisition module is used to obtain tumor tissue microscopic images and clinical characteristics of the diagnosed patient object;

[0014] A semantic encoding module, used for semantically encoding the clinical features to obtain a clinical feature semantic understanding feature vector;

[0015] A morphological feature extraction module, used for extracting features from the tumor tissue microscopic image by a tumor tissue morphological feature extractor based on a deep neural network model to obtain a tumor tissue microscopic feature map;

[0016] A target significance analysis module, used for performing target significance analysis on the tumor tissue microscopic feature map to obtain a sequence of target significant tumor tissue microscopic local feature vectors;

[0017] An attention-like interactive fusion module is used to perform attention-like interactive fusion processing on the sequence of the clinical feature semantic understanding feature vector and the target visualized tumor tissue microscopic local feature vector to obtain a multimodal medical semantic fusion feature;

[0018] The tumor type label module is used to determine the tumor type label based on the multimodal medical semantic fusion features. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 Flow chart of a method for intelligently processing multimodal biomedical data according to an embodiment of the present application.

[0021] Figure 2Schematic diagram of the architecture of the intelligent processing method of multimodal biomedical data according to an embodiment of the present application.

[0022] Figure 3 4 is a block diagram of an intelligent processing system for multimodal biomedical data according to an embodiment of the present application.

[0023] Figure 4 Schematic diagram of a scenario of an intelligent processing method for multimodal biomedical data according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] Unless otherwise specified, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the present application. The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit the scope of this application.

[0026] In the embodiments of the present application, it should be noted that, unless otherwise specified and limited, the term "connection" should be understood in a broad sense. For example, it can be an electrical connection or a connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0027] It should be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\third" can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\third" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0028] The diagnosis and treatment of tumors is a complex process that requires comprehensive consideration of a variety of biomedical data. The general process of tumor diagnosis and treatment includes: First, the doctor will collect the patient's clinical information, including medical history, symptoms, signs, etc. This information can help the doctor understand the patient's disease condition and guide subsequent diagnosis and treatment decisions. Commonly used imaging examinations include X-rays, CT scans, MRIs, and ultrasounds. These examinations can provide information such as the location, size, and morphology of the tumor to help doctors determine the nature and stage of the tumor.

[0029] Histological examination is one of the key steps in diagnosing tumors. Tumor tissue samples are obtained through biopsy or surgical resection. After examination and analysis by pathologists, the type, grade and histological characteristics of the tumor can be determined, providing important diagnostic basis. Molecular biological examination can further refine the diagnosis and prognosis evaluation of tumors. For example, by detecting gene mutations, protein expression levels, chromosomal abnormalities, etc. of tumors, it can help determine the molecular subtype of tumors and guide the selection of individualized treatment strategies. The detection of some tumor-specific markers also has a certain auxiliary role in the diagnosis and treatment of tumors. For example, prostate-specific antigen (PSA) plays an important role in the diagnosis and monitoring of prostate cancer. Genetic examination can help determine the genetic susceptibility and familial nature of certain tumors and guide genetic counseling and screening for patients and families. Based on the patient's clinical information, imaging examinations and pathological results, doctors can stage the tumor, that is, determine the extent and severity of the tumor. Staging helps to formulate appropriate treatment plans and predict prognosis. Based on the type, stage, molecular characteristics of the tumor and the patient's overall condition, the doctor will develop an individualized treatment plan. Common tumor treatments include surgery, radiotherapy, chemotherapy, targeted therapy, immunotherapy, etc., and sometimes a combination of multiple treatment methods is used.

[0030] Traditional tumor classification methods are mainly based on manual experience or single-modality data, such as tissue morphological characteristics, clinical manifestations, etc. These methods have some limitations, including subjectivity, limitations, and inadequacy to fully understand the complexity of tumors. Histological classification is the basis of tumor classification. Tumors are divided into different types based on the morphological characteristics, tissue structure, and growth pattern of tumor cells. This classification method relies on the experience and expertise of pathologists, and is subjective and has certain errors.

[0031] Molecular marker classification uses molecular changes within tumor cells to classify different types of tumors. For example, tumors can be classified by detecting changes in molecular levels such as gene mutations and protein expression in tumor cells. However, the limitation of this method is that it only considers a single molecular marker and cannot fully reflect the complexity of the tumor.

[0032] Clinical classification is to classify tumors into different types based on the patient's clinical manifestations and pathological characteristics. For example, breast cancer can be classified according to the patient's age, tumor size, lymph node metastasis, etc. This method is simple and easy, but lacks accurate assessment of the tumor's molecular characteristics.

[0033] Traditional tumor classification methods can help doctors make diagnosis and treatment decisions to a certain extent, but they have some limitations. With the development of biomedical technology, more and more research is devoted to developing tumor classification methods based on multimodal data, such as integrating multiple biomedical data (such as genomics, transcriptomics, proteomics, etc.) and machine learning algorithms to improve the accuracy of tumor classification and the effectiveness of personalized treatment.

[0034] Figure 1 Flow chart of a method for intelligently processing multimodal biomedical data according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of the architecture of the intelligent processing method of multimodal biomedical data according to an embodiment of the present application. Figure 1 and Figure 2 As shown, the intelligent processing method of multimodal biomedical data includes: 110, obtaining tumor tissue microscopic images and clinical characteristics of the diagnosed patient object; 120, semantically encoding the clinical characteristics to obtain clinical characteristic semantic understanding feature vectors; 130, extracting features from the tumor tissue microscopic images through a tumor tissue morphological feature extractor based on a deep neural network model to obtain a tumor tissue microscopic feature map; 140, performing target significance analysis on the tumor tissue microscopic feature map to obtain a sequence of target explicit tumor tissue microscopic local feature vectors; 150, performing attention-like interactive fusion processing on the sequence of the clinical characteristic semantic understanding feature vector and the target explicit tumor tissue microscopic local feature vector to obtain a multimodal medical semantic fusion feature; 160, determining the tumor type label based on the multimodal medical semantic fusion feature.

[0035] In step 110, the tumor tissue microscopic images and clinical characteristics of the diagnosed patient are obtained. Ensure that the obtained tumor tissue microscopic images and clinical characteristics are accurate and complete to ensure the reliability of subsequent analysis. The tumor tissue microscopic images provide the morphological characteristics of the tumor, and the clinical characteristics provide the clinical manifestations and pathological characteristics of the patient, which are the basis for tumor classification.

[0036] In step 120, the clinical features are semantically encoded to obtain a semantically understood feature vector of the clinical features. An appropriate encoding method is selected to convert the clinical features into a semantically understood feature vector, including a bag-of-words model, word embedding, etc. Through semantic encoding, the clinical features are converted into a feature vector that can be understood and processed by a computer, providing a basis for subsequent feature fusion and classification.

[0037] In step 130, the tumor tissue microscopic image is subjected to feature extraction by a tumor tissue morphological feature extractor based on a deep neural network model to obtain a tumor tissue microscopic feature map. An appropriate deep neural network model, such as a convolutional neural network (CNN), is selected to extract morphological features in the tumor tissue microscopic image. The selection and training of the model need to take into account the diversity and scale of the data. Through the deep neural network model, morphological features in the tumor tissue microscopic image, such as cell shape, nuclear size, etc., can be automatically extracted, providing a basis for subsequent feature fusion and classification.

[0038] In step 140, the target significance analysis is performed on the tumor tissue microscopic feature map to obtain a sequence of target manifested tumor tissue microscopic local feature vectors. Using target significance analysis methods, such as significance maps, attention mechanisms, etc., to highlight the significant features of the tumor area is helpful to extract the microscopic local features of the tumor tissue and reduce the interference of non-tumor areas. Through target significance analysis, important areas in the tumor tissue microscopic image can be highlighted, local features can be extracted, and the expression and classification effect of tumor features can be enhanced.

[0039] In step 150, the sequence of the clinical feature semantic understanding feature vector and the target visualized tumor tissue microscopic local feature vector is subjected to attention-like interactive fusion processing to obtain a multimodal medical semantic fusion feature. Using the attention-like mechanism, the clinical features and the tumor tissue microscopic local features are interactively fused to obtain a multimodal medical semantic fusion feature, which can be achieved by calculating the attention weight and weighted fusion of features. Through the multimodal medical semantic fusion feature, the clinical features and the tumor tissue microscopic local features are comprehensively utilized to improve the accuracy and robustness of tumor classification.

[0040] In step 160, the tumor type label is determined based on the multimodal medical semantic fusion features. A suitable classification algorithm, such as a support vector machine (SVM), a random forest, etc., is used to classify the tumor based on the multimodal medical semantic fusion features. Through the comprehensive analysis of the multimodal medical semantic fusion features, the type label of the tumor can be determined more accurately, providing guidance for the diagnosis and treatment of the patient.

[0041] It should be understood that since tumor tissue microscopic images can provide detailed information about the morphological characteristics of tumor cells, such as the size, shape and cell structure of the tumor, this information is very important for the classification and diagnosis of tumors. Clinical characteristics include the patient's personal information, medical history and symptoms, which can provide a description of the overall condition of the patient. These two types of data belong to different data types and have complementary semantic information about the patient's condition. In addition, considering that different patients may have different tumor characteristics and clinical conditions, in order to more fully and accurately detect and judge the tumor type of the patient, it is necessary to combine these two types of data to obtain more comprehensive and multi-dimensional patient tumor information.

[0042] Specifically, the technical concept of the present application is to collect tumor tissue microscopic images and clinical characteristics of patient subjects, and introduce image processing and semantic analysis algorithms at the back end to perform data correlation analysis of the tumor tissue microscopic images and clinical characteristics of the patient subjects, so as to detect and judge the tumor type, and provide strong support for early diagnosis and personalized treatment of tumors.

[0043] More specifically, in the technical solution of the present application, first, the tumor tissue microscopic image and clinical characteristics of the diagnosed patient object are obtained. It should be understood that the tumor tissue microscopic image is a pathological image, which is an image obtained by techniques such as tissue sectioning and staining. The tumor tissue microscopic image can provide information about tumor morphology, tissue structure and cell characteristics. Analysis and feature extraction of the image are conducive to extracting key features of tumor cells in the image, such as nuclear morphology, cell arrangement, etc. The clinical characteristics include clinical information of the patient, such as age, gender, medical history, symptoms, etc. This information can provide the clinical background and patient characteristics of the tumor. Extracting these feature information can help discover feature patterns related to tumor type and grade.

[0044] Next, considering that the clinical characteristics include the patient's clinical information, such as age, gender, medical history, symptoms, etc., this information has semantic characteristics. Therefore, in order to effectively understand the clinical characteristics of the patient object semantically to capture the important semantic information in the clinical characteristics, improve the expression ability and discrimination of the clinical characteristics of the patient object, so as to facilitate the subsequent classification of tumor types, in the technical solution of this application, the clinical characteristics are further semantically encoded to obtain a clinical characteristic semantic understanding feature vector.

[0045] In a specific embodiment of the present application, the clinical features are semantically encoded to obtain a clinical feature semantic understanding feature vector, including: performing word segmentation processing on the clinical features to convert the clinical features into a word sequence composed of multiple words; using the word embedding layer of the clinical feature semantic encoder including the word embedding layer to map each word in the word sequence to a word vector to obtain a sequence of word vectors; and, using the clinical feature semantic encoder including the word embedding layer to perform global context-based semantic encoding on the sequence of word vectors to obtain the clinical feature semantic understanding feature vector.

[0046] Then, a tumor tissue morphology feature extractor based on a convolutional neural network model with excellent performance in extracting implicit features from images is used to perform feature extraction on the tumor tissue microscopic image to extract implicit feature distribution information about the tumor tissue in the tumor tissue microscopic image, thereby obtaining a tumor tissue microscopic feature map.

[0047] Among them, the deep neural network model is a convolutional neural network model.

[0048] It should be understood that there may be a large amount of background feature information and feature information of non-tumor areas in the tumor tissue microscopic feature map, and in the actual morphological analysis and type detection tasks of tumor cells, the focus of attention is usually the tumor area. Therefore, in the technical solution of the present application, in order to be able to adjust the weights of different areas in the tumor tissue microscopic feature map so that the feature extraction task focuses on the tumor tissue features, it is necessary to further pass the tumor tissue microscopic feature map through the target object significant attention layer to obtain a target manifested tumor tissue microscopic feature map. Processing through the target object significant attention layer can focus attention on the tumor area in the image, highlighting the important feature information of the tumor tissue. In this way, the model's attention to the key features of the tumor area can be improved, reducing the interference of background and irrelevant information, which helps to more accurately analyze and identify tumor types.

[0049] In a specific embodiment of the present application, a target significance analysis is performed on the tumor tissue microscopic feature map to obtain a sequence of target manifested tumor tissue microscopic local feature vectors, including: passing the tumor tissue microscopic feature map through a target object significant attention layer to obtain a target manifested tumor tissue microscopic feature map; performing feature matrix expansion on each local feature of the target manifested tumor tissue microscopic feature map along the channel dimension to obtain a sequence of target manifested tumor tissue microscopic local feature vectors.

[0050] Further, considering that the feature vectors of each local feature along the channel dimension of the target visualized tumor tissue microscopic feature map represent the implicit feature information of the tumor tissue of the patient object, these feature information not only have a mutual correlation relationship, but also can be compared with the clinical semantic information of the patient object to more accurately detect and judge the tumor type. Therefore, in the technical solution of the present application, after further expanding the feature matrix of each local feature along the channel dimension of the target visualized tumor tissue microscopic feature map to obtain a sequence of target visualized tumor tissue microscopic local feature vectors, the sequence of target visualized tumor tissue microscopic local feature vectors and the clinical feature semantic understanding feature vector are processed based on the class attention interaction fusion module to obtain a multimodal medical semantic fusion feature vector. In particular, the class attention interaction fusion module can automatically select and focus on important features related to classification and diagnosis by learning attention weights. Through the class attention mechanism, the tumor tissue microscopic local feature vector and the clinical feature semantic understanding feature vector can be weighted fused, so that important features are given higher weights, improving the expressiveness and discrimination of features. In this way, by interacting and fusing different types of patient object features, richer and more discriminative feature representations can be extracted, and the multimodal medical semantic fusion feature vector with greater discrimination and expression ability can be obtained, which is conducive to improving the accuracy of subsequent classification.

[0051] In a specific embodiment of the present application, the sequence of the clinical feature semantic understanding feature vector and the target visualized tumor tissue microscopic local feature vector are subjected to attention-like interactive fusion processing to obtain a multimodal medical semantic fusion feature, including: based on an attention-like interactive fusion module, the sequence of the target visualized tumor tissue microscopic local feature vector and the clinical feature semantic understanding feature vector are processed to obtain a multimodal medical semantic fusion feature vector as the multimodal medical semantic fusion feature.

[0052] Specifically, based on the attention-like interaction fusion module, the sequence of the target visualized tumor tissue microscopic local feature vector and the clinical feature semantic understanding feature vector are processed to obtain a multimodal medical semantic fusion feature vector as the multimodal medical semantic fusion feature, including: based on the attention-like interaction fusion module, the sequence of the target visualized tumor tissue microscopic local feature vector and the clinical feature semantic understanding feature vector are processed with the following fusion formula to obtain the multimodal medical semantic fusion feature vector; wherein the fusion formula is:

[0053]

[0054] Where v represents the semantic understanding feature vector of the clinical feature, and A represents 1×N w The matrix, Nw is equal to the scale of the clinical feature semantic understanding feature vector, and B is 1×N h The matrix, N h is equal to the number of target visualized tumor tissue microscopic local feature vectors in the sequence of the target visualized tumor tissue microscopic local feature vectors, σ is the Sigmoid function, s is the weight coefficient, M w and M h represents the convolution operation of 1×1 convolution kernel, h i represents each target manifested tumor tissue microscopic local feature vector in the sequence of the target manifested tumor tissue microscopic local feature vector, N represents the scale of each target manifested tumor tissue microscopic local feature vector in the sequence of the target manifested tumor tissue microscopic local feature vector, and v' represents the multimodal medical semantic fusion feature vector.

[0055] Then, the multimodal medical semantic fusion feature vector is passed through a classifier to obtain a classification result, and the classification result is used to represent a tumor type label. Specifically, the label of the classifier is a tumor type label. Therefore, after obtaining the classification result, the tumor type can be detected and judged based on the classification result, thereby providing strong support for early diagnosis and personalized treatment of tumors.

[0056] In a specific embodiment of the present application, based on the multimodal medical semantic fusion features, determining the tumor type label includes: passing the multimodal medical semantic fusion feature vector through a classifier to obtain a classification result, and the classification result is used to represent the tumor type label.

[0057] Specifically, in a specific embodiment of the present application, the multimodal medical semantic fusion feature vector is passed through a classifier to obtain a classification result, and the classification result is used to represent a tumor type label, including: using multiple fully connected layers of the classifier to fully connect encode the multimodal medical semantic fusion feature vector to obtain an encoded classification feature vector; and, passing the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

[0058] In the above technical solution, in the process of performing attention-like interactive fusion processing on the sequence of the clinical feature semantic understanding feature vector and the target visualized tumor tissue microscopic local feature vector, the position mean vector of the sequence of the target visualized tumor tissue microscopic local feature vector is used as the global semantic representation of its image features, and a nonlinear activation function is used to capture the semantic association information between the clinical feature semantic understanding feature vector and the global semantic representation of the image features of the tumor tissue microscopic image as the fusion weight between the two to apply the attention-like mechanism to the sequence of the clinical feature semantic understanding feature vector and the target visualized tumor tissue microscopic local feature vector to obtain the multimodal medical semantic fusion feature vector. However, considering that the sequence of the clinical feature semantic understanding feature vector and the target visualized tumor tissue microscopic local feature vector has feature modality differences and feature distribution density differences, when calculating the semantic association information between the two as the fusion weight between the two, the inconsistency of the feature granularity will cause the obtained multimodal medical semantic fusion feature vector to have a more significant local distribution difference, resulting in a deviation in its harmony relative to the predetermined category of the classifier, affecting the classification iteration effect.

[0059] Based on this, in a preferred embodiment, passing the multimodal medical semantic fusion feature vector through a classifier to obtain a classification result includes the following steps: first, calculating the autocorrelation matrix of the multimodal medical semantic fusion feature vector and its own transpose, and calculating the inner product of the i-th and j-th row vectors of the autocorrelation matrix as the matrix value of the (i, j)-th position of the weight matrix to obtain the weight matrix, then, after matrix multiplication of the weight matrix and the autocorrelation matrix, further matrix-vector multiplication is performed with the multimodal medical semantic fusion feature vector to obtain a correction vector, finally, the correction vector is dot-multiplied with the multimodal medical semantic fusion feature vector to obtain an optimized multimodal medical semantic fusion feature vector, and the optimized multimodal medical semantic fusion feature vector is passed through the classifier to obtain the classification result.

[0060] The optimization steps are specifically expressed as follows:

[0061]

[0062] Among them, V represents the multimodal medical semantic fusion feature vector, T represents the transpose of the vector, M represents the autocorrelation matrix, and V i and V j represents the i-th and j-th row vectors of the autocorrelation matrix, M′ represents the weight matrix, m i,j represents the matrix value of the (i, j)th position of the weight matrix, L represents the height of the autocorrelation matrix, and R represents a set of real numbers. represents matrix multiplication, ⊙ represents dot product, and V′ represents the optimized multimodal medical semantic fusion feature vector.

[0063] That is, in this preferred embodiment, based on the self-correlation dimension of the multimodal medical semantic fusion feature vector as the object to be modulated, the spatial sub-dimensional complexity of the high-dimensional feature space of the feature distribution is used to perform correlation expansion, thereby introducing the decomposable correlation dimension offset of the multimodal medical semantic fusion feature vector into the heterogeneous correlation embedding space, thereby enhancing its correlation self-consistency through the joint fine-tuning of the decomposable dimension set represented by the heterogeneous correlation embedding space, so as to improve the feature set of the multimodal medical semantic fusion feature vector in the predetermined category harmony in the associated target classification domain, thereby improving the speed of classification training and the accuracy of the classification results when the multimodal medical semantic fusion feature vector is classified by the classifier. In this way, the tumor type can be automatically detected and judged based on the tumor tissue microscopic image and clinical characteristics of the patient object, providing strong support for the early diagnosis and personalized treatment of tumors.

[0064] In summary, the intelligent processing method of multimodal biomedical data based on the embodiment of the present application is explained, which collects tumor tissue microscopic images and clinical characteristics of patient subjects, and introduces image processing and semantic analysis algorithms at the back end to perform data correlation analysis of the tumor tissue microscopic images and clinical characteristics of the patient subjects, so as to detect and judge the tumor type, providing strong support for early diagnosis and personalized treatment of tumors.

[0065] In one embodiment of the present application, Figure 3 FIG. 1 is a block diagram of an intelligent processing system for multimodal biomedical data according to an embodiment of the present application. Figure 3 As shown, according to the embodiment of the present application, the intelligent processing system 200 of multimodal biomedical data includes: a diagnosed patient information acquisition module 210, which is used to acquire the tumor tissue microscopic image and clinical characteristics of the diagnosed patient object; a semantic encoding module 220, which is used to semantically encode the clinical characteristics to obtain a clinical characteristic semantic understanding feature vector; a morphological feature extraction module 230, which is used to extract features from the tumor tissue microscopic image through a tumor tissue morphological feature extractor based on a deep neural network model to obtain a tumor tissue microscopic feature map; a target significance analysis module 240, which is used to perform target significance analysis on the tumor tissue microscopic feature map to obtain a sequence of target explicit tumor tissue microscopic local feature vectors; a class attention interactive fusion module 250, which is used to perform class attention interactive fusion processing on the sequence of the clinical characteristic semantic understanding feature vector and the target explicit tumor tissue microscopic local feature vector to obtain a multimodal medical semantic fusion feature; a tumor type label module 260, which is used to determine the tumor type label based on the multimodal medical semantic fusion feature.

[0066] In the intelligent processing system for multimodal biomedical data, the semantic encoding module is used to: perform word segmentation on the clinical features to convert the clinical features into a word sequence consisting of multiple words; use the word embedding layer of the clinical feature semantic encoder including the word embedding layer to map each word in the word sequence to a word vector to obtain a sequence of word vectors; and use the clinical feature semantic encoder including the word embedding layer to perform global contextual semantic encoding on the sequence of word vectors to obtain the clinical feature semantic understanding feature vector.

[0067] Here, those skilled in the art will appreciate that the specific functions and operations of the various units and modules in the above-mentioned intelligent processing system for multimodal biomedical data have been described in the above reference. Figure 1 to Figure 2 The description of the intelligent processing method of multimodal biomedical data has been introduced in detail, and therefore, its repeated description will be omitted.

[0068] As described above, the intelligent processing system 200 for multimodal biomedical data according to the embodiment of the present application can be implemented in various terminal devices, such as a server for intelligent processing of multimodal biomedical data. In one example, the intelligent processing system 200 for multimodal biomedical data according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the intelligent processing system 200 for multimodal biomedical data can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the intelligent processing system 200 for multimodal biomedical data can also be one of the many hardware modules of the terminal device.

[0069] Alternatively, in another example, the intelligent processing system 200 for multimodal biomedical data and the terminal device may also be separate devices, and the intelligent processing system 200 for multimodal biomedical data may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0070] Figure 4 FIG. 1 is a schematic diagram of a scenario of an intelligent processing method for multimodal biomedical data according to an embodiment of the present application. Figure 4 As shown, in this application scenario, first, a tumor tissue microscopic image of a diagnosed patient object is obtained (for example, Figure 4 C1) and clinical features (e.g. Figure 4 Then, the acquired tumor tissue microscopic images and clinical features are input into a server (e.g., Figure 4In S) shown in , the server is capable of processing the tumor tissue microscopic image and the clinical characteristics based on an intelligent processing algorithm of multimodal biomedical data to determine a tumor type label.

[0071] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0072] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0073] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for intelligent processing of multimodal biomedical data, characterized in that: include: Obtaining tumor tissue microscopic images and clinical characteristics of the diagnosed patient subject; semantically encoding the clinical features to obtain a clinical feature semantic understanding feature vector; Extracting features from the tumor tissue microscopic image using a tumor tissue morphology feature extractor based on a deep neural network model to obtain a tumor tissue microscopic feature map; Performing target significance analysis on the tumor tissue microscopic feature map to obtain a sequence of target significant tumor tissue microscopic local feature vectors; Performing attention-like interactive fusion processing on the sequence of the clinical feature semantic understanding feature vector and the target visualized tumor tissue microscopic local feature vector to obtain a multimodal medical semantic fusion feature; Based on the multimodal medical semantic fusion features, a tumor type label is determined.

2. The intelligent processing method for multimodal biomedical data according to claim 1, characterized in that: Semantically encoding the clinical features to obtain a clinical feature semantic understanding feature vector includes: Performing word segmentation processing on the clinical features to convert the clinical features into a word sequence consisting of multiple words; Using the word embedding layer of the clinical feature semantic encoder including the word embedding layer, each word in the word sequence is mapped to a word vector to obtain a sequence of word vectors; and The clinical feature semantic encoder including the word embedding layer is used to perform global context semantic encoding on the sequence of word vectors to obtain the clinical feature semantic understanding feature vector.

3. The intelligent processing method of multimodal biomedical data according to claim 2, characterized in that: The deep neural network model is a convolutional neural network model.

4. The intelligent processing method for multimodal biomedical data according to claim 3, characterized in that: Performing target significance analysis on the tumor tissue microscopic feature map to obtain a sequence of target significant tumor tissue microscopic local feature vectors, including: Passing the tumor tissue microscopic feature map through a target object salient attention layer to obtain a target manifested tumor tissue microscopic feature map; Each local feature of the target visualized tumor tissue microscopic feature map along the channel dimension is expanded into a feature matrix to obtain a sequence of the target visualized tumor tissue microscopic local feature vectors.

5. The intelligent processing method of multimodal biomedical data according to claim 4, characterized in that: The sequence of the clinical feature semantic understanding feature vector and the target visualized tumor tissue microscopic local feature vector is subjected to attention-like interactive fusion processing to obtain a multimodal medical semantic fusion feature, including: based on an attention-like interactive fusion module, the sequence of the target visualized tumor tissue microscopic local feature vector and the clinical feature semantic understanding feature vector are processed to obtain a multimodal medical semantic fusion feature vector as the multimodal medical semantic fusion feature.

6. The intelligent processing method of multimodal biomedical data according to claim 5, characterized in that: Based on the attention-like interactive fusion module, the sequence of the target visualized tumor tissue microscopic local feature vector and the clinical feature semantic understanding feature vector are processed to obtain a multimodal medical semantic fusion feature vector as the multimodal medical semantic fusion feature, including: based on the attention-like interactive fusion module, the sequence of the target visualized tumor tissue microscopic local feature vector and the clinical feature semantic understanding feature vector are processed with the following fusion formula to obtain the multimodal medical semantic fusion feature vector; Among them, the fusion formula is: Where v represents the semantic understanding feature vector of the clinical feature, and A represents 1×N w The matrix, N w is equal to the scale of the clinical feature semantic understanding feature vector, and B is 1×N h The matrix, N h is equal to the number of target visualized tumor tissue microscopic local feature vectors in the sequence of the target visualized tumor tissue microscopic local feature vectors, σ is the Sigmoid function, s is the weight coefficient, M w and M h represents the convolution operation of 1×1 convolution kernel, h i represents each target manifested tumor tissue microscopic local feature vector in the sequence of the target manifested tumor tissue microscopic local feature vector, N represents the scale of each target manifested tumor tissue microscopic local feature vector in the sequence of the target manifested tumor tissue microscopic local feature vector, and v' represents the multimodal medical semantic fusion feature vector.

7. The intelligent processing method of multimodal biomedical data according to claim 6, characterized in that: Based on the multimodal medical semantic fusion features, a tumor type label is determined, including: The multimodal medical semantic fusion feature vector is passed through a classifier to obtain a classification result, and the classification result is used to represent a tumor type label.

8. The intelligent processing method for multimodal biomedical data according to claim 7, characterized in that: The multimodal medical semantic fusion feature vector is passed through a classifier to obtain a classification result, and the classification result is used to represent a tumor type label, including: Using multiple fully connected layers of the classifier to perform fully connected encoding on the multimodal medical semantic fusion feature vector to obtain an encoded classification feature vector; and The encoded classification feature vector is passed through the Softmax classification function of the classifier to obtain the classification result.

9. An intelligent processing system for multimodal biomedical data, characterized in that: include: A diagnosed patient information acquisition module is used to obtain tumor tissue microscopic images and clinical characteristics of the diagnosed patient object; A semantic encoding module, used for semantically encoding the clinical features to obtain a clinical feature semantic understanding feature vector; A morphological feature extraction module, used for extracting features from the tumor tissue microscopic image by a tumor tissue morphological feature extractor based on a deep neural network model to obtain a tumor tissue microscopic feature map; A target significance analysis module, used for performing target significance analysis on the tumor tissue microscopic feature map to obtain a sequence of target significant tumor tissue microscopic local feature vectors; An attention-like interactive fusion module is used to perform attention-like interactive fusion processing on the sequence of the clinical feature semantic understanding feature vector and the target visualized tumor tissue microscopic local feature vector to obtain a multimodal medical semantic fusion feature; The tumor type label module is used to determine the tumor type label based on the multimodal medical semantic fusion features.

10. The intelligent processing system for multimodal biomedical data according to claim 9, characterized in that: The semantic encoding module is used to: Performing word segmentation processing on the clinical features to convert the clinical features into a word sequence consisting of multiple words; Using the word embedding layer of the clinical feature semantic encoder including the word embedding layer, each word in the word sequence is mapped to a word vector to obtain a sequence of word vectors; as well as The clinical feature semantic encoder including the word embedding layer is used to perform global context semantic encoding on the sequence of word vectors to obtain the clinical feature semantic understanding feature vector.

Images