Intelligent correlation analysis system for multi-modal medical data based on deep learning

Abstract

The application relates to the technical field of data processing, and particularly relates to a multi-modal medical data intelligent correlation analysis system based on deep learning. The system comprises a data acquisition module, which is used for acquiring a plurality of first data and a plurality of second data of a target patient, wherein the first data is structured medical data, and the second data is unstructured medical data; a keyword extraction module, which is used for extracting a plurality of keywords of each second data; a clustering module, which is used for clustering the plurality of second data based on the plurality of keywords of each second data to obtain a plurality of class clusters; a time sequence analysis module, which is used for performing time sequence analysis on a treatment time period based on the plurality of class clusters to determine a plurality of diagnosis and treatment time segments; and a correlation analysis module, which is used for performing correlation analysis on the second data and the first data in each diagnosis and treatment time segment according to the plurality of class clusters to obtain a correlation analysis result. The application can improve the correlation analysis effect on multi-modal medical data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a multimodal medical data intelligent correlation analysis system based on deep learning. Background Technology

[0002] With the continuous improvement of medical informatization, hospitals and various medical institutions have accumulated a large amount of complex medical data, including both structured and unstructured data. This data primarily includes electronic medical records (EMR), imaging data (such as CT, MRI, and ultrasound), pathological slide images, biochemical test data, genomic data, and vital sign monitoring data. This multimodal, multi-source, and multi-temporal-scale data contains rich clinical value. Effective integration and correlation analysis of this medical data could reveal individualized disease progression patterns, providing strong support for subsequent treatment response prediction, personalized treatment plan development, and prognostic assessment.

[0003] However, different modalities of medical data exhibit differences in data structure, dimensionality, and semantics. Traditional data analysis methods struggle to effectively extract deep-seated feature relationships from multimodal medical data, leading to insufficient or obscured potential clinical value. Furthermore, medical data generated by different patients at different stages of diagnosis and treatment often involves numerous examinations to assist doctors in diagnosing conditions. This diagnostic data may interfere with the analysis of correlations between other medical data generated after diagnosis, further weakening the effectiveness of related technologies in uncovering deep feature relationships among multimodal medical data.

[0004] In other words, the relevant technologies are not very effective in correlation analysis of multimodal medical data. Summary of the Invention

[0005] To address the technical problem of poor performance in correlation analysis of multimodal medical data using existing technologies, the present invention aims to provide an intelligent correlation analysis system for multimodal medical data based on deep learning. The specific technical solution adopted is as follows:

[0006] In a first aspect, one embodiment of the present invention provides a multimodal medical data intelligent correlation analysis system based on deep learning, the system comprising:

[0007] The data acquisition module is used to acquire multiple first data and multiple second data of the target patient, wherein the first data is structured medical data and the second data is unstructured medical data;

[0008] The keyword extraction module is used to extract keywords from each second data point, resulting in multiple keywords for each second data point.

[0009] The clustering module is used to cluster multiple second data points based on multiple keywords of each second data point, resulting in multiple clusters.

[0010] The time-series analysis module is used to perform time-series analysis on the treatment time periods corresponding to multiple second data based on multiple clusters, so as to determine multiple treatment time segments, and the multiple treatment time segments correspond one-to-one with the multiple treatment processes of the target patient;

[0011] The association analysis module is used to perform association analysis on the second data and the first data within each treatment time segment based on the multiple clusters, so as to obtain the association analysis results.

[0012] In one embodiment, the clustering module includes:

[0013] The related word determination unit is used to determine the related words of each keyword in each second data based on the occurrence position and frequency of each keyword in each second data. The related words are the keywords in the corresponding second data that are most relevant to the diagnosis and treatment information indicated by the corresponding keyword.

[0014] The word difference analysis unit is used to analyze the degree of difference between each keyword in each second data and its corresponding similar keywords based on each keyword, related words of the keyword, and word weight of the keyword, so as to obtain multiple word difference indices for each keyword in each second data. The similar keywords are the keywords with the highest similarity to the corresponding keywords in the corresponding second data. Each keyword and its corresponding similar keywords are located in different second data. The word weight is used to represent the importance of the corresponding keyword in the corresponding second data.

[0015] Clustering units are used to cluster multiple sets of data based on the word difference indices corresponding to each keyword in each set of data, resulting in multiple clusters.

[0016] In one embodiment, the related word determination unit is specifically used for:

[0017] Calculate the absolute value of the difference between the frequency of the first keyword and the second keyword in the target second data to obtain the word frequency difference value, wherein the target second data is any one of the plurality of second data, and the first keyword and the second keyword are any two different keywords in the target second data;

[0018] Calculate the minimum position difference between each occurrence position of the first keyword and multiple occurrence positions of the second keyword to obtain multiple minimum word distances, wherein the multiple minimum word distances correspond one-to-one with the multiple occurrence positions of the first keyword;

[0019] Calculate the mean of multiple minimum word distances to obtain the positional difference;

[0020] The reciprocal of the product of the word frequency difference and the position difference is calculated to obtain the diagnosis and treatment relevance, which is used to represent the degree of correlation between the diagnosis and treatment information indicated by the first keyword and the diagnosis and treatment information indicated by the second keyword.

[0021] Among them, the related words of the first keyword are: the keywords with the highest diagnostic and treatment relevance to the first keyword among the multiple keywords included in the target second data.

[0022] In one embodiment, the word difference analysis unit is specifically used for:

[0023] Obtain multiple similar keywords corresponding to the target keyword, wherein the target keyword is any keyword in any second set of data;

[0024] The cosine similarity between the first word vector and multiple second word vectors is calculated to obtain multiple first similarities of the target keyword. The first word vector is the word vector of the target keyword, and the multiple second word vectors correspond one-to-one with multiple similar keywords associated with the target keyword. The second word vector is the word vector of the corresponding similar keyword.

[0025] The cosine similarity between the third word vector and multiple fourth word vectors is calculated to obtain multiple principal similarities of the target keyword and multiple second similarities corresponding to the target keyword. The third word vector is the word vector of the related words of the target keyword, and the multiple fourth word vectors correspond one-to-one with multiple similar keywords associated with the target keyword. The fourth word vector is the word vector of the related words of the corresponding similar keywords.

[0026] Calculate the product of each first similarity and the corresponding second similarity of the target keyword to obtain multiple word similarity parameters of the target keyword. The multiple word similarity parameters correspond one-to-one with the multiple similar keywords associated with the target keyword.

[0027] Calculate the word weight of the target keyword and the ratio of multiple word similarity parameters of the target keyword to obtain the word difference index corresponding to the target keyword.

[0028] In one embodiment, the clustering unit is specifically used for:

[0029] Aggregate the multiple word difference indices corresponding to each keyword in each second data to obtain a set of multiple difference indices for each second data. The multiple difference indices correspond one-to-one with the multiple remaining second data except the corresponding second data. The multiple word difference indices of each keyword correspond one-to-one with the set of multiple difference indices of the second data in which the keyword is located.

[0030] Calculate the sum of multiple word difference indices included in each difference index set of each second data point to obtain the distance between multiple data points corresponding to each second data point;

[0031] Based on the multiple data distances corresponding to each second data point, multiple second data points are clustered to obtain multiple clusters.

[0032] In one embodiment, the time series analysis module includes:

[0033] The disorder feature analysis unit is used to determine the examination disorder feature value for each treatment date in the treatment time period based on multiple clusters and the number of second data corresponding to each treatment date in the treatment time period.

[0034] The initial consultation date determination unit is used to analyze the examination disorder characteristic values ​​of each treatment date in the treatment period in order to determine multiple initial consultation dates in the treatment period.

[0035] The time segmentation unit is used to segment the treatment time period based on the multiple initial diagnosis dates to obtain the multiple diagnosis and treatment time segments.

[0036] In one embodiment, the disorder feature analysis unit is specifically used for:

[0037] Calculate the ratio of the number of at least one cluster associated with each treatment date to the total number of multiple clusters to obtain the cluster percentage corresponding to each treatment date;

[0038] Calculate the product of the amount of data in the second data corresponding to each treatment date and the proportion of the cluster corresponding to that treatment quality to obtain the examination disorder feature value for each treatment date.

[0039] In one embodiment, the initial diagnosis date determination unit is specifically used for:

[0040] Using the disordered feature values ​​as the sorting comparison index, the disordered feature values ​​of multiple examinations for multiple treatment dates included in the treatment time period are sorted to obtain the target sequence;

[0041] Calculate the absolute value of the difference between adjacent sequence elements in the target sequence to obtain multiple element differences;

[0042] The largest element difference among multiple element differences is determined as the target element difference;

[0043] Among the two adjacent sequence elements corresponding to the difference of the target element, the sequence element with the smaller disorder feature value is identified as the boundary sequence element;

[0044] In the target sequence, the sequence elements whose disorder feature values ​​are greater than those of the boundary sequence elements are identified as initial diagnostic elements;

[0045] The treatment date corresponding to each initial diagnosis element is determined as the initial diagnosis date, so that multiple initial diagnosis dates are obtained.

[0046] In one embodiment, the time segmentation unit is specifically used for:

[0047] Multiple treatment dates between adjacent initial consultation dates are identified as multiple dates to be screened;

[0048] Among multiple clusters, the key cluster for each date to be identified is determined, wherein the key cluster is the cluster with the largest number of corresponding second data in the date to be identified;

[0049] Calculate the ratio of the number of keyword characters to the total number of characters for each second data point in the corresponding date to be identified for each key cluster, so as to obtain multiple keyword densities for each key cluster in the date to be identified;

[0050] Calculate the average keyword density of multiple keyword clusters for each date to be identified, so as to obtain the average word density of the key clusters for each date to be identified;

[0051] Calculate the ratio of the average word density to the average total number of characters for each key cluster of the date to be identified, in order to obtain the re-examination information density for each date to be identified;

[0052] The clinical event index of follow-up visits for each date to be screened is obtained by multiplying the follow-up information density of each date to be screened by the number of second data included in the corresponding key cluster on that date.

[0053] Among multiple dates to be identified, the dates with a clinical event index greater than or equal to the event index threshold are identified as the follow-up dates, so as to obtain multiple follow-up dates;

[0054] The treatment time period is divided based on the multiple initial consultation dates and the multiple follow-up consultation dates to obtain multiple treatment time segments.

[0055] In one embodiment, the association analysis module is specifically used for:

[0056] Calculate the product of the cumulative number of recorded days corresponding to each cluster in the target treatment time segment and the total number of included second data to obtain the cluster description of each cluster in the target treatment time segment, wherein the target treatment time segment is any one of multiple treatment time segments;

[0057] Calculate the variance of the number of second data included in each cluster in the target treatment time segment on other days besides the first day of the target treatment time segment, so as to obtain the cluster number variance of each cluster in the target treatment time segment;

[0058] Calculate the ratio of the cluster descriptor to the variance of the number of clusters for each cluster in the target diagnosis and treatment time segment to obtain the cluster feature value for each cluster in the target diagnosis and treatment time segment;

[0059] The cluster with the largest cluster feature value in the target diagnosis and treatment time segment is identified as the target disease description cluster;

[0060] In the target disease description cluster, among the multiple second data included in the target diagnosis and treatment time segment, the average number of multiple non-keyword characters included in each second data is calculated to obtain the disease development description quantity of each second data in the target disease description cluster. The number of non-keyword characters is the total number of other characters in the sentence to which the corresponding keyword belongs, excluding the corresponding keyword.

[0061] Acquire multiple daily slope data of the target first data within the target treatment time segment, wherein the target first data is any one of the multiple first data located in the target treatment time segment;

[0062] In the target disease description cluster, among the multiple second data included in the target diagnosis and treatment time segment, the absolute value of the difference between the disease development description of each second data and the adjacent daily slope data of the target first data is calculated to obtain the data difference value of each second data in the target disease description cluster. The adjacent daily slope data is the daily slope data with the shortest time difference between the indicated date and the date of the corresponding second data among the multiple daily slope data of the target first data.

[0063] Based on the data difference value of each second data in the target disease description cluster, a target association value is determined, wherein the association analysis result includes the target association value, which is used to represent the degree of data association between the target disease description cluster and the target first data.

[0064] Secondly, another embodiment of the present invention provides a method for intelligent correlation analysis of multimodal medical data based on deep learning, the method comprising:

[0065] Acquire multiple first data and multiple second data of the target patient, wherein the first data is structured medical data and the second data is unstructured medical data;

[0066] Extract keywords from each second data point to obtain multiple keywords for each second data point;

[0067] Based on multiple keywords for each second data point, multiple second data points are clustered to obtain multiple clusters;

[0068] Based on multiple clusters, time-series analysis is performed on the treatment time periods corresponding to multiple second data to determine multiple diagnosis and treatment time segments, and the multiple diagnosis and treatment time segments correspond one-to-one with the multiple diagnosis and treatment processes of the target patient;

[0069] Based on the multiple clusters, a correlation analysis is performed on the second data and the first data within each treatment time segment to obtain the correlation analysis results.

[0070] Thirdly, in another embodiment of the present invention, an electronic device is provided, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described in the second aspect above.

[0071] Fourthly, in another embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in the second aspect above.

[0072] The present invention has the following beneficial effects:

[0073] This invention, after acquiring multiple structured and unstructured medical data of a target patient, extracts keywords from each unstructured medical data point and clusters these data points based on these keywords to achieve initial differentiation. Then, it performs time-series analysis on the clusters to further determine multiple treatment time segments, i.e., the multiple treatment processes of the target patient. Finally, based on the clusters obtained from the aforementioned clustering operation, it analyzes the correlation between unstructured and structured medical data in each treatment process. This allows for the identification of unstructured medical data associated with structured medical data in each treatment process, as well as the degree of correlation between structured medical data and their corresponding unstructured medical data. This accurately achieves automatic correlation between structured and unstructured medical data, improving the effectiveness of correlation analysis for multimodal medical data. Attached Figure Description

[0074] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 A schematic diagram of the structure of a deep learning-based multimodal medical data intelligent correlation analysis system provided in one embodiment of the present invention;

[0076] Figure 2 This is a flowchart illustrating a method for intelligent correlation analysis of multimodal medical data based on deep learning, provided in one embodiment of the present invention.

[0077] Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0078] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a deep learning-based multimodal medical data intelligent correlation analysis system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0079] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0080] The following description, in conjunction with the accompanying drawings, details the specific solution of the deep learning-based multimodal medical data intelligent correlation analysis system provided by this invention.

[0081] This invention proposes a deep learning-based intelligent correlation analysis system for multimodal medical data. Please refer to [link / reference]. Figure 1 The diagram illustrates a structural schematic of a deep learning-based multimodal medical data intelligent correlation analysis system 100 according to an embodiment of the present invention. The system 100 includes the following modules:

[0082] The data acquisition module 101 is used to acquire multiple first data and multiple second data of the target patient.

[0083] The first data is structured medical data, and the second data is unstructured medical data.

[0084] The aforementioned target patient can be understood as any single patient, and the multiple first data points and multiple second data points constitute the multimodal medical data of the target cache.

[0085] In this invention, structured medical data can be understood as medical data with a specific data format (such as key-value pairs), for example, the various test indicators in a medical test report exist in the form of key-value pairs; while unstructured medical data can be understood as textual medical data, such as medical records, radiological imaging reports, clinical record texts, etc.

[0086] For example, with authorization from the organization to which the medical system belongs, the aforementioned multiple first data and multiple second data can be directly obtained from the medical system's database. Furthermore, to protect patient privacy, the identity data recorded in the database containing patient identity information (such as name, age, home address, etc.) can be mapped to characters or numbers without any semantic meaning. Then, several structured and unstructured medical data corresponding to the patient can be connected and fed into the multimodal medical data intelligent association analysis system described in this invention for association analysis.

[0087] The keyword extraction module 102 is used to extract keywords from each second data point to obtain multiple keywords for each second data point.

[0088] The keywords mentioned above are words in the corresponding second data (i.e., unstructured medical data) used to describe key information about the target patient's condition. For example, words indicating the type of disease of the target patient (such as pneumonia, hypertension, diabetes, etc.), words indicating key indicators of the target patient's condition (fasting blood glucose, blood pressure, etc.), words indicating key drugs used for the treatment of the target patient (such as insulin, metformin, etc.), etc.

[0089] By extracting keywords, we can focus on analyzing the semantic relevance between keywords in unstructured medical data and those in structured medical data, thereby reducing the interference of non-keywords on subsequent association analysis and enabling more accurate association analysis between unstructured and structured medical data.

[0090] In one example, the keyword extraction operation described above can be performed using a keyword extraction model, which can be obtained by fine-tuning and training the ClinicalBERT model.

[0091] Specifically, the process of obtaining the keyword extraction model can be as follows:

[0092] We collect a large amount of medical texts, such as medical records, radiology reports, and clinical records. For each medical text, we divide it into a training set and a validation set. We then manually or semi-automatically tag the keywords in each medical text and convert the keywords into BIO tag format.

[0093] Set the learning rate of the ClinicalBERT model to 2×10. -5 The batch size is 16, and the maximum number of training rounds is 3-5. The labeled medical texts in the training set are input into the ClinicalBERT model in batches. The loss between the predicted value output by the ClinicalBERT model and the real BIO label is calculated. The AdamW optimizer is used to continuously optimize the model parameters and minimize the loss value.

[0094] After each round of training, the model metrics (such as the F1-score for each keyword) are evaluated on the validation set to train the keyword extraction model.

[0095] The process of extracting multiple keywords for each second data point using a keyword extraction model can be described as follows:

[0096] Each second data point is input into the keyword extraction model for processing to obtain several BIO tags output by the keyword extraction model for each second data point. The consecutive B-KEYWORDs and I-KEYWORDs in the several BIO tags output by each second data point are combined to form keywords, thus obtaining multiple keywords for each second data point.

[0097] Clustering module 103 is used to cluster multiple second data based on multiple keywords of each second data to obtain multiple clusters.

[0098] In this step, before analyzing the correlation between unstructured and structured medical data, clustering is used to internally integrate and differentiate different unstructured medical data. This eliminates differences caused by factors such as expression methods and identifies the frequency of keyword occurrence in different unstructured medical data. Based on this, subsequent correlation analysis can be carried out, making the subsequent correlation analysis more accurate and effective.

[0099] Furthermore, the clustering module includes:

[0100] The related word determination unit is used to determine the related words of each keyword in each second data based on the occurrence position and frequency of each keyword in each second data. The related words are the keywords in the corresponding second data that are most relevant to the diagnosis and treatment information indicated by the corresponding keyword.

[0101] The word difference analysis unit is used to analyze the degree of difference between each keyword in each second data and its corresponding similar keywords based on each keyword, related words of the keyword, and word weight of the keyword, so as to obtain multiple word difference indices for each keyword in each second data. The similar keywords are the keywords with the highest similarity to the corresponding keywords in the corresponding second data. Each keyword and its corresponding similar keywords are located in different second data. The word weight is used to represent the importance of the corresponding keyword in the corresponding second data.

[0102] Clustering units are used to cluster multiple sets of data based on the word difference indices corresponding to each keyword in each set of data, resulting in multiple clusters.

[0103] For example, the word weights mentioned above can be the average of multiple attention weights corresponding to the keyword in the reasoning process of the aforementioned keyword extraction model.

[0104] Keywords and related terms can be used to indicate treatment pathways to a certain extent. For example, when "diabetes" and "insulin" appear simultaneously in unstructured medical data, it usually indicates a treatment pathway primarily focused on diabetes. However, in treatment pathways not primarily focused on diabetes (such as when "diabetes" is described as a complication), descriptions of "insulin" usually do not appear. In other words, by identifying related terms, the co-occurrence features between different keywords can be extracted to a certain extent. The treatment pathways indicated by the corresponding unstructured medical data can be revealed through the synergy of keywords and related terms, which facilitates more accurate clustering of different unstructured medical data.

[0105] The introduction of word weights aims to analyze the differences in treatment paths indicated by different keywords among different unstructured medical data, while further considering the importance of each keyword in the corresponding unstructured medical data. This allows for the assignment of higher computational weights to keywords with greater importance during the assessment of semantic distance between different unstructured medical data, thereby further improving the accuracy of the final clustering results.

[0106] In this context, the similar keywords corresponding to each keyword can be understood as: among the multiple keywords in the corresponding unstructured medical data (different from the unstructured medical data in which the keyword is located), the keyword with the highest degree of similarity to the keyword.

[0107] The similarity between keywords can be calculated in the following ways:

[0108] Extract the word vectors of the two keywords respectively (word vectors can be extracted using the Word2Vec tool), and calculate the cosine similarity between the two word vectors, thus using it as a quantitative value of the similarity between the two keywords.

[0109] Furthermore, the related word determination unit is specifically used for:

[0110] Calculate the absolute value of the difference between the frequency of the first keyword and the second keyword in the target second data to obtain the word frequency difference value, wherein the target second data is any one of the plurality of second data, and the first keyword and the second keyword are any two different keywords in the target second data;

[0111] Calculate the minimum position difference between each occurrence position of the first keyword and multiple occurrence positions of the second keyword to obtain multiple minimum word distances, wherein the multiple minimum word distances correspond one-to-one with the multiple occurrence positions of the first keyword;

[0112] Calculate the mean of multiple minimum word distances to obtain the positional difference;

[0113] The reciprocal of the product of the word frequency difference and the position difference is calculated to obtain the diagnosis and treatment relevance, which is used to represent the degree of correlation between the diagnosis and treatment information indicated by the first keyword and the diagnosis and treatment information indicated by the second keyword.

[0114] Among them, the related words of the first keyword are: the keywords with the highest diagnostic and treatment relevance to the first keyword among the multiple keywords included in the target second data.

[0115] In the above settings, the differences in the frequency and position of occurrence of different keywords in the same unstructured medical data are analyzed to accurately indicate the degree of correlation between the diagnostic and treatment information indicated by the two keywords. The closer the frequency of occurrence of the two keywords, the greater the probability that they appear together and indicate the same diagnostic and treatment path, and the higher the diagnostic and treatment correlation. Similarly, the closer the frequency of occurrence of the two keywords, the greater the probability that they appear adjacently and indicate the same diagnostic and treatment path, and the higher the diagnostic and treatment correlation, and vice versa.

[0116] For example, if we set the first keyword to appear at a certain position as 10, and the second keyword appears at multiple positions as 5, 50, and 102, then the minimum word distance corresponding to the first keyword at that position is 5 (minimum word distance 5 = first keyword appearance position 10 - second keyword nearest appearance position 5).

[0117] In one example, the diagnostic relevance T between the i-th keyword and the j-th keyword in unstructured medical data is... ij It can be represented as:

[0118]

[0119] Where i is any positive integer less than or equal to the total number of keywords included in the current unstructured medical data, j is any positive integer less than or equal to the total number of keywords included in the current unstructured medical data, and n j This represents the frequency of the j-th keyword in the current unstructured medical data, where n is the frequency of occurrence. i This indicates the frequency of the i-th keyword in the current unstructured medical data. This represents the average of multiple minimum word distances between the i-th keyword and the j-th keyword, and c represents the adjustment coefficient, which is equal to 1 to prevent the denominator of the above formula from being zero.

[0120] Furthermore, the word difference analysis unit is specifically used for:

[0121] Obtain multiple similar keywords corresponding to the target keyword, wherein the target keyword is any keyword in any second set of data;

[0122] The cosine similarity between the first word vector and multiple second word vectors is calculated to obtain multiple first similarities of the target keyword. The first word vector is the word vector of the target keyword, and the multiple second word vectors correspond one-to-one with multiple similar keywords associated with the target keyword. The second word vector is the word vector of the corresponding similar keyword.

[0123] The cosine similarity between the third word vector and multiple fourth word vectors is calculated to obtain multiple principal similarities of the target keyword and multiple second similarities corresponding to the target keyword. The third word vector is the word vector of the related words of the target keyword, and the multiple fourth word vectors correspond one-to-one with multiple similar keywords associated with the target keyword. The fourth word vector is the word vector of the related words of the corresponding similar keywords.

[0124] Calculate the product of each first similarity and the corresponding second similarity of the target keyword to obtain multiple word similarity parameters of the target keyword. The multiple word similarity parameters correspond one-to-one with the multiple similar keywords associated with the target keyword.

[0125] Calculate the word weight of the target keyword and the ratio of multiple word similarity parameters of the target keyword to obtain the word difference index corresponding to the target keyword.

[0126] Based on the above settings, the similarity of word vectors between each keyword and its similar keywords, the similarity of word vectors between related words of each keyword and related words of its similar keywords, and the word weight of each keyword are calculated to evaluate the degree of difference between the two most similar keywords between two different unstructured medical data from multiple perspectives, thereby ensuring the accuracy of the calculated word difference index.

[0127] Furthermore, the clustering unit is specifically used for:

[0128] Aggregate the multiple word difference indices corresponding to each keyword in each second data to obtain a set of multiple difference indices for each second data. The multiple difference indices correspond one-to-one with the multiple remaining second data except the corresponding second data. The multiple word difference indices of each keyword correspond one-to-one with the set of multiple difference indices of the second data in which the keyword is located.

[0129] Calculate the sum of multiple word difference indices included in each difference index set of each second data point to obtain the distance between multiple data points corresponding to each second data point;

[0130] Based on the multiple data distances corresponding to each second data point, multiple second data points are clustered to obtain multiple clusters.

[0131] The smaller the data distance, the higher the similarity between the two corresponding second data points, which means that the medical information described by the two corresponding second data points is more similar.

[0132] In one example, the data distance S(a, b) between unstructured medical data description text a and another unstructured medical data description text b can be expressed as:

[0133]

[0134] Where, m a α represents the total number of keywords in the unstructured medical data description text 'a'. qn Let α be the cosine similarity (i.e., the aforementioned first similarity) between the nth keyword in the unstructured medical data description text 'a' and the corresponding similar keywords in the unstructured medical data description text 'b'. pn The cosine similarity (also known as the second similarity) of word vectors between related words of the nth keyword in unstructured medical data description text a and related words of similar keywords in unstructured medical data description text b. This represents the word weight of the nth keyword in the unstructured medical data description text a, and norm(.) represents the normalization function.

[0135] In applications, the distance between multiple data points corresponding to each second data point can be used as a metric to measure the distance between different second data points. Multiple second data points can be clustered using the K-means clustering algorithm to obtain multiple clusters. The clustering parameter K is obtained using the elbow method.

[0136] The time series analysis module 104 is used to perform time series analysis on the treatment time periods corresponding to multiple second data based on multiple clusters, so as to determine multiple diagnosis and treatment time segments.

[0137] The multiple treatment time segments correspond one-to-one with the multiple treatment processes of the target patient.

[0138] Because the primary focus of target patients may differ in different treatment processes (e.g., the initial consultation focuses on exploring the patient's specific symptoms, while the follow-up consultation focuses on exploring changes in the patient's symptoms), and there is a strong correlation between unstructured and structured medical data generated based on the corresponding focus in each treatment process, identifying multiple treatment time segments within the treatment period helps to accurately locate multiple treatment processes. Based on this, it is possible to more accurately identify the unstructured and structured medical data generated based on the corresponding focus in each treatment process, thereby ensuring accurate correlation between unstructured and structured medical data.

[0139] Furthermore, the time series analysis module includes:

[0140] The disorder feature analysis unit is used to determine the examination disorder feature value for each treatment date in the treatment time period based on multiple clusters and the number of second data corresponding to each treatment date in the treatment time period.

[0141] The initial consultation date determination unit is used to analyze the examination disorder characteristic values ​​of each treatment date in the treatment period in order to determine multiple initial consultation dates in the treatment period.

[0142] The time segmentation unit is used to segment the treatment time period based on the multiple initial diagnosis dates to obtain the multiple diagnosis and treatment time segments.

[0143] Among them, the richness of the types of unstructured medical data for each treatment date can be determined based on multiple clusters (which can also be understood as the richness of the treatment direction). Specifically, the more clusters that the multiple second data included in each treatment date correspond to in multiple clusters, the higher the richness of the types of unstructured medical data for that treatment date (which can also be understood as the richness of the treatment direction). The more it matches the situation of the target patient undergoing multiple examinations in the initial stage of a treatment process (used to determine the target patient's condition), the higher the probability that the treatment date belongs to the start date of the target patient in the corresponding treatment process (i.e., the initial consultation date).

[0144] Similarly, the number of second data points corresponding to each treatment date indicates that the more data items in the unstructured medical data for that treatment date, the better it matches the multiple examinations performed by the target patient in the initial treatment stage of a single treatment process. The higher the probability that the treatment date belongs to the start date (i.e., the initial consultation date) of the target patient in the corresponding treatment process, the higher the probability that the treatment date belongs to the start date (i.e., the initial consultation date) of the target patient.

[0145] Specifically, the disorder feature analysis unit is used for:

[0146] Calculate the ratio of the number of at least one cluster associated with each treatment date to the total number of multiple clusters to obtain the cluster percentage corresponding to each treatment date;

[0147] Calculate the product of the amount of data in the second data corresponding to each treatment date and the proportion of the cluster corresponding to that treatment quality to obtain the examination disorder feature value for each treatment date.

[0148] Among them, the second data generated within the treatment date is the second data corresponding to the treatment date, the cluster to which the second data corresponding to the treatment date belongs is the cluster associated with the treatment date, and the data volume of the second data corresponding to the treatment date is the total number of multiple second data generated within the treatment date.

[0149] The larger the disorder eigenvalue, the more secondary data the target patient generates within the corresponding treatment date, and the more diverse the indicated treatment directions. This is more consistent with the data characteristics of medical data generated by the patient in the initial stage of treatment, and the more suitable the corresponding treatment date is as a clinical event anchor.

[0150] Based on the above settings, the determined initial consultation date can be more accurate and reliable.

[0151] Specifically, the initial diagnosis date determination unit is used for:

[0152] Using the disordered feature values ​​as the sorting comparison index, the disordered feature values ​​of multiple examinations for multiple treatment dates included in the treatment time period are sorted to obtain the target sequence;

[0153] Calculate the absolute value of the difference between adjacent sequence elements in the target sequence to obtain multiple element differences;

[0154] The largest element difference among multiple element differences is determined as the target element difference;

[0155] Among the two adjacent sequence elements corresponding to the difference of the target element, the sequence element with the smaller disorder feature value is identified as the boundary sequence element;

[0156] In the target sequence, the sequence elements whose disorder feature values ​​are greater than those of the boundary sequence elements are identified as initial diagnostic elements;

[0157] The treatment date corresponding to each initial diagnosis element is determined as the initial diagnosis date, so that multiple initial diagnosis dates are obtained.

[0158] In the above setup, after determining the examination disorder feature value for each treatment date, the examination disorder feature values ​​for multiple treatment dates are sorted by size to find the two elements with the largest difference in examination disorder feature values ​​between adjacent elements. The element with the larger examination disorder feature value among the two found elements is determined as the boundary sequence element, and multiple initial diagnosis elements are determined accordingly, thereby determining multiple initial diagnosis dates. This can adapt to situations where the examination disorder feature value corresponding to the initial diagnosis date is significantly higher than that of non-initial diagnosis dates, and minimizes possible human interference in the determination of the initial diagnosis date, making the determined multiple initial diagnosis dates more accurate.

[0159] In the target sequence mentioned above, multiple sequence elements can be arranged in ascending (or descending) order of the disordered feature values.

[0160] Furthermore, the time segmentation unit is specifically used for:

[0161] Multiple treatment dates between adjacent initial consultation dates are identified as multiple dates to be screened;

[0162] Among multiple clusters, the key cluster for each date to be identified is determined, wherein the key cluster is the cluster with the largest number of corresponding second data in the date to be identified;

[0163] Calculate the ratio of the number of keyword characters to the total number of characters for each second data point in the corresponding date to be identified for each key cluster, so as to obtain multiple keyword densities for each key cluster in the date to be identified;

[0164] Calculate the average keyword density of multiple keyword clusters for each date to be identified, so as to obtain the average word density of the key clusters for each date to be identified;

[0165] Calculate the ratio of the average word density to the average total number of characters for each key cluster of the date to be identified, in order to obtain the re-examination information density for each date to be identified;

[0166] The clinical event index of follow-up visits for each date to be screened is obtained by multiplying the follow-up information density of each date to be screened by the number of second data included in the corresponding key cluster on that date.

[0167] Among multiple dates to be identified, the dates with a clinical event index greater than or equal to the event index threshold are identified as the follow-up dates, so as to obtain multiple follow-up dates;

[0168] The treatment time period is divided based on the multiple initial consultation dates and the multiple follow-up consultation dates to obtain multiple treatment time segments.

[0169] The average total number of characters in each key cluster of dates to be identified is the average total number of characters in multiple second data sets of that key cluster.

[0170] In the above settings, in addition to determining the initial consultation date for each treatment process, the follow-up consultation date for each treatment process will also be determined, so as to complete a more refined time segmentation and make the subsequent correlation between unstructured medical data and structured medical data more accurate.

[0171] For example, the process of segmenting the treatment time period based on the multiple initial consultation dates and the multiple follow-up consultation dates to obtain multiple treatment time segments can be as follows:

[0172] Using the multiple initial consultation dates and the multiple follow-up consultation dates as time anchors, the time segments between adjacent time anchors within the treatment period are divided to obtain multiple treatment time segments.

[0173] During the initial consultation process (i.e., the treatment process indicated by the initial consultation date), patients usually need to undergo a wide variety of medical examinations to determine the specific symptoms. In the follow-up consultation stage (i.e., the treatment process indicated by the follow-up consultation date), since the patient's condition has been preliminarily determined, the focus will be on the dynamic tracking of specific diseases or symptoms. This will significantly increase the keyword density in unstructured medical data.

[0174] Based on the above settings, the key clusters for each date to be identified are first determined, that is, the main unstructured medical data types corresponding to each date to be identified are determined. Then, the keyword density of the key clusters for each date to be identified is analyzed to adapt to the high keyword density characteristics in the follow-up consultation process. Combined with the number of secondary data included in the key clusters for each date to be identified, the follow-up consultation clinical event index for each date to be identified is comprehensively evaluated, that is, the probability that each date to be identified belongs to the follow-up consultation date is evaluated, which can make the determined follow-up consultation date more accurate.

[0175] In one example, the event index threshold mentioned above can be set to 0.45.

[0176] The association analysis module 105 is used to perform association analysis on the second data and the first data within each treatment time segment based on the multiple clusters, so as to obtain the association analysis results.

[0177] Specifically, the correlation analysis module is used for:

[0178] Calculate the product of the cumulative number of recorded days corresponding to each cluster in the target treatment time segment and the total number of included second data to obtain the cluster description of each cluster in the target treatment time segment, wherein the target treatment time segment is any one of multiple treatment time segments;

[0179] Calculate the variance of the number of second data included in each cluster in the target treatment time segment on other days besides the first day of the target treatment time segment, so as to obtain the cluster number variance of each cluster in the target treatment time segment;

[0180] Calculate the ratio of the cluster descriptor to the variance of the number of clusters for each cluster in the target diagnosis and treatment time segment to obtain the cluster feature value for each cluster in the target diagnosis and treatment time segment;

[0181] The cluster with the largest cluster feature value in the target diagnosis and treatment time segment is identified as the target disease description cluster;

[0182] In the target disease description cluster, among the multiple second data included in the target diagnosis and treatment time segment, the average number of multiple non-keyword characters included in each second data is calculated to obtain the disease development description quantity of each second data in the target disease description cluster. The number of non-keyword characters is the total number of other characters in the sentence to which the corresponding keyword belongs, excluding the corresponding keyword.

[0183] Acquire multiple daily slope data of the target first data within the target treatment time segment, wherein the target first data is any one of the multiple first data located in the target treatment time segment;

[0184] In the target disease description cluster, among the multiple second data included in the target diagnosis and treatment time segment, the absolute value of the difference between the disease development description of each second data and the adjacent daily slope data of the target first data is calculated to obtain the data difference value of each second data in the target disease description cluster. The adjacent daily slope data is the daily slope data with the shortest time difference between the indicated date and the date of the corresponding second data among the multiple daily slope data of the target first data.

[0185] Based on the data difference value of each second data in the target disease description cluster, a target association value is determined, wherein the association analysis result includes the target association value, which is used to represent the degree of data association between the target disease description cluster and the target first data.

[0186] The cumulative number of recorded days and the number of texts (i.e., the total number of second data included) for each cluster in the target treatment time segment are used to obtain the cluster description quantity for each cluster in the target treatment time segment. This allows for a quantitative representation of the text richness of each cluster in the target treatment time segment. The higher the text richness, the greater the probability that the corresponding cluster indicates the main content that the target treatment time segment focuses on, meaning that the feasibility of selecting the corresponding cluster to associate with the first data is higher.

[0187] The variance of the number of clusters in each cluster within the target treatment time segment is used to quantify the fluctuation in the number of texts in the cluster. The smaller the variance of the number of clusters, the greater the probability that the corresponding cluster indicates the main content of interest in the target treatment time segment, that is, the higher the feasibility of selecting the corresponding cluster to associate with the first data.

[0188] By combining the comprehensive cluster description quantity and the variance of the cluster quantity, the disease description clusters corresponding to the content of interest in each treatment time segment can be selected, thereby accurately completing the association between unstructured and structured medical data.

[0189] In this invention, keywords are used to describe the main characteristics of the target patient's condition (such as specific symptoms, key medications), while non-keywords are other words in the corresponding second data besides keywords, which are used to represent the text richness / text complexity of the corresponding second data's description of the target patient's condition.

[0190] The higher the amount of disease progression description in the second data, the more complex the physical condition of the target patient is in the corresponding second data. In this case, the greater the probability that the target patient will show unstable signs during the examination, and vice versa.

[0191] As for obtaining multiple daily slope data of the target primary data within the target treatment time segment, it is to quantify the fluctuation of multiple data of the target primary data on multiple treatment dates within the target treatment time segment. The larger the daily slope data, the more unstable the physical signs of the target patient in the corresponding examination on the corresponding date, and vice versa.

[0192] For example, the process of obtaining multiple daily slope data of the target first data within the target treatment time segment can be as follows:

[0193] Curve fitting is performed based on multiple data values ​​of the target first data within the target treatment time segment (e.g., curve fitting is performed using the least squares method) to obtain the fitted curve of the target first data within the target treatment time segment.

[0194] Based on the fitted curve of the target first data within the target treatment time segment, the slope of the curve for multiple data values ​​within the target treatment time segment is calculated to obtain multiple daily slope data of the target first data within the target treatment time segment.

[0195] By calculating the absolute value of the difference between the disease development description value and the slope value of adjacent days in each of the multiple second data included in the target disease description cluster, the degree of difference between each second data and the target first data can be accurately quantified.

[0196] For example, the process of determining the target association value based on the data difference value of each second data in the target disease description cluster can be as follows:

[0197] Calculate the mean of the data difference values ​​for each second data point in the target disease description cluster to obtain the difference index between the target disease description cluster and the target first data point;

[0198] The difference index between the target disease description cluster and the target first data is normalized to obtain the difference normalization index between the target disease description cluster and the target first data.

[0199] The target association value can be obtained by subtracting the difference normalization index between the target disease description cluster and the target first data from 1.

[0200] The larger the target correlation value, the greater the correlation (also known as the degree of correlation) between the target disease description cluster and the target first data, and vice versa.

[0201] In summary, after acquiring multiple structured and unstructured medical data of a target patient, this invention extracts keywords from each unstructured medical data point and clusters these data points based on these keywords to achieve initial differentiation. Then, time-series analysis is performed on the clusters to further determine multiple treatment time segments, i.e., the multiple treatment processes of the target patient. Finally, based on the clusters obtained from the aforementioned clustering operation, the association between unstructured and structured medical data in each treatment process is analyzed. This allows for the identification of unstructured medical data associated with structured medical data in each treatment process, as well as the degree of association between structured medical data and their corresponding unstructured medical data. This accurately achieves automatic association between structured and unstructured medical data, improving the effectiveness of association analysis for multimodal medical data.

[0202] It should be noted that the system provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above.

[0203] This invention proposes a deep learning-based intelligent correlation analysis method for multimodal medical data. Please refer to [link / reference]. Figure 2 The diagram illustrates a flowchart of a deep learning-based intelligent correlation analysis method for multimodal medical data, provided by an embodiment of the present invention. The method includes:

[0204] Step S1: Obtain multiple first data points and multiple second data points for the target patient.

[0205] The first data is structured medical data, and the second data is unstructured medical data.

[0206] Step S2: Extract keywords from each second data point to obtain multiple keywords for each second data point.

[0207] Step S3: Based on the multiple keywords of each second data point, cluster the multiple second data points to obtain multiple clusters.

[0208] Step S4: Perform time series analysis on the treatment time periods corresponding to multiple second data based on multiple clusters to determine multiple diagnosis and treatment time segments.

[0209] The multiple treatment time segments correspond one-to-one with the multiple treatment processes of the target patient.

[0210] Step S5: Based on the multiple clusters, perform correlation analysis on the second data and the first data within each treatment time segment to obtain the correlation analysis results.

[0211] The above embodiments of the intelligent association analysis method for multimodal medical data based on deep learning and the embodiment of the intelligent association analysis system for multimodal medical data based on deep learning belong to the same concept. The specific implementation process is detailed in the system embodiment and will not be repeated here.

[0212] This invention also provides an electronic device. Please refer to [link to relevant documentation]. Figure 3 The electronic device may include a processor 301, a memory 302, and a program 3021 stored in the memory 302 and capable of running on the processor 301.

[0213] When program 3021 is executed by processor 301, it can achieve the following: Figure 2 Any steps in the corresponding method embodiments and the achievement of the same beneficial effects will not be repeated here.

[0214] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by hardware related to program instructions, and the program can be stored in a readable medium.

[0215] This invention also provides a readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described functions. Figure 2 Any step in the corresponding method embodiment can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0216] The computer-readable storage medium of this invention can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0217] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0218] The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0219] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or terminal. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0220] This invention also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to achieve the deep learning-based multimodal medical data intelligent correlation analysis method provided in the above embodiments.

[0221] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0222] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A deep learning-based intelligent correlation analysis system for multimodal medical data, characterized in that, The system includes: The data acquisition module is used to acquire multiple first data and multiple second data of the target patient, wherein the first data is structured medical data and the second data is unstructured medical data; The keyword extraction module is used to extract keywords from each second data point, resulting in multiple keywords for each second data point. The clustering module is used to cluster multiple second data points based on multiple keywords of each second data point, resulting in multiple clusters. The time series analysis module is used to perform time series analysis on the treatment time periods corresponding to multiple second data based on multiple clusters, so as to determine multiple treatment time segments, and the multiple treatment time segments correspond one-to-one with the multiple treatment processes of the target patient; The association analysis module is used to perform association analysis on the second data and the first data within each treatment time segment based on the multiple clusters, so as to obtain the association analysis results; The clustering module includes: The related word determination unit is used to determine the related words of each keyword in each second data based on the occurrence position and frequency of each keyword in each second data. The related words are the keywords in the corresponding second data that are most relevant to the diagnosis and treatment information indicated by the corresponding keyword. The word difference analysis unit is used to analyze the degree of difference between each keyword in each second data and its corresponding similar keywords based on each keyword, related words of the keyword, and word weight of the keyword, so as to obtain multiple word difference indices for each keyword in each second data. The similar keywords are the keywords with the highest similarity to the corresponding keywords in the corresponding second data. Each keyword and its corresponding similar keywords are located in different second data. The word weight is used to represent the importance of the corresponding keyword in the corresponding second data. Clustering units are used to cluster multiple sets of secondary data based on multiple word difference indices corresponding to each keyword in each set of secondary data, thereby obtaining multiple clusters. The time series analysis module includes: The disorder feature analysis unit is used to determine the examination disorder feature value for each treatment date in the treatment time period based on multiple clusters and the number of second data corresponding to each treatment date in the treatment time period. The initial consultation date determination unit is used to analyze the examination disorder characteristic values ​​of each treatment date in the treatment period in order to determine multiple initial consultation dates in the treatment period. A time segmentation unit is used to segment the treatment time period based on the multiple initial diagnosis dates to obtain the multiple diagnosis and treatment time segments; The correlation analysis module is specifically used for: Calculate the product of the cumulative number of recorded days corresponding to each cluster in the target treatment time segment and the total number of included second data to obtain the cluster description of each cluster in the target treatment time segment, wherein the target treatment time segment is any one of multiple treatment time segments; Calculate the variance of the number of second data included in each cluster in the target treatment time segment on other days besides the first day of the target treatment time segment, so as to obtain the cluster number variance of each cluster in the target treatment time segment; Calculate the ratio of the cluster descriptor to the variance of the number of clusters for each cluster in the target diagnosis and treatment time segment to obtain the cluster feature value for each cluster in the target diagnosis and treatment time segment; The cluster with the largest cluster feature value in the target diagnosis and treatment time segment is identified as the target disease description cluster; In the target disease description cluster, among the multiple second data included in the target diagnosis and treatment time segment, the average number of multiple non-keyword characters included in each second data is calculated to obtain the disease development description quantity of each second data in the target disease description cluster. The number of non-keyword characters is the total number of other characters in the sentence to which the corresponding keyword belongs, excluding the corresponding keyword. Acquire multiple daily slope data of the target first data within the target treatment time segment, wherein the target first data is any one of the multiple first data located in the target treatment time segment; In the target disease description cluster, among the multiple second data included in the target diagnosis and treatment time segment, the absolute value of the difference between the disease development description of each second data and the adjacent daily slope data of the target first data is calculated to obtain the data difference value of each second data in the target disease description cluster. The adjacent daily slope data is the daily slope data with the shortest time difference between the indicated date and the date of the corresponding second data among the multiple daily slope data of the target first data. Based on the data difference value of each second data in the target disease description cluster, a target association value is determined, wherein the association analysis result includes the target association value, which is used to represent the degree of data association between the target disease description cluster and the target first data.

2. The deep learning-based multimodal medical data intelligent correlation analysis system according to claim 1, characterized in that, The related word determination unit is specifically used for: Calculate the absolute value of the difference between the frequency of the first keyword and the second keyword in the target second data to obtain the word frequency difference value, wherein the target second data is any one of the plurality of second data, and the first keyword and the second keyword are any two different keywords in the target second data; Calculate the minimum position difference between each occurrence position of the first keyword and multiple occurrence positions of the second keyword to obtain multiple minimum word distances, wherein the multiple minimum word distances correspond one-to-one with the multiple occurrence positions of the first keyword; Calculate the mean of multiple minimum word distances to obtain the positional difference; The reciprocal of the product of the word frequency difference and the position difference is calculated to obtain the diagnosis and treatment relevance, which is used to represent the degree of correlation between the diagnosis and treatment information indicated by the first keyword and the diagnosis and treatment information indicated by the second keyword. Among them, the related words of the first keyword are: the keywords with the highest diagnostic and treatment relevance to the first keyword among the multiple keywords included in the target second data.

3. The deep learning-based multimodal medical data intelligent correlation analysis system according to claim 1, characterized in that, The word difference analysis unit is specifically used for: Obtain multiple similar keywords corresponding to the target keyword, wherein the target keyword is any keyword in any second set of data; The cosine similarity between the first word vector and multiple second word vectors is calculated to obtain multiple first similarities of the target keyword. The first word vector is the word vector of the target keyword, and the multiple second word vectors correspond one-to-one with multiple similar keywords associated with the target keyword. The second word vector is the word vector of the corresponding similar keyword. The cosine similarity between the third word vector and multiple fourth word vectors is calculated to obtain multiple principal similarities of the target keyword and multiple second similarities corresponding to the target keyword. The third word vector is the word vector of the related words of the target keyword, and the multiple fourth word vectors correspond one-to-one with multiple similar keywords associated with the target keyword. The fourth word vector is the word vector of the related words of the corresponding similar keywords. Calculate the product of each first similarity and the corresponding second similarity of the target keyword to obtain multiple word similarity parameters of the target keyword. The multiple word similarity parameters correspond one-to-one with the multiple similar keywords associated with the target keyword. Calculate the word weight of the target keyword and the ratio of multiple word similarity parameters of the target keyword to obtain the word difference index corresponding to the target keyword.

4. The deep learning-based multimodal medical data intelligent correlation analysis system according to claim 1, characterized in that, The clustering unit is specifically used for: Aggregate the multiple word difference indices corresponding to each keyword in each second data to obtain a set of multiple difference indices for each second data. The multiple difference indices correspond one-to-one with the multiple remaining second data except the corresponding second data. The multiple word difference indices of each keyword correspond one-to-one with the set of multiple difference indices of the second data in which the keyword is located. Calculate the sum of multiple word difference indices included in each difference index set of each second data point to obtain the distance between multiple data points corresponding to each second data point; Based on the multiple data distances corresponding to each second data point, multiple second data points are clustered to obtain multiple clusters.

5. The deep learning-based multimodal medical data intelligent correlation analysis system according to claim 1, characterized in that, The disorder feature analysis unit is specifically used for: Calculate the ratio of the number of at least one cluster associated with each treatment date to the total number of multiple clusters to obtain the cluster percentage corresponding to each treatment date; Calculate the product of the amount of data in the second data corresponding to each treatment date and the proportion of the cluster corresponding to that treatment date to obtain the examination disorder feature value for each treatment date.

6. The deep learning-based multimodal medical data intelligent correlation analysis system according to claim 1, characterized in that, The initial diagnosis date determination unit is specifically used for: Using the disordered feature values ​​as the sorting comparison index, the disordered feature values ​​of multiple examinations for multiple treatment dates included in the treatment time period are sorted to obtain the target sequence; Calculate the absolute value of the difference between adjacent sequence elements in the target sequence to obtain multiple element differences; The largest element difference among multiple element differences is determined as the target element difference; Among the two adjacent sequence elements corresponding to the difference of the target element, the sequence element with the smaller disorder feature value is identified as the boundary sequence element; In the target sequence, the sequence elements whose disorder feature values ​​are greater than those of the boundary sequence elements are identified as initial diagnostic elements; The treatment date corresponding to each initial diagnosis element is determined as the initial diagnosis date, so that multiple initial diagnosis dates are obtained.

7. The deep learning-based multimodal medical data intelligent correlation analysis system according to claim 1, characterized in that, The time segmentation unit is specifically used for: Multiple treatment dates between adjacent initial consultation dates are identified as multiple dates to be screened; Among multiple clusters, the key cluster for each date to be identified is determined, wherein the key cluster is the cluster with the largest number of corresponding second data in the date to be identified; Calculate the ratio of the number of keyword characters to the total number of characters for each second data point in the corresponding date to be identified for each key cluster, so as to obtain multiple keyword densities for each key cluster in the date to be identified; Calculate the average keyword density of multiple keyword clusters for each date to be identified, so as to obtain the average word density of the key clusters for each date to be identified; Calculate the ratio of the average word density to the average total number of characters for each key cluster of the date to be identified, in order to obtain the re-examination information density for each date to be identified; The clinical event index of follow-up visits for each date to be screened is obtained by multiplying the follow-up information density of each date to be screened by the number of second data included in the corresponding key cluster on that date. Among multiple dates to be identified, the dates with a clinical event index greater than or equal to the event index threshold are identified as the follow-up dates, so as to obtain multiple follow-up dates; The treatment time period is divided based on the multiple initial consultation dates and the multiple follow-up consultation dates to obtain multiple treatment time segments.

Images