An inquiry information processing method and system based on big data analysis

Abstract

The present application relates to the field of data processing, and more particularly to a diagnosis information processing method and system based on big data analysis. The method comprises the following steps: first, obtaining a low-dimensional embedding vector containing symptom diagnosis associated information; then, constructing a structured covariance matrix by solving an optimization problem, wherein the optimization problem takes the sample covariance matrix as a data fidelity term and takes a graph Laplacian matrix derived from the embedding vector as a regularization term, thereby injecting medical knowledge into the matrix structure; finally, using principal components derived from the structured covariance matrix to reduce the dimensionality of high-dimensional diagnosis features. The present application can effectively retain the symptom combination information which is crucial for clinical diagnosis, avoid the loss of key features, and significantly improve the quality of the features after dimensionality reduction.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method and system for processing medical consultation information based on big data analysis. Background Technology

[0002] In the field of smart healthcare, in order to extract valuable clinical information from massive amounts of unstructured medical consultation texts in natural language form to support advanced data applications such as clustering of similar cases, assisted diagnosis, and epidemiological analysis, a necessary prerequisite is to transform this textual information into standardized structured data that computers can process. Currently, a common technical approach is as follows: First, methods such as term frequency-inverse document frequency are used to vectorize the medical consultation text into high-dimensional feature vectors; then, to solve the "curse of dimensionality" problem and improve computational efficiency, principal component analysis (PCA) is used to reduce the dimensionality of this high-dimensional vector.

[0003] However, when this classic data processing workflow is directly applied to the professional medical field, its inherent principles, based on pure statistical assumptions, expose two fundamental, interconnected flaws: the neglect of the value of symptom combinations. The essence of clinical diagnosis lies in the identification of syndromes, that is, the overall pattern presented by multiple symptom combinations. For example, the symptom combination of "fever, rash, and joint pain" has a diagnostic value far greater than the simple sum of the values ​​of its individual symptoms. Traditional PCA algorithms are based on second-order covariance matrices, and their mathematical structure can only capture linear relationships between pairwise variables, failing to effectively express and preserve the high-order, non-linear symptom combination information composed of multiple symptoms.

[0004] While existing technologies attempt to improve PCA by weighting individual high-value symptoms, this fails to address the problem of neglecting the value of symptom combinations. Therefore, a core technical challenge urgently needs to be addressed: how to design a computationally feasible and statistically robust novel dimensionality reduction method that can fully utilize prior knowledge in the medical field to identify and retain key symptom combination information, while avoiding the erroneous bias of traditional PCA methods towards high-frequency, low-value features. Summary of the Invention

[0005] To address the issue of preventing the destruction of symptom combination information by dimensionality reduction, this invention provides a method and system for processing consultation information based on big data analysis.

[0006] In a first aspect, the present invention provides a method for processing medical consultation information based on big data analysis, employing the following technical solution:

[0007] A method for processing medical consultation information based on big data analysis includes the following steps:

[0008] Collect and preprocess the text of the consultation information;

[0009] Obtain a low-dimensional embedding vector for each of multiple standardized symptoms, wherein the low-dimensional embedding vector implicitly contains diagnostic association information between its corresponding symptom and other symptoms;

[0010] A sample covariance matrix is ​​calculated based on the high-dimensional standardized feature vectors of multiple consultation records.

[0011] By solving an optimization problem that uses the sample covariance matrix as a data fidelity term and a graph Laplacian matrix derived from the low-dimensional embedding vector of the symptoms as a regularization term, a structured covariance matrix that reflects the value of symptom combination is constructed.

[0012] The high-dimensional standardized feature vectors are reduced in dimensionality using principal components derived from the structured covariance matrix to generate low-dimensional data that retains symptom information.

[0013] This invention fundamentally solves the problem of traditional dimensionality reduction methods losing key diagnostic information due to their reliance on data variance alone, by constructing a structured covariance matrix guided by prior medical knowledge. This method ensures that the dimensionality reduction process actively identifies and retains symptom combinations with high diagnostic value, rather than erroneously favoring high-frequency, low-value, non-specific descriptions. This generates low-dimensional features with clearer clinical significance and higher information content, laying a solid foundation for subsequent precise medical big data analysis.

[0014] Preferably, the low-dimensional embedding vector is learned by learning on a weighted symptom-diagnosis bipartite graph, the nodes of which include symptom nodes and disease diagnosis nodes.

[0015] Preferably, the method for obtaining the edge weights of the bipartite graph includes:

[0016] Edge weights are calculated using a point-by-point mutual information method.

[0017] This invention employs point-by-point mutual information to calculate weights, effectively measuring the strength of the association between symptoms and diagnoses, and giving reasonable weight to low-frequency but diagnostically significant key symptoms. This avoids the problem of underestimating rare but crucial symptom-disease pairs by simply using co-occurrence frequency, making the weight distribution of the atlas more consistent with clinical diagnostic logic and providing a more accurate quantitative basis for learning high-quality symptom embeddings.

[0018] Preferably, the learning process includes:

[0019] A random walk is performed on the bipartite graph to generate a sequence of nodes, and the sequence is trained using the Word2Vec model.

[0020] This invention, by performing random walks on the graph and combining them with the Word2Vec model, can efficiently transform complex, high-order topological information in the graph into low-dimensional, dense vector representations. This method cleverly avoids the combinatorial explosion and data sparsity problems encountered when directly mining high-order symptom combinations.

[0021] Preferably, the graph Laplacian matrix is ​​calculated based on a symptom similarity matrix, which is constructed by calculating the cosine similarity between the low-dimensional embedding vectors of any two symptoms.

[0022] Preferably, the method for obtaining the high-dimensional standardized feature vector of the consultation record includes:

[0023] The consultation information text is linked to a standardized symptom dictionary as medical entities, and then vectorized using a term frequency-inverse document frequency algorithm.

[0024] Preferably, the structured covariance matrix satisfies the following relationship:

[0025]

[0026] in, Represents the standard sample covariance matrix. Represents the covariance variable matrix. Denotes the square of the Frobenius norm. This represents the graph Laplacian matrix calculated based on the symptom similarity matrix. Represents the trace of a matrix. This represents the preset regularization coefficient. Indicates a positive semidefinite constraint. This represents the structured covariance matrix.

[0027] This invention provides a principled and adjustable fusion framework. The data fidelity term in the relational expression ensures that the final model does not deviate from the real data distribution, while the regularization term injects symptom combination knowledge into the model. The regularization coefficient allows researchers to flexibly adjust the weights of both based on data quality and the reliability of the knowledge base, thus enabling the model to possess both data-driven flexibility and knowledge-guided accuracy.

[0028] Preferably, the dimensionality reduction process includes:

[0029] The high-dimensional normalized feature vectors are projected onto a projection matrix consisting of the feature vectors corresponding to the largest eigenvalues ​​of the structured covariance matrix to obtain low-dimensional data.

[0030] This invention enables dimensionality reduction of data along the direction that best reflects the combination of symptoms. Since the projection matrix is ​​derived from a knowledge-guided structured covariance matrix, its principal components are no longer simply the direction of maximum data variance, but rather the direction jointly determined by variance and clinical structure. Therefore, the projected low-dimensional data retains the most valuable symptom information for downstream tasks to the greatest extent, achieving high-quality information compression.

[0031] Preferred options also include:

[0032] The low-dimensional data obtained after the dimensionality reduction process is then classified.

[0033] Secondly, the present invention provides a consultation information processing system based on big data analysis, which adopts the following technical solution:

[0034] A consultation information processing system based on big data analysis includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned consultation information processing method based on big data analysis is implemented.

[0035] By adopting the above technical solution, a computer program is generated from the above-mentioned method for processing consultation information based on big data analysis and stored in a memory so that it can be loaded and executed by a processor. In this way, a terminal device can be made based on the memory and the processor for convenient use.

[0036] The present invention has the following technical effects:

[0037] This invention fundamentally solves the problem of traditional dimensionality reduction methods losing key diagnostic information due to their reliance on data variance alone, by constructing a structured covariance matrix guided by prior medical knowledge. This method ensures that the dimensionality reduction process actively identifies and retains symptom combinations with high diagnostic value, rather than erroneously favoring high-frequency, low-value, non-specific descriptions. This generates low-dimensional features with clearer clinical significance and higher information content, laying a solid foundation for subsequent precise medical big data analysis. Attached Figure Description

[0038] Figure 1 This is a flowchart of a method for processing consultation information based on big data analysis provided in Embodiment 1 of the present invention;

[0039] Figure 2 This is a symptom relationship comparison diagram provided in Embodiment 1 of the present invention;

[0040] Figure 3 This is a comparison chart of classification accuracy provided in Embodiment 1 of the present invention. Detailed Implementation

[0041] Example 1:

[0042] Embodiment 1 of the present invention discloses a method for processing consultation information based on big data analysis. Refer to Figure 1 , including steps S1 - step S6:

[0043] S1: Collect the consultation information text and perform preprocessing.

[0044] S10: Collect the consultation information text.

[0045] Specifically, obtain the consultation information text from the medical database.

[0046] S11: Perform preprocessing on the consultation information text.

[0047] It should be noted that the collected consultation information text is not standardized, and the collected text information needs to be standardized for subsequent analysis.

[0048] Optionally, as an example, the method for preprocessing the consultation information text includes:

[0049] Step 1, perform word segmentation on the consultation information text.

[0050] Optionally, as an example, performing word segmentation on the consultation information text includes:

[0051] The Jieba word segmentation tool can be used for word segmentation.

[0052] Step 2, perform stop word filtering.

[0053] Optionally, as an example, performing stop word filtering includes:

[0054] Based on a preset stop word list containing common words and words without actual diagnostic significance in the medical field such as "of", "already", "I", "Hello, doctor", etc., remove these stop words from the result of word segmentation.

[0055] Step 3, perform word form normalization.

[0056] Optionally, as an example, performing word form normalization includes:

[0057] First, based on an authoritative medical knowledge base, offline construct a comprehensive symptom dictionary containing standardized symptom concepts and their synonyms and alternative names. As an example, the authoritative medical knowledge base can be the SNOMED-CT library or the ICD-10 library, and there is no specific limitation on the authoritative medical knowledge base used.

[0058] Then, the maximum positive matching algorithm is used to match the words in the processed consultation information text with the standardized symptom dictionary, and all words in the text that can be matched are uniformly replaced with their corresponding unique standard symptom concept IDs.

[0059] Finally, the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm is used to transform each consultation record after entity linking and standardization into a high-dimensional standardized feature vector.

[0060] S2: Obtain the low-dimensional embedding vector for each of the multiple standardized symptoms.

[0061] It should be noted that, in order to address the issue of removing symptom combination information during the traditional PCA dimensionality reduction process, the PCA dimensionality reduction process needs to be supervised. During supervision, the symptom combination information from the consultation data is required. This embodiment uses low-dimensional embedding vectors to reflect the symptom combination information.

[0062] Preferably, as an example, a low-dimensional embedding vector is obtained for each of the multiple standardized symptoms, including:

[0063] Step 1: Construct a symptom-diagnosis bipartite diagram.

[0064] Based on a massive dataset of anonymized historical electronic medical records with authoritative final diagnostic labels, a symptom-diagnosis bipartite graph is constructed. This graph contains two types of nodes: symptom nodes and disease nodes. If a symptom and a disease co-occur in historical medical records, an edge is drawn between them, and the edge weight is calculated using Pointwise Mutual Information (PMI) to measure the strength of the association between nodes.

[0065] Step 2: Learn the low-dimensional embedding vector from the symptom-diagnosis bipartite graph.

[0066] First, graph embedding algorithms such as Node2Vec are used to embed the pre-constructed weighted bipartite graph. Above, a biased random walk is performed to generate a node sequence that can capture higher-order correlation information.

[0067] Then, these node sequences are input into the Word2Vec model for training. After training, each symptom node in the graph is represented as a low-dimensional, dense embedding vector that implicitly contains diagnostic association information between its corresponding symptom and other symptoms, denoted as the low-dimensional embedding vector.

[0068] It is understandable that constructing a symptom-diagnosis bipartite graph can link symptoms together, and then learning these relationships from the graph can lead to the acquisition of low-dimensional embedding vectors containing combined information about symptoms.

[0069] S3: Calculate a sample covariance matrix.

[0070] First, all the high-dimensional standardized feature vectors are used to form a high-dimensional standardized data matrix.

[0071] Then, the high-dimensional standardized data matrix is ​​subjected to zero-mean processing to eliminate the influence of data scale.

[0072] Finally, the standard sample covariance matrix of the zero-mean high-dimensional standardized data matrix is ​​calculated using the PCA algorithm.

[0073] S4: Construct a structured covariance matrix that reflects the value of symptom combination.

[0074] It should be noted that the low-dimensional embedding vector is symptom association information learned based on prior medical knowledge. In order to prevent the removal of symptom association information during the dimensionality reduction process, it can be used to supervise the dimensionality reduction process so that the information after dimensionality reduction also conforms to this symptom association information, thereby preventing the removal of symptom association information after dimensionality reduction.

[0075] Preferably, as an example, a structured covariance matrix that reflects the value of symptom combination is constructed, including:

[0076] Step 1: Construct a symptom similarity matrix.

[0077] Calculate the cosine similarity of the low-dimensional embedding vectors of every two symptom nodes, and denote the matrix formed by the cosine similarity of the low-dimensional embedding vectors of all two symptom nodes as the symptom similarity matrix.

[0078] Step 2: Construct a structured covariance matrix that can reflect the value of symptom combination.

[0079]

[0080] in, The standard sample covariance matrix is ​​represented by the covariance matrix obtained through dimensionality reduction analysis using the traditional PCA algorithm. It reflects more of the statistical information in the diagnostic text. Let represent the covariance variable matrix, which is the covariance matrix to be solved. Denotes the square of the Frobenius norm. This represents the graph Laplacian matrix calculated based on the symptom similarity matrix. Represents the trace of a matrix. This represents the preset regularization coefficient, used to control... The degree of determination of the final solution Indicates a positive semidefinite constraint. This represents the structured covariance matrix.

[0081] Understandable, This is used to control the obtained covariance matrix so that it does not deviate too much from the covariance matrix obtained by the traditional PCA algorithm, so that the dimensionality reduction of the diagnostic text information does not deviate too much from the statistical information. This is used to control the symptom similarity information in the obtained covariance matrix so that it does not deviate from the symptom similarity information obtained from authoritative prior medical sources, thereby minimizing the disruption of the correlation information between symptoms before and after dimensionality reduction. By making The value of is kept as small as possible to control the dimensionality reduction process, so that the reduced information has less impact on the original statistical information and also less impact on the correlation information between symptoms. This allows the data dimensionality to be reduced while retaining the desired symptom information.

[0082] S5: Use the principal components to perform dimensionality reduction on the high-dimensional standardized data matrix.

[0083] Preferably, as an example, the dimensionality reduction of the high-dimensional standardized data matrix using the principal components includes:

[0084] The structured covariance matrix is ​​subjected to standard eigenvalue decomposition to obtain a new set of principal components that can maximize the preservation of syndrome information. The first k principal components are selected to form a projection matrix, where k represents a preset value used to control the number of dimensionality reductions.

[0085] Figure 2 A diagram showing the relationship between symptoms was displayed. Figure 2 The dataset contains subgraphs A, B, and C. Subgraph A represents the symptom relationship image before dimensionality reduction, subgraph B represents the symptom relationship image after dimensionality reduction using the method described in this scheme, and subgraph C represents the symptom relationship image after dimensionality reduction using standard PCA. Comparison shows that, compared to the traditional PCA algorithm, the method in this scheme can better preserve the relationships between symptoms.

[0086] The high-dimensional normalized data matrix after zero-mean processing is projected onto this new projection matrix to obtain the dimensionality-reduced data.

[0087] S6: Classify the data based on the dimensionality reduction.

[0088] Optionally, as an example, clustering based on the dimensionality-reduced data includes:

[0089] The dimensionality-reduced data is classified into several categories using a text classification algorithm to facilitate subsequent pathological retrieval or auxiliary diagnosis. As an example, the text classification algorithm can be the Transformer algorithm or other algorithms; this embodiment does not impose specific limitations.

[0090] Figure 3 The image shows a comparison of classification accuracy. As can be seen from the image, the classification accuracy of the data after dimensionality reduction obtained by the method in this scheme is 97.5%, while the classification accuracy of the data after dimensionality reduction obtained by the standard PCA algorithm is 52.5%. Therefore, the method in this scheme can better preserve the original information during dimensionality reduction, thus obtaining more accurate classification results.

[0091] Embodiment 1 of the present invention also discloses a consultation information processing system based on big data analysis, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, a consultation information processing method based on big data analysis according to the present invention is implemented.

[0092] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0093] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (DRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (DRAM), high-bandwidth memory, hybrid memory cube, etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device.

Claims

1. A method for processing inquiry information based on big data analysis, characterized in that, Including the following steps: Collect and preprocess the text of the consultation information; Obtain a low-dimensional embedding vector for each of multiple standardized symptoms, wherein the low-dimensional embedding vector implicitly contains diagnostic association information between its corresponding symptom and other symptoms; A sample covariance matrix is ​​calculated based on the high-dimensional standardized feature vectors of multiple consultation records. By solving an optimization problem that uses the sample covariance matrix as a data fidelity term and a graph Laplacian matrix derived from the low-dimensional embedding vector of the symptoms as a regularization term, a structured covariance matrix that reflects the value of symptom combination is constructed. The high-dimensional standardized feature vector is reduced in dimensionality using the principal components derived from the structured covariance matrix to generate low-dimensional data that retains syndrome information. The structured covariance matrix satisfies the following relationship: in, Represents the standard sample covariance matrix. Represents the covariance variable matrix. Denotes the square of the Frobenius norm. This represents the graph Laplacian matrix calculated based on the symptom similarity matrix. Represents the trace of a matrix. This represents the preset regularization coefficient. Indicates a positive semidefinite constraint. This represents the structured covariance matrix.

2. The method for processing medical consultation information based on big data analysis according to claim 1, characterized in that, The low-dimensional embedding vector is learned by learning on a weighted symptom-diagnosis bipartite graph, whose nodes include symptom nodes and disease diagnosis nodes. 3.The diagnosis information processing method based on big data analysis according to claim 2, characterized in that, The method for obtaining the edge weights of the bipartite graph includes: Edge weights are calculated using a point-by-point mutual information method. 4.The diagnosis information processing method based on big data analysis according to claim 2, characterized in that, The learning process includes: A random walk is performed on the bipartite graph to generate a sequence of nodes, and the sequence is trained using the Word2Vec model. 5.The diagnosis information processing method based on big data analysis of claim 1, wherein, The graph Laplacian matrix is ​​calculated based on a symptom similarity matrix, which is constructed by calculating the cosine similarity between the low-dimensional embedding vectors of any two symptoms. 6.The diagnosis information processing method based on big data analysis according to claim 1, characterized in that, The method for obtaining the high-dimensional standardized feature vector of the consultation record includes: The consultation information text is linked to a standardized symptom dictionary as medical entities, and then vectorized using a term frequency-inverse document frequency algorithm. 7.The diagnosis information processing method based on big data analysis of claim 1, wherein, The dimensionality reduction process includes: The high-dimensional normalized feature vectors are projected onto a projection matrix consisting of the feature vectors corresponding to the largest eigenvalues ​​of the structured covariance matrix to obtain low-dimensional data. 8.The diagnosis information processing method based on big data analysis of claim 1, wherein, Also includes: The low-dimensional data obtained after the dimensionality reduction process is then classified. 9.A diagnosis information processing system based on big data analysis, characterized by, include: A processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement a method for processing medical information based on big data analysis according to any one of claims 1-8.

Images