Python language-based intelligent medical data storage, processing and analysis platform

Abstract

The invention relates to the technical field of medical information processing, in particular to a Python language-based medical data storage, processing and analysis intelligent platform, which comprises a data acquisition module, a determination module, a word segmentation module, a labeling module, an adjustment module, a conversion module, a dimension reduction module, a correction module, a generation module, a storage module and an analysis module. According to the method, the processing efficiency and accuracy of the medical document data are remarkably improved through the automatic text processing and analysis process, the complexity of different texts can be flexibly handled through dynamic adjustment of the segmentation granularity and the dimension reduction process, the accuracy of data processing is ensured, and by combining the analysis capability of the Gaussian mixture model, the data processing efficiency is improved. The platform can extract effective information from massive medical literatures, provides powerful support for subsequent medical research and decision making, and effectively solves the problems that medical auditing efficiency is low and real-time medical data changes cannot be quickly responded due to dependence on manual intervention and lagging of rule updating.

Description

Technical Field

[0001] The present invention relates to the field of medical information processing technology, and in particular to a medical data storage, processing and analysis intelligent platform based on Python language. Background Art

[0002] With the continuous development of medical informatization, the amount of data generated in the medical field has exploded, especially the accumulation of text information such as medical literature, clinical records, and medical records. Traditional manual analysis and manual processing methods have been unable to cope with the rapid growth and complexity of massive data, and are unable to effectively extract the potential medical knowledge and diagnosis and treatment information. At the same time, the accuracy of medical research and clinical decision-making has placed higher and higher requirements on data analysis. How to extract valuable information from a large amount of medical text and perform effective data storage, processing, and analysis has become an important topic in the process of medical informatization.

[0003] The patent document of Chinese patent application publication number CN108648810A discloses a data processing method and device for medical review, which method includes the following steps: S01, in response to the operation of modifying medical review rules triggered by the modification operation entry, calling and displaying the review logic configuration interface, wherein the modification operation entry for modifying the medical review rules is preset; S02, when it is monitored that the review logic configuration interface receives a first modification request instruction, providing a first modification window for editing and modifying the preset original medical review rules, and when it is monitored that the review logic configuration interface receives a second modification request instruction, providing a second modification window for creating new medical review rules; S03, receiving the edited and modified medical review rules in the first modification window, or receiving the newly created medical review rules in the second modification window, and using the received medical review rules as the current new medical review rules.

[0004] It can be seen that this method emphasizes manual editing and modification of the interface of the review rules, relies on manual input and editing, is easily affected by human factors, has low operating efficiency, and is difficult to automatically process large-scale data; in actual applications, the modification and adjustment of medical review rules requires a large amount of medical literature, medical record data and real-time data to make decisions, and traditional manual modification methods cannot effectively handle such large-scale and complex data. Summary of the invention

[0005] To this end, the present invention provides a medical data storage, processing and analysis intelligent platform based on Python language, which is used to overcome the problems in the prior art of low medical review efficiency and inability to quickly respond to real-time medical data changes due to reliance on manual intervention and delayed rule updates through automated data processing, intelligent analysis and dynamic rule adjustment.

[0006] To achieve the above objectives, the present invention provides a Python-based medical data storage, processing and analysis intelligent platform, comprising:

[0007] A data collection module is used to collect text data in the process of medical document input in real time;

[0008] A determination module, connected to the data acquisition module, for determining a number of complex texts according to the text data;

[0009] A word segmentation module, which is connected to the determination module and is used to segment each of the complex texts using a preset word segmentation tool at a preset segmentation granularity to form a plurality of phrases;

[0010] A marking module, connected to the word segmentation module, for marking the part of speech of each phrase according to a preset marking tool to form a plurality of marked phrases;

[0011] An adjustment module, connected to the annotation module, for adjusting the preset segmentation granularity according to the grammatical consistency of each annotation phrase at any adjacent time within a preset adjustment time length;

[0012] A conversion module, connected to the annotation module, for converting each of the annotation phrases according to a preset word vector model to form a plurality of word vectors;

[0013] A dimension reduction module, connected to the conversion module, for reducing the dimension of each word vector by a preset dimension reduction amount according to the dimensions of all word vectors within a preset determination time length, to form a plurality of dimension reduction vectors;

[0014] A correction module, connected to the dimension reduction module, for correcting a preset dimension reduction amount according to the semantic similarity of any adjacent dimension reduction vectors and a preset standard similarity;

[0015] A generation module, which is connected to the conversion module and the dimension reduction module respectively, and is used to generate a plurality of sentence vectors according to all the word vectors and the dimension reduction vectors;

[0016] A storage module, connected to the generation module, for associating the sentence vector with the marked phrase and storing the associated sentence vector in a preset database;

[0017] An analysis module is connected to the storage module and is used to analyze the sentence vector and the labeled phrase according to a preset mixed Gaussian model to form an analysis report.

[0018] Furthermore, the determining module includes:

[0019] a vocabulary recognition unit, for recognizing different words in the text data to form a plurality of different words, and recognizing a total word in the text data to form a plurality of total words;

[0020] A complexity calculation unit connected to the vocabulary recognition unit, for calculating the ratio of the number of the different words to the number of the total words to form text complexity;

[0021] A distribution calculation unit connected to the complexity calculation unit, for calculating the standard deviation of the text complexity within a preset historical time period to form a complexity distribution degree;

[0022] A distribution degree comparison unit, connected to the distribution calculation unit, for comparing the complex distribution degree with a preset standard distribution degree to form a distribution comparison result;

[0023] A determination unit is connected to the distribution comparison unit and is used to determine a number of complex texts according to the distribution comparison result, the comparison result of the text complexity and the preset standard complexity.

[0024] Furthermore, the word segmentation module includes:

[0025] A stop word recognition unit, used for recognizing stop words in the complex text to form a plurality of stop words;

[0026] A segmentation determination unit connected to the stop word recognition unit, for determining whether to perform segmentation according to a preset stop word list and each of the stop words, and forming a segmentation determination result;

[0027] The word segmentation unit is connected to the segmentation determination unit and is used to segment each of the complex texts according to the segmentation determination result, the preset segmentation granularity and the preset word segmentation tool to form a plurality of phrases.

[0028] Furthermore, the adjustment module includes:

[0029] A sorting unit, used to sort the marked phrases according to their parts of speech to form a part of speech sequence;

[0030] A consistency calculation unit connected to the sorting unit, for calculating the cosine similarity of the part-of-speech sequences at any adjacent moments to form a grammatical consistency;

[0031] An adjustment unit is connected to the consistency calculation unit and is used to adjust the preset segmentation granularity according to the grammatical consistency.

[0032] Furthermore, the adjustment unit includes:

[0033] A consistency comparison subunit, used to compare the grammatical consistency with a preset standard consistency to form a consistency comparison result;

[0034] An adjustment subunit is connected to the consistency comparison subunit and is used to adjust the preset segmentation granularity according to the consistency comparison result, the preset standard consistency and a preset adjustment coefficient.

[0035] Furthermore, the dimension reduction module includes:

[0036] A complexity calculation unit, used to calculate the variance of the dimension to form complexity;

[0037] A complex fluctuation calculation unit, connected to the complexity calculation unit, for calculating the standard deviation of the complexity to form a complex fluctuation value;

[0038] A dimension reduction unit is connected to the complex fluctuation calculation unit and is used to reduce the dimension of each word vector according to the complex fluctuation value to form a plurality of reduced dimension vectors.

[0039] Furthermore, the dimension reduction unit comprises:

[0040] A dimensionality reduction comparison subunit, which is connected to the complex fluctuation calculation unit and is used to compare the complex fluctuation value with a preset standard complex fluctuation value to form a dimensionality reduction comparison result;

[0041] A dimensionality reduction operation subunit is connected to the dimensionality reduction comparison subunit and is used to reduce the dimensionality of each word vector according to the dimensionality reduction comparison result, a preset dimensionality reduction model and a preset dimensionality reduction quantity to form a plurality of dimensionality reduction vectors.

[0042] Furthermore, the correction module includes:

[0043] A similarity calculation unit, used to calculate the cosine similarity of the reduced-dimensional vector to form semantic similarity;

[0044] A correction unit is connected to the similarity calculation unit and is used to correct the preset dimensionality reduction amount according to a relative deviation between the semantic similarity and the preset standard similarity and a preset correction coefficient.

[0045] Furthermore, the generating module comprises:

[0046] A combining unit, used for combining all the word vectors and each of the dimensionality reduction vectors by weighted averaging to form a plurality of temporary sentence vectors;

[0047] A generating unit is connected to the combining unit and is used to perform regularization processing on each of the temporary sentence vectors according to a preset post-processing algorithm to form a plurality of sentence vectors.

[0048] Furthermore, the analysis module includes:

[0049] An analysis unit, configured to cluster the sentence vectors according to the preset mixed Gaussian model to form a clustering result;

[0050] An extraction unit connected to the analysis unit, for extracting the part-of-speech distribution features of the marked phrase to form a feature extraction result;

[0051] An output unit is connected to the analysis unit and the extraction unit respectively, and is used to output an analysis report according to the clustering result and the feature extraction result.

[0052] Compared with the prior art, the beneficial effect of the present invention is that the processing efficiency and accuracy of medical document data are significantly improved through automated text processing and analysis processes. The dynamic adjustment of segmentation granularity and dimensionality reduction process can flexibly cope with the complexity of different texts and ensure the accuracy of data processing. Combined with the analysis capability of mixed Gaussian models, the platform can extract effective information from massive medical literature and provide strong support for subsequent medical research and decision-making. At the same time, the database storage and report generation functions of the platform enhance the accessibility and visualization of data, which is helpful for data mining and knowledge discovery in the medical field, and effectively solves the problems of low efficiency of medical review and inability to quickly respond to real-time medical data changes due to reliance on manual intervention and delayed rule updates.

[0053] Furthermore, by comprehensively considering the lexical composition and complexity distribution of the text, it can ensure that when recognizing complex texts, the lexical diversity and structural complexity of the text can be accurately captured, rather than relying solely on a single indicator.

[0054] Furthermore, through the multi-level processing of the word segmentation module, the interference of stop words can be effectively avoided, making the subsequent text analysis more accurate. Stop word recognition and reasonable segmentation judgment ensure that meaningless components in the text do not participate in subsequent analysis, improving the quality and efficiency of word segmentation. The preset segmentation granularity and word segmentation tools further optimize the segmentation effect and ensure that the generated phrases have high semantic coherence.

[0055] Furthermore, by dynamically adjusting the segmentation granularity based on grammatical consistency, it helps to optimize the accuracy and effect of word segmentation. In different texts, the complexity of the grammatical structure may vary. Adjusting the segmentation granularity based on the consistency of the part of speech at adjacent moments can flexibly respond to changes in the grammatical structure in the text, ensure that semantic information is not lost during the segmentation process, effectively avoid the problem of over-segmentation or under-segmentation caused by fixed granularity, and improve the accuracy and effectiveness of overall text processing.

[0056] Furthermore, by comparing the grammatical consistency with the preset standard consistency, the adjustment unit can accurately control the adjustment of the word segmentation granularity to make it more consistent with the actual grammatical structure of the text. This adjustment mechanism can not only improve the grammatical adaptability of word segmentation and avoid over-segmentation or under-segmentation, but also flexibly adjust according to the complexity of the actual text, thereby improving the accuracy and robustness of text processing.

[0057] Furthermore, the diversity and stability of word vectors are measured by calculating complexity and fluctuation values, so as to achieve targeted dimensionality reduction operations. Through dimensionality reduction, the impact of redundant information is reduced, while the semantic features of word vectors are retained, improving the efficiency and accuracy of data processing.

[0058] Furthermore, through the precise coordination of dimensionality reduction comparison and dimensionality reduction operations, it is ensured that the dimensionality reduction process can be optimized according to the complexity and volatility of word vectors, avoiding excessive loss of information. By comparing with preset standards, the system can automatically adjust the dimensionality reduction strategy to achieve more efficient semantic representation and better computing performance.

[0059] Furthermore, the number of dimensionality reduction is dynamically adjusted by combining the deviation between semantic similarity and preset standard similarity, ensuring that the dimensionality reduction operation further reduces the loss of redundant data while retaining semantic information. By adjusting the number of dimensionality reduction, the quality of word vectors can be improved, the accuracy and consistency of semantic representation can be enhanced, and the effect of subsequent analysis and model training can be improved.

[0060] Furthermore, by combining word vectors and dimensionality reduction vectors with weighted average, sentence vectors can comprehensively reflect the semantic information in the text, and regularization processing is used to improve the stability and consistency of the generated sentence vectors. Regularization ensures the standardization of the vectors and avoids the adverse effects of excessively large or small values ​​on subsequent analysis, which helps to improve the accuracy and robustness of the model.

[0061] Furthermore, sentence vector clustering by mixed Gaussian model can effectively classify similar sentences into the same category, helping to reveal the underlying structure and pattern in the text. Combined with the extraction of part-of-speech distribution features, the ability to analyze the deep grammatical structure of the text is further improved. The final analysis report provides accurate data support and decision-making basis for subsequent text processing, classification or application. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of the medical data storage, processing and analysis intelligent platform based on Python language in this embodiment;

[0063] Figure 2 A decision logic diagram for determining a complex text by a determination unit in this embodiment;

[0064] Figure 3 A decision logic diagram for adjusting a preset segmentation granularity of the adjustment subunit in this embodiment;

[0065] Figure 4 This is a decision logic diagram for dimensionality reduction performed by the dimensionality reduction operation subunit of this embodiment. DETAILED DESCRIPTION

[0066] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0067] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0068] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0069] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0070] See also Figure 1 As shown, it is a schematic diagram of the medical data storage, processing and analysis intelligent platform based on Python language in this embodiment;

[0071] This embodiment provides a Python-based medical data storage, processing and analysis intelligent platform, including:

[0072] A data collection module is used to collect text data in the process of medical document input in real time;

[0073] A determination module, connected to the data acquisition module, for determining a number of complex texts according to the text data;

[0074] A word segmentation module, which is connected to the determination module and is used to segment each of the complex texts using a preset word segmentation tool at a preset segmentation granularity to form a plurality of phrases;

[0075] A marking module, connected to the word segmentation module, for marking the part of speech of each phrase according to a preset marking tool to form a plurality of marked phrases;

[0076] An adjustment module, connected to the annotation module, for adjusting the preset segmentation granularity according to the grammatical consistency of each annotation phrase at any adjacent time within a preset adjustment time length;

[0077] A conversion module, connected to the annotation module, for converting each of the annotation phrases according to a preset word vector model to form a plurality of word vectors;

[0078] A dimension reduction module, connected to the conversion module, for reducing the dimension of each word vector by a preset dimension reduction amount according to the dimensions of all word vectors within a preset determination time length, to form a plurality of dimension reduction vectors;

[0079] A correction module, connected to the dimension reduction module, for correcting a preset dimension reduction amount according to the semantic similarity of any adjacent dimension reduction vectors and a preset standard similarity;

[0080] A generation module, which is connected to the conversion module and the dimension reduction module respectively, and is used to generate a plurality of sentence vectors according to all the word vectors and the dimension reduction vectors;

[0081] A storage module, connected to the generation module, for associating the sentence vector with the marked phrase and storing the associated sentence vector in a preset database;

[0082] An analysis module is connected to the storage module and is used to analyze the sentence vector and the labeled phrase according to a preset mixed Gaussian model to form an analysis report.

[0083] The data acquisition module is based on natural language processing (NLP) technology. It parses structured and unstructured texts in medical documents and extracts relevant data in real time to support subsequent analysis.

[0084] Word embedding is a technique that maps words into low-dimensional, dense vectors in a vector space to represent semantic information between words.

[0085] The preset word segmentation tool is an algorithm used to segment complex text into words or phrases, depending on the language characteristics and application scenarios of the text. Generally, a standard word segmentation library is used, depending on the complexity of the corpus. In this embodiment, the Jieba word segmentation tool is used, which has high accuracy and wide applicability and is suitable for Chinese medical text processing.

[0086] The preset segmentation granularity refers to the degree to which the text is divided into multiple small units (such as phrases) during word segmentation. It depends on the structure of the text and the needs of analysis. It is usually set between 2 and 5 words. In this embodiment, it is set to 2 to 4 words. It aims to ensure that the phrases after word segmentation can represent meaningful medical terms or concepts and avoid excessive splitting into meaningless word units.

[0087] The preset annotation tool is a tool or model used to annotate words or phrases in a text with parts of speech or grammatical tags. Depending on the language used and the task requirements, a general-purpose annotation tool is usually used, such as spaCy or NLTK in the Python language, which is set to include the main part-of-speech categories. In this embodiment, spaCy is used for part-of-speech tagging, which can quickly process complex grammatical structures in medical texts.

[0088] Grammatical consistency refers to the consistency measure of the grammatical structure between phrases in adjacent time steps. It depends on the grammatical structure of the text and the accuracy of the annotation tool. It is generally evaluated by calculating the grammatical tree structure of adjacent phrases. It is usually set between 0 and 1. In this embodiment, it is set to 0.85, which can ensure the stability of word segmentation and annotation and contribute to the accuracy of text analysis.

[0089] The preset word vector model is a model for converting each word in the text into a low-dimensional dense vector. Depending on the corpus used and the task requirements, the Word2Vec model or the G loVe model is generally used, and the word vector is set to 300 to 768 dimensions. In this embodiment, the G loVe word vector model is used, which is set to 300 dimensions. It can take into account both computational efficiency and representation ability while ensuring model performance, and is suitable for the ambiguity processing of medical texts.

[0090] The preset dimensionality reduction number refers to the number of dimensions reduced in the dimensionality reduction operation, which is used to reduce computational complexity and redundant information. It depends on the dimension of the word vector and the requirements of the target task. It is usually set to between 30 and 100 dimensions. In this embodiment, it is set to 50 dimensions, which can not only retain sufficient semantic information, but also reduce the computational burden and improve the efficiency of subsequent model processing.

[0091] Semantic similarity measures the semantic similarity between two word vectors, which depends on the quality of the word vectors and the characteristics of the target task.

[0092] The preset judgment time refers to the time range or data length considered when performing word vector dimensionality reduction or other analysis, which is used to determine the data processing and judgment within this time period, depending on the frequency of data changes, processing requirements, and real-time requirements of the algorithm. In general, the value of the preset judgment time should be set according to the specific application scenario, usually between a few seconds and a few minutes. In this embodiment, it is set to 30 seconds, which can balance the real-time nature of data processing and the computational burden of the system, avoid excessive data noise due to too short a time window, and also ensure that the system is sensitive enough to respond to rapidly changing data streams.

[0093] The preset standard similarity refers to the standard value used to judge the similarity of word vectors after dimensionality reduction. It depends on the needs of the application scenario, especially the semantic consistency requirements. It is usually set between 0.85 and 0.95. In this embodiment, it is set to 0.9, which can ensure that the word vectors maintain a high semantic similarity after dimensionality reduction, which is helpful for subsequent analysis and modeling.

[0094] The Gaussian Mixture Model (GMM) is a tool for probability distribution modeling that can help analyze multimodal data and estimate the distribution characteristics of the data, depending on the distribution type and complexity of the data.

[0095] First, the data acquisition module collects input medical text data in real time. Then, the determination module determines the complex text based on the text data and divides it into phrases through the word segmentation module. The annotation module marks the part of speech for the phrase, and the adjustment module further adjusts the segmentation granularity according to the grammatical consistency. Subsequently, the conversion module converts the annotated phrase into a word vector, the dimensionality reduction module reduces the word vector dimension by dimensionality reduction, and the correction module adjusts the number of dimensionality reductions according to the semantic similarity. The generation module combines the word vector and the dimensionality reduction vector to generate a sentence vector, and stores these vectors and phrases in the database. Finally, the analysis module applies a mixed Gaussian model to analyze the stored data and generate a detailed analysis report, all of which are implemented in Python.

[0096] The processing efficiency and accuracy of medical document data have been significantly improved through automated text processing and analysis processes. The dynamic adjustment of segmentation granularity and dimensionality reduction process can flexibly cope with the complexity of different texts and ensure the accuracy of data processing. Combined with the analysis capabilities of mixed Gaussian models, the platform can extract effective information from massive medical literature and provide strong support for subsequent medical research and decision-making. At the same time, the platform's database storage and report generation functions enhance the accessibility and visualization of data, which is helpful for data mining and knowledge discovery in the medical field, and effectively solves the problems of low efficiency of medical review and inability to quickly respond to real-time medical data changes due to reliance on manual intervention and delayed rule updates.

[0097] Please continue reading Figure 2 As shown, it is a decision logic diagram of the determination unit determining complex text in this embodiment;

[0098] Specifically, the determination module includes:

[0099] a vocabulary recognition unit, for recognizing different words in the text data to form a plurality of different words, and recognizing a total word in the text data to form a plurality of total words;

[0100] A complexity calculation unit connected to the vocabulary recognition unit, for calculating the ratio of the number of the different words to the number of the total words to form text complexity;

[0101] A distribution calculation unit connected to the complexity calculation unit, for calculating the standard deviation of the text complexity within a preset historical time period to form a complexity distribution degree;

[0102] A distribution degree comparison unit, connected to the distribution calculation unit, for comparing the complex distribution degree with a preset standard distribution degree to form a distribution comparison result;

[0103] A determination unit is connected to the distribution degree comparison unit, and is used to determine a number of the complex texts when the distribution comparison result shows that the complexity distribution degree is greater than the preset standard distribution degree, and the text complexity is greater than the preset standard complexity.

[0104] The preset standard distribution refers to a standard value of text complexity distribution set according to historical data or experience, which depends on the average structural complexity of the text type and field, and is usually set within the standard deviation range of complexity variation in the text, between 0.3 and 0.5. In this embodiment, it is set to 0.4, which is intended to capture the fluctuation range of common structures and term distributions in medical documents, so that the system can identify texts beyond the conventional complexity range.

[0105] The preset standard complexity refers to a value set by analyzing the average complexity of historical data sets, which depends on the frequency and diversity of domain-specific text vocabulary. This value can be set according to the vocabulary distribution of the training data, usually between 0.2 and 0.6. In this embodiment, it is set to 0.5, which can effectively distinguish common concise texts from complex term-intensive texts based on the analysis results of the diversity of vocabulary usage in medical documents.

[0106] First, the vocabulary recognition unit analyzes the input text data to identify different words and all words in the text, thereby forming different words and total words. Next, the complexity calculation unit calculates the ratio between different words and total words to obtain the text complexity. Then, the distribution calculation unit uses the standard deviation of text complexity in the historical data to calculate the complexity distribution, and compares the calculated distribution with the preset standard distribution through the distribution comparison unit to form a distribution comparison result. Finally, the determination unit judges based on the comparison results that when the complexity of the text is greater than the standard complexity and the complexity distribution exceeds the standard distribution, it is confirmed as a complex text, and these texts are determined for subsequent processing.

[0107] By comprehensively considering the lexical composition and complexity distribution of the text, it is possible to ensure that when identifying complex texts, the lexical diversity and structural complexity of the text can be accurately captured, rather than relying solely on a single indicator.

[0108] Specifically, the word segmentation module includes:

[0109] A stop word recognition unit, used for recognizing stop words in the complex text to form a plurality of stop words;

[0110] A segmentation determination unit connected to the stop word recognition unit, for determining whether to perform segmentation according to a preset stop word list and each of the stop words, and forming a segmentation determination result;

[0111] The word segmentation unit is connected to the segmentation determination unit and is used to segment each of the complex texts according to the preset segmentation granularity and the preset word segmentation tool to form a plurality of phrases when the segmentation determination result is to perform segmentation.

[0112] Preset stop word list is a set comprising common vocabulary without actual semantic meaning, generally including pronouns, conjunctions, auxiliary words etc., depending on text type and language model, usually based on frequent occurrence and meaningless vocabulary in a large amount of corpora.Usually, the size of stop word list can vary according to text length and processing target, and is generally set to hundreds to thousands of vocabulary.In the present embodiment, stop word list is set to about 500, to adapt to the characteristics of medical text, ensure to remove meaningless vocabulary without missing key information.By reasonably setting the size of stop word list, can effectively reduce the amount of calculation, improve the efficiency of subsequent text analysis, avoid too much segmentation error simultaneously, ensure the accuracy in text processing.

[0113] First, the stop words in the text are identified by the stop word recognition unit, and each stop word in the text is judged according to the preset stop word list to decide whether to segment it. If it is determined that segmentation is required, the text is segmented; otherwise, the segmentation operation is skipped. Finally, the word segmentation unit performs actual segmentation operations on the complex text according to the segmentation judgment results, the preset segmentation granularity and the word segmentation tool, dividing it into several phrases for subsequent processing.

[0114] Through the multi-level processing of the word segmentation module, the interference of stop words can be effectively avoided, making the subsequent text analysis more accurate. Stop word identification and reasonable segmentation judgment ensure that meaningless components in the text do not participate in subsequent analysis, improving the quality and efficiency of word segmentation. The preset segmentation granularity and word segmentation tools further optimize the segmentation effect and ensure that the generated phrases have high semantic coherence.

[0115] Specifically, the adjustment module includes:

[0116] A sorting unit, used to sort the marked phrases according to their parts of speech to form a part of speech sequence;

[0117] A consistency calculation unit connected to the sorting unit, for calculating the cosine similarity of the part-of-speech sequences at any adjacent moments to form a grammatical consistency;

[0118] An adjustment unit is connected to the consistency calculation unit and is used to adjust the preset segmentation granularity according to the grammatical consistency.

[0119] First, the sorting unit sorts the tagged phrases according to their parts of speech to form a part-of-speech sequence. Then, the consistency calculation unit calculates the cosine similarity between the part-of-speech sequences at adjacent moments to quantify the grammatical consistency between the parts of speech. Finally, the adjustment unit dynamically adjusts the preset segmentation granularity based on the calculated grammatical consistency, thereby ensuring that the word segmentation operation can avoid excessive or insufficient refinement while retaining sufficient grammatical information.

[0120] Dynamically adjusting the segmentation granularity based on grammatical consistency helps optimize the accuracy and effect of word segmentation. The complexity of grammatical structures may vary in different texts. Adjusting the segmentation granularity based on the consistency of parts of speech at adjacent moments can flexibly respond to changes in grammatical structures in the text, ensure that semantic information is not lost during the segmentation process, effectively avoid the problem of over-segmentation or under-segmentation caused by fixed granularity, and improve the accuracy and effectiveness of overall text processing.

[0121] Please continue reading Figure 3 As shown, it is a decision logic diagram of the adjustment subunit adjusting the preset segmentation granularity in this embodiment;

[0122] Specifically, the adjustment unit includes:

[0123] A consistency comparison subunit, used to compare the grammatical consistency with a preset standard consistency to form a consistency comparison result;

[0124] An adjustment subunit is connected to the consistency comparison subunit, and is used to adjust the preset segmentation granularity according to the relative deviation between the grammatical consistency and the preset standard consistency and a preset adjustment coefficient when the consistency comparison result is that the grammatical consistency is less than the preset standard consistency, and the adjusted preset segmentation granularity is positively correlated with the relative deviation between the grammatical consistency and the preset standard consistency.

[0125] The preset standard consistency refers to a reference value set when adjusting the word segmentation granularity, which is used to measure the minimum standard of grammatical consistency. It depends on the grammatical complexity of the text and the processing goal. It is usually set between 0.7 and 0.9. In this embodiment, it is set to 0.85, which aims to balance the fine granularity of text segmentation and grammatical consistency to ensure the accuracy and readability of text processing. By reasonably setting the standard consistency, excessive splitting or merging can be avoided, thereby ensuring that the processed text can still maintain its original grammatical structure.

[0126] The preset adjustment coefficient is a parameter used to adjust the word segmentation granularity, which is dynamically adjusted according to the comparison results of grammatical consistency, depending on the specific application scenario and processing requirements, and is usually set between 0.1 and 0.5. In this embodiment, it is set to 0.2, the purpose is to appropriately relax the range of granularity adjustment when the grammatical consistency is low, so that the text segmentation can adapt to complex grammatical structures without being overly refined and affecting processing efficiency. By setting a suitable adjustment coefficient, the flexibility of the system can be optimized to ensure that the best word segmentation effect can be obtained in various texts.

[0127] The specific adjustment process is completed through two sub-units. First, the consistency comparison sub-unit compares the calculated grammatical consistency with the preset standard consistency to generate a consistency comparison result. Then, the adjustment sub-unit dynamically adjusts the word segmentation granularity based on the consistency comparison result, standard consistency and preset adjustment coefficient to ensure the accuracy and flexibility of the word segmentation operation. The preset standard consistency and adjustment coefficient provide the system with a basis for flexible adjustment of granularity, so that the segmentation granularity is optimized according to the grammatical consistency of the actual text.

[0128] By comparing the grammatical consistency with the preset standard consistency, the adjustment unit can accurately control the adjustment of the word segmentation granularity to make it more consistent with the actual grammatical structure of the text. This adjustment mechanism can not only improve the grammatical adaptability of word segmentation and avoid over-segmentation or under-segmentation, but also flexibly adjust according to the complexity of the actual text, improving the accuracy and robustness of text processing.

[0129] Specifically, the dimension reduction module includes:

[0130] A complexity calculation unit, used to calculate the variance of the dimension to form complexity;

[0131] A complex fluctuation calculation unit, connected to the complexity calculation unit, for calculating the standard deviation of the complexity to form a complex fluctuation value;

[0132] A dimension reduction unit is connected to the complex fluctuation calculation unit and is used to reduce the dimension of each word vector according to the complex fluctuation value to form a plurality of reduced dimension vectors.

[0133] First, the complexity calculation unit calculates the variance of the word vector dimension to quantify the complexity of the word vector. Then, the complex fluctuation calculation unit calculates the standard deviation of the complexity to obtain the complex fluctuation value, which is used to measure the volatility of the word vector. Finally, the dimensionality reduction unit reduces the dimensionality of the word vector according to the complex fluctuation value to generate the reduced dimensionality word vector.

[0134] The diversity and stability of word vectors are measured by calculating complexity and fluctuation values, so as to achieve targeted dimensionality reduction operations. Through dimensionality reduction, the impact of redundant information is reduced, while the semantic features of word vectors are retained, improving the efficiency and accuracy of data processing.

[0135] Please continue reading Figure 4 As shown, it is a decision logic diagram of the dimension reduction operation subunit in this embodiment;

[0136] Specifically, the dimension reduction unit includes:

[0137] A dimensionality reduction comparison subunit, which is connected to the complex fluctuation calculation unit and is used to compare the complex fluctuation value with a preset standard complex fluctuation value to form a dimensionality reduction comparison result;

[0138] A dimensionality reduction operation subunit is connected to the dimensionality reduction comparison subunit and is used to reduce the dimension of each word vector according to a preset dimensionality reduction model and a preset dimensionality reduction amount to form a plurality of dimensionality reduction vectors when the dimensionality reduction comparison result is a complex fluctuation value greater than a preset standard complex fluctuation value.

[0139] The preset dimensionality reduction model is a predefined mathematical or statistical model used to compress high-dimensional data into low-dimensional data, depending on the dimensionality reduction goal (such as maximizing the interpretability of the data, maintaining the main features of the data, or improving computational efficiency) and the dimensionality reduction algorithm used (such as PCA and t-SNE). Usually, different dimensionality reduction models are selected according to task requirements. For example, the PCA model is usually used for linear dimensionality reduction, while t-SNE is suitable for nonlinear dimensionality reduction. In this embodiment, the preset dimensionality reduction model selects PCA. The purpose is to retain the main information of the data as much as possible through linear transformation, which can effectively reduce the dimension, reduce the computational burden, and keep the variability of the data as much as possible.

[0140] The preset number of dimensionality reduction refers to the target dimension that is desired to be obtained after the word vector is reduced in dimension. This number usually depends on the number of features of the data, the computational efficiency, and the complexity of the model. It is usually set between 10% and 30% of the original dimension to reduce the computational complexity while maintaining the integrity of the information as much as possible. In this embodiment, the preset number of dimensionality reduction is set to 100 dimensions, that is, the 300-dimensional vector of each word is reduced to 100 dimensions, while retaining the semantic information and the main features of the data as much as possible, which can effectively reduce the amount of calculation and storage requirements, while still retaining sufficient accuracy in semantic expression, which can significantly reduce processing complexity and speed up analysis without excessive loss of key information.

[0141] The word vector is accurately reduced in dimension through two sub-units. First, the dimensionality reduction comparison sub-unit calculates the complex fluctuation value of the word vector and compares it with the preset standard complex fluctuation value to obtain the dimensionality reduction comparison result. Then, the dimensionality reduction operation sub-unit applies the corresponding dimensionality reduction operation according to the dimensionality reduction comparison result, the preset dimensionality reduction model and the number of dimensionality reduction, reduces the dimensionality of the word vector, and generates the reduced dimensionality word vector.

[0142] Through the precise coordination of dimensionality reduction comparison and dimensionality reduction operations, the dimensionality reduction process can be optimized according to the complexity and volatility of word vectors, avoiding excessive loss of information. By comparing with preset standards, the system can automatically adjust the dimensionality reduction strategy to achieve more efficient semantic representation and better computing performance.

[0143] Specifically, the correction module includes:

[0144] A similarity calculation unit, used to calculate the cosine similarity of the reduced-dimensional vector to form semantic similarity;

[0145] A correction unit is connected to the similarity calculation unit and is used to correct the preset dimensionality reduction amount according to the relative deviation between the semantic similarity and the preset standard similarity and a preset correction coefficient, wherein the relative deviation between the semantic similarity and the preset standard similarity is positively correlated with the corrected preset dimensionality reduction amount.

[0146] The preset correction coefficient is a parameter used to control the sensitivity and adjustment range in the process of correcting the dimensionality reduction quantity. It determines the strength of the correction operation based on the deviation between the predetermined standard similarity and the actual semantic similarity calculated. It depends on the needs of the specific application, such as the need for more accurate semantic matching or the tolerance of certain semantic losses. It is usually set between 0 and 1, and is set to 0.1 in this embodiment, which can avoid over-adjustment while ensuring semantic accuracy, and prevent the dimensionality reduction operation from being too aggressive and causing the loss of useful information.

[0147] First, the cosine similarity between the reduced-dimensionality vectors is calculated to obtain the semantic similarity of each reduced-dimensionality vector. Subsequently, the correction unit corrects the preset reduced-dimensionality quantity according to the relative deviation between the actual calculated semantic similarity and the preset standard similarity, combined with the preset correction coefficient. The corrected reduced-dimensionality quantity will better match the actual semantic consistency, thereby optimizing the reduced-dimensionality result.

[0148] By combining the deviation between semantic similarity and preset standard similarity to dynamically adjust the number of dimensionality reduction, we can ensure that the dimensionality reduction operation further reduces the loss of redundant data while retaining semantic information. By adjusting the number of dimensionality reduction, we can improve the quality of word vectors, enhance the accuracy and consistency of semantic representation, and thus improve the effect of subsequent analysis and model training.

[0149] Specifically, the generation module includes:

[0150] A combining unit, used for combining all the word vectors and each of the dimensionality reduction vectors by weighted averaging to form a plurality of temporary sentence vectors;

[0151] A generating unit is connected to the combining unit and is used to perform regularization processing on each of the temporary sentence vectors according to a preset post-processing algorithm to form a plurality of sentence vectors.

[0152] The preset post-processing algorithm refers to an algorithm for further processing the temporary sentence vector after it is generated, with the purpose of optimizing the quality of the sentence vector, improving its representation ability, and ensuring that they can better adapt to subsequent tasks and analysis, depending on task requirements, data characteristics, and computing resources. This embodiment uses L2 normalization to ensure that the generated sentence vectors have the same modulus, thereby avoiding the influence of different sentence vector length differences on the similarity calculation.

[0153] First, the combination unit performs weighted average combination of all word vectors and dimensionality reduction vectors to generate temporary sentence vectors. Then, the generation unit applies a preset post-processing algorithm to regularize these temporary sentence vectors to ensure that the generated sentence vectors are within a certain range and meet semantic consistency and computational stability, thereby forming the final sentence vector.

[0154] By combining word vectors and dimensionality reduction vectors with weighted average, sentence vectors can comprehensively reflect the semantic information in the text, and regularization processing is used to improve the stability and consistency of the generated sentence vectors. Regularization ensures the standardization of the vectors and avoids the adverse effects of excessively large or small values ​​on subsequent analysis, which helps to improve the accuracy and robustness of the model.

[0155] Specifically, the analysis module includes:

[0156] An analysis unit, configured to cluster the sentence vectors according to the preset mixed Gaussian model to form a clustering result;

[0157] An extraction unit connected to the analysis unit, for extracting the part-of-speech distribution features of the labeled phrase using a conditional random field model (CRF) to form a feature extraction result;

[0158] An output unit is connected to the analysis unit and the extraction unit respectively, and is used to output an analysis report according to the clustering result and the feature extraction result.

[0159] The analysis unit clusters the generated sentence vectors using a preset mixed Gaussian model, identifies the similarities between different sentence vectors, and forms clustering results. Next, the extraction unit extracts the part-of-speech distribution features based on the annotated phrases in the sentence and generates feature extraction results. Finally, the output unit generates and outputs an analysis report based on the clustering results and feature extraction results, providing an in-depth understanding and analysis of the text.

[0160] Sentence vector clustering by mixed Gaussian model can effectively classify similar sentences into the same category, helping to reveal the underlying structure and patterns in the text. Combined with the extraction of part-of-speech distribution features, the ability to analyze the deep grammatical structure of the text is further improved. The final analysis report provides accurate data support and decision-making basis for subsequent text processing, classification or application.

[0161] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0162] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A Python-based medical data storage, processing and analysis intelligent platform, characterized in that: include: A data collection module is used to collect text data in the process of medical document input in real time; A determination module, connected to the data acquisition module, for determining a number of complex texts according to the text data; A word segmentation module, which is connected to the determination module and is used to segment each of the complex texts using a preset word segmentation tool at a preset segmentation granularity to form a plurality of phrases; A marking module, connected to the word segmentation module, for marking the part of speech of each phrase according to a preset marking tool to form a plurality of marked phrases; An adjustment module, connected to the annotation module, for adjusting the preset segmentation granularity according to the grammatical consistency of each annotation phrase at any adjacent time within a preset adjustment time length; A conversion module, connected to the annotation module, for converting each of the annotation phrases according to a preset word vector model to form a plurality of word vectors; A dimension reduction module, connected to the conversion module, for reducing the dimension of each word vector by a preset dimension reduction amount according to the dimensions of all word vectors within a preset determination time length, to form a plurality of dimension reduction vectors; A correction module, connected to the dimension reduction module, for correcting a preset dimension reduction amount according to the semantic similarity of any adjacent dimension reduction vectors and a preset standard similarity; A generation module, which is connected to the conversion module and the dimension reduction module respectively, and is used to generate a plurality of sentence vectors according to all the word vectors and the dimension reduction vectors; A storage module, connected to the generation module, for associating the sentence vector with the marked phrase and storing the associated sentence vector in a preset database; An analysis module is connected to the storage module and is used to analyze the sentence vector and the labeled phrase according to a preset mixed Gaussian model to form an analysis report.

2. The Python-based medical data storage, processing and analysis intelligent platform according to claim 1 is characterized in that: The determination module comprises: a vocabulary recognition unit, for recognizing different words in the text data to form a plurality of different words, and recognizing a total word in the text data to form a plurality of total words; A complexity calculation unit connected to the vocabulary recognition unit, for calculating the ratio of the number of the different words to the number of the total words to form text complexity; A distribution calculation unit connected to the complexity calculation unit, for calculating the standard deviation of the text complexity within a preset historical time period to form a complexity distribution degree; A distribution degree comparison unit, connected to the distribution calculation unit, for comparing the complex distribution degree with a preset standard distribution degree to form a distribution comparison result; A determination unit is connected to the distribution comparison unit and is used to determine a number of complex texts according to the distribution comparison result, the comparison result of the text complexity and the preset standard complexity.

3. The Python-based medical data storage, processing and analysis intelligent platform according to claim 2 is characterized in that: The word segmentation module comprises: A stop word recognition unit, used for recognizing stop words in the complex text to form a plurality of stop words; A segmentation determination unit connected to the stop word recognition unit, for determining whether to perform segmentation according to a preset stop word list and each of the stop words, and forming a segmentation determination result; The word segmentation unit is connected to the segmentation determination unit and is used to segment each of the complex texts according to the segmentation determination result, the preset segmentation granularity and the preset word segmentation tool to form a plurality of phrases.

4. The Python-based medical data storage, processing and analysis intelligent platform according to claim 3 is characterized in that: The adjustment module comprises: A sorting unit, used to sort the marked phrases according to their parts of speech to form a part of speech sequence; A consistency calculation unit connected to the sorting unit, for calculating the cosine similarity of the part-of-speech sequences at any adjacent moments to form a grammatical consistency; An adjustment unit is connected to the consistency calculation unit and is used to adjust the preset segmentation granularity according to the grammatical consistency.

5. The Python-based medical data storage, processing and analysis intelligent platform according to claim 4 is characterized in that: The adjustment unit comprises: A consistency comparison subunit, used to compare the grammatical consistency with a preset standard consistency to form a consistency comparison result; An adjustment subunit is connected to the consistency comparison subunit and is used to adjust the preset segmentation granularity according to the consistency comparison result, the preset standard consistency and a preset adjustment coefficient.

6. The Python-based medical data storage, processing and analysis intelligent platform according to claim 5 is characterized in that: The dimension reduction module comprises: A complexity calculation unit, used to calculate the variance of the dimension to form complexity; A complex fluctuation calculation unit, connected to the complexity calculation unit, for calculating the standard deviation of the complexity to form a complex fluctuation value; A dimension reduction unit is connected to the complex fluctuation calculation unit and is used to reduce the dimension of each word vector according to the complex fluctuation value to form a plurality of reduced dimension vectors.

7. The Python-based medical data storage, processing and analysis intelligent platform according to claim 6 is characterized in that: The dimension reduction unit comprises: A dimensionality reduction comparison subunit, which is connected to the complex fluctuation calculation unit and is used to compare the complex fluctuation value with a preset standard complex fluctuation value to form a dimensionality reduction comparison result; A dimensionality reduction operation subunit is connected to the dimensionality reduction comparison subunit and is used to reduce the dimensionality of each word vector according to the dimensionality reduction comparison result, a preset dimensionality reduction model and a preset dimensionality reduction quantity to form a plurality of dimensionality reduction vectors.

8. The Python-based medical data storage, processing and analysis intelligent platform according to claim 7 is characterized in that: The correction module comprises: A similarity calculation unit, used to calculate the cosine similarity of the reduced-dimensional vector to form semantic similarity; A correction unit is connected to the similarity calculation unit and is used to correct the preset dimensionality reduction amount according to a relative deviation between the semantic similarity and the preset standard similarity and a preset correction coefficient.

9. The Python-based medical data storage, processing and analysis intelligent platform according to claim 8, characterized in that: The generation module comprises: A combining unit, used for combining all the word vectors and each of the dimensionality reduction vectors by weighted averaging to form a plurality of temporary sentence vectors; A generating unit is connected to the combining unit and is used to perform regularization processing on each of the temporary sentence vectors according to a preset post-processing algorithm to form a plurality of sentence vectors.

10. The Python-based medical data storage, processing and analysis intelligent platform according to claim 9, characterized in that: The analysis module comprises: An analysis unit, configured to cluster the sentence vectors according to the preset mixed Gaussian model to form a clustering result; An extraction unit connected to the analysis unit, for extracting the part-of-speech distribution features of the marked phrase to form a feature extraction result; An output unit is connected to the analysis unit and the extraction unit respectively, and is used to output an analysis report according to the clustering result and the feature extraction result.

Images