Data comprehensive analysis method and system, electronic equipment and storage medium
By preprocessing and recursively building historical medical data, building a distributed data warehouse, performing data mining and matching, the problem that traditional analysis methods cannot integrate historical and real-time data is solved, and efficient analysis of medical data and teaching quality is achieved. promote.
Patent Information
- Application Number
- CN202411451855.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-17
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-10-17
AI Technical Summary
Traditional medical data statistical analysis methods are limited to a single data source, making it difficult to effectively integrate historical medical data with real-time data, lacking fast query and flexible analysis mechanisms, and cannot meet the teaching needs under large-scale data.
By obtaining historical medical data for preprocessing and recursive construction, a distributed data warehouse is built, data mining and matching is carried out, and the differences between data are determined to optimize the analysis and guidance of teaching data.
It improves the retrieval efficiency and teaching quality of medical data, and can quickly respond to and provide in-depth teaching insights under large-scale data.
Smart Images

Figure CN118964363A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a data comprehensive analysis method, system, electronic device and storage medium. Background Art
[0002] In the current era of big data, the medical field is also facing the challenge of managing and analyzing massive amounts of data. Traditional statistical analysis methods for medical data are often limited to processing a single data source, resulting in limited comprehensiveness and accuracy of analysis results. In addition, with the popularization of online medical care and intelligent systems, the speed and diversity of data generation are growing exponentially, making data processing and storage a major problem.
[0003] In the traditional mentor-apprentice teaching model, teaching cases are often related to the teacher's experience, resulting in insufficient cases. The existing data processing processes are mostly focused on data cleaning and simple statistical analysis, and lack of efficient integration of historical medical data and real-time data to achieve in-depth insights and personalized guidance for case-based teaching processes. The lack of effective data organization and fast query mechanisms makes it difficult to support rapid response and flexible analysis requirements under large-scale data. Summary of the invention
[0004] The problem solved by the present invention is how to analyze medical data to improve teaching effects.
[0005] To solve the above problems, the present invention provides a data comprehensive analysis method, comprising: Acquire historical medical data of each data node, and pre-process the historical medical data to obtain first standard data; Recursively construct the first standard data according to a preset construction method to obtain storage data and storage data index; For each of the data nodes, at least one data warehouse is constructed according to the stored data and the stored data index, and all the data warehouses are used as distributed data warehouses; Acquiring actual data, and performing the preprocessing on the actual data to obtain second standard data; Performing data mining on all the stored data in the distributed data warehouse to obtain mining data; Using a preset method, the mined data that matches the second standard data is used as matching data; The difference between the matching data and the actual data is determined according to the matching data.
[0006] Optionally, the acquiring historical medical data of each data node, preprocessing the historical medical data, and obtaining the first standard data includes: Cleaning and standardizing the historical medical data, and converting the data into a first data vector; The set of the first data vectors is used as the first standard data.
[0007] Optionally, recursively constructing the first standard data according to a preset construction method to obtain storage data and storage data indexes includes: Randomly select a dimension and a split point from the first standard data; According to the dimension and the segmentation point, the first data vector is segmented into two first-level groups to form a first node, wherein the first-level group is a direct child node of the first node; In each of the first-level groups under the first node, a new dimension and a new split point are randomly selected respectively, and each of the first-level groups is further split into two second-level groups; Select a new dimension and a new split point, split the secondary group until the termination condition is met, complete the recursive construction, use the first standard data after the recursive construction as the storage data, and use the structure composed of all groups and all nodes obtained by the recursive construction as the storage data index, wherein the termination condition includes: there is only one first data vector in each n-level group, or, the preset number of splits is reached.
[0008] Optionally, performing data mining on all the stored data in the distributed data warehouse to obtain mining data includes: According to the stored data index, the approximate nearest neighbor of the second standard data is retrieved in the distributed data warehouse, wherein if there are multiple data warehouses, and the similarity between the type of the historical medical data in the multiple data warehouses and the type of the second standard data exceeds a first similarity threshold, the approximate nearest neighbors of the second standard data in the multiple data warehouses are searched in parallel; The approximate nearest neighbor is used as the mining data.
[0009] Optionally, the using the mined data matching the second standard data as matching data by a preset method includes: Constructing a co-occurrence matrix by a co-occurrence analysis method, wherein the co-occurrence matrix is used to record the number of co-occurrences of the mining data and the second standard data; Determining a first co-occurrence frequency and a first co-occurrence similarity between the mined data and the second standard data in the co-occurrence matrix; The mined data whose first co-occurrence frequency is higher than a frequency threshold and whose first co-occurrence similarity is higher than a second similarity threshold is used as the matching data.
[0010] Optionally, after constructing the co-occurrence matrix by the co-occurrence analysis method, the method further includes: Determining, in the co-occurrence matrix, a second co-occurrence frequency and a second co-occurrence similarity of all the stored data and the second standard data; taking the stored data, which has the second co-occurrence frequency higher than the frequency threshold and the second co-occurrence similarity higher than the second similarity threshold and does not exist in the matching data, as optimization data; The structure of the co-occurrence matrix is optimized based on the second co-occurrence frequency and the second co-occurrence similarity corresponding to the optimization data.
[0011] Optionally, determining the difference between the matching data and the actual data according to the matching data includes: Parsing the matching data to obtain multi-dimensional data contained in the matching data, wherein the multi-dimensional data includes a user code, a doctor code, an evaluation result, and a target completion degree; Obtaining the factor similarity between each dimension of the multi-dimensional data through the Pearson correlation coefficient matrix; taking the multi-dimensional data whose factor similarity is greater than a third similarity threshold as target data; Determine the factors and eigenvalues of the target data by using the minimum residual method, and retain the factors with eigenvalues greater than 1 as target factors; Rotate the target factor to obtain a factor loading matrix; Determining, according to the factor loading matrix, multi-dimensional data in the second standard data that is lower than the first preset score as low-score data; Extracting multi-dimensional data with a score higher than a second preset score from the matching data as high-score data; The low-scoring data is optimized according to the high-scoring data until the score of the low-scoring data is higher than a third preset score, wherein the first preset score is less than or equal to the third preset score, and the third preset score is less than or equal to the second preset score.
[0012] In a second aspect, the present invention further provides a data comprehensive analysis system, comprising: A first acquisition module, used to acquire historical medical data of each data node, pre-process the historical medical data, and obtain first standard data; A first construction module, used for recursively constructing the first standard data according to a preset construction method to obtain storage data and storage data index; A second construction module is used to construct at least one data warehouse for each of the data nodes according to the stored data and the stored data index, and use all the data warehouses as distributed data warehouses; A second acquisition module is used to acquire actual data, perform the preprocessing on the actual data, and obtain second standard data; A mining module, used for performing data mining on all the stored data in the distributed data warehouse to obtain mining data; A matching module, used to use the mined data that matches the second standard data as matching data by a preset method; The optimization module is used to determine the difference between the matching data and the actual data according to the matching data.
[0013] In a third aspect, the present invention further provides an electronic device, comprising a memory and a processor; The memory is used to store computer programs; The processor is used to implement the data comprehensive analysis method as described above when executing the computer program.
[0014] In a fourth aspect, the present invention further provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the data comprehensive analysis method as described above is implemented.
[0015] Compared with the prior art, the present invention takes into account the storage and retrieval of actual data, stores historical medical data through a distributed storage system, pre-processes the historical medical data, and obtains the first standard data as the basis for optimizing medical data. For high-data-volume, distributed medical data storage scenarios, the first standard data is recursively constructed to obtain storage data and storage data indexes, and multiple data warehouses are constructed according to the number of data nodes, which can effectively utilize the advantages of distributed storage and improve retrieval efficiency. Data mining is performed on the storage data of the data warehouse, and it is matched with the second standard data to obtain matching data. The actual data is optimized based on the matching data. When the medical data is distributed in different storage locations, the matching data that matches the second standard data can be quickly retrieved using the data index, and the difference between the matching data and the actual data can be found based on the matching data. By mining and analyzing previous data, it is used to improve the teaching quality for inexperienced users. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 Schematic diagram of the process of the comprehensive data analysis method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a detailed flow chart of step S100 of the data comprehensive analysis method according to an embodiment of the present invention; Figure 3 Schematic diagram of a detailed flow chart of step S600 of the data comprehensive analysis method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.
[0018] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.
[0019] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0021] An embodiment of the present invention provides a data comprehensive analysis method, comprising: Step S100, acquiring historical medical data of each data node, and preprocessing the historical medical data to obtain first standard data.
[0022] In one embodiment, a data node represents a component responsible for storing actual data. For example, when historical medical data is stored in multiple hospitals or multiple departments, each hospital or department serves as a data node. Historical medical data is stored through a storage medium, wherein the historical medical data includes past teaching data that can be recorded, such as audio data, video data, text data, etc., specifically data such as diagnostic videos, diagnostic audio, teaching courseware, teaching handouts, and textbook contents. Teaching tasks and diagnostic history are used as storage indexes as historical medical data. For example, all public data such as diagnostic videos, diagnostic audio, teaching courseware, teaching handouts, and textbook contents on January 1, 2000 are used as a piece of historical medical data. By using historical medical data as a database for teaching, through case analysis, cases matching the current actual data are obtained, which are used to analyze the differences between the two, and provide more case guidance for master-apprentice teaching.
[0023] Perform pre-processing such as data screening, deduplication, and noise reduction on historical medical data to obtain the first standard data. For example, delete unrecognizable teaching courseware and teaching handout data; delete duplicate audio data when there is duplicate content in audio data and video data; perform noise reduction on noisy audio data, and perform noise reduction on blurry image data or video data. The first standard data obtained by processing is the standardized data stored in the data warehouse.
[0024] Step S200: recursively construct the first standard data according to a preset construction method to obtain storage data and storage data index.
[0025] In one embodiment, since the historical medical data comes from a wide range of sources, such as various units or other types of nodes, and some nodes may not be convenient to disclose the historical medical data, it is necessary to store the data and construct a data index for retrieval under each node. For distributed storage of historical medical data, improving retrieval efficiency is of great significance, and constructing a suitable data index will greatly improve retrieval efficiency.
[0026] Step S300: for each of the data nodes, at least one data warehouse is constructed according to the stored data and the stored data index, and all the data warehouses are used as distributed data warehouses.
[0027] In one embodiment, some data nodes have a large amount of data and need to store the stored data separately to improve retrieval efficiency. Therefore, for each data node, at least one data warehouse is constructed and all data warehouses are used as distributed data warehouses.
[0028] Step S400, acquiring actual data, and performing the preprocessing on the actual data to obtain second standard data.
[0029] Acquire actual data, standardize the data, and obtain second standard data that is consistent with the format of the stored data. The two have the same format and are used to assist in comparing the differences between actual data and historical medical data, and optimize the actual data based on the differences, so as to realize a teaching optimization method based on data analysis.
[0030] Step S500: performing data mining on all the stored data in the distributed data warehouse to obtain mining data.
[0031] Analyze and mine the stored data in the distributed data warehouse to determine the characteristics of the stored data. After obtaining the actual data, the matching data can be quickly retrieved and determined based on the data index and the characteristics of the stored data.
[0032] Step S600: using a preset method to take the mined data that matches the second standard data as matching data.
[0033] Step S700: determining differences between the matching data and the actual data based on the matching data.
[0034] After determining the matching data that matches the second standard data from the distributed data warehouse, the shortcomings of the actual data are optimized according to the advantages of the matching data to form an effective optimization method to improve the teaching quality. In the traditional master-apprentice teaching model, the teaching effect and teaching examples are related to the teacher's experience reserves. Through the data processing method in the embodiment of the present invention, historical medical data similar to the current case can be mined from the database and determined as matching data, and the differences and similarities between it and the actual data can be extracted for teaching, making the cases richer and effectively improving the teaching quality.
[0035] Alternatively, if Figure 2 As shown, the acquiring of historical medical data of each data node, preprocessing of the historical medical data, and obtaining the first standard data include: Step S110, cleaning and standardizing the historical medical data, and converting them into a first data vector.
[0036] Step S120: taking the set of the first data vectors as the first standard data.
[0037] In one embodiment, first, public historical medical data is collected from sources such as information systems and online platforms. These data include but are not limited to basic information (such as age, basic personal information), etc. The data exists in structured (such as database tables) and unstructured (such as text, image data) forms. Clean the data, including: Deduplication: Identify and remove duplicate records to ensure the uniqueness of each observation point. Missing value processing: For missing data, adopt appropriate strategies to handle, such as using the mean and median to fill in numerical data, or fill in categorical data based on reasonable inference of contextual logic. Outlier detection and processing: Identify and process outliers through statistical analysis (such as box plot analysis), and you can choose to remove, correct or use more complex statistical methods (such as winsorization) for smoothing. Consistency check: Ensure that the data field format is consistent, such as standardizing the date format, unifying the case of text data, and removing irrelevant symbols.
[0038] After data cleaning, the data is standardized to eliminate the dimension effect, so that different features are comparable and convenient for subsequent analysis. Common standardization methods are: Minimum-maximum normalization: map all eigenvalues to the interval [0,1]. Z-Score normalization: make the data conform to the standard normal distribution, that is, each eigenvalue minus its mean and divided by its standard deviation. For the historical medical data mentioned above, select the appropriate normalization method according to the characteristics and process each feature independently.
[0039] After the standardization process is completed, each data record can be converted into a fixed-length data vector, in which each element represents the standardized value of a feature. For example, if the teaching data contains three features, each data record will be converted into a data vector of the form [0.65, 0.92, 0.78].
[0040] All individual data vectors are aggregated to form a set, which is the "first standard data". This set is the basis for subsequent data analysis, ensuring that all data are processed in a unified scale and format, making it easier for algorithms to understand and process, and reducing deviations caused by data inconsistency.
[0041] Optionally, recursively constructing the first standard data according to a preset construction method to obtain storage data and storage data indexes includes: Randomly select a dimension and a split point from the first standard data; According to the dimension and the segmentation point, the first data vector is segmented into two first-level groups to form a first node, wherein the first-level group is a direct child node of the first node; In each of the first-level groups under the first node, a new dimension and a new split point are randomly selected respectively, and each of the first-level groups is further split into two second-level groups; Select a new dimension and a new split point, split the secondary group until the termination condition is met, complete the recursive construction, use the first standard data after the recursive construction as the storage data, and use the structure composed of all groups and all nodes obtained by the recursive construction as the storage data index, wherein the termination condition includes: there is only one first data vector in each n-level group, or, the preset number of splits is reached.
[0042] Determine the preset parameters for recursively constructing a decision tree, including but not limited to the maximum recursive depth (preset number of splits), whether to perform pruning, etc. These parameters are set according to the characteristics of historical medical data and the analysis objectives to avoid overfitting or underfitting. Randomly select a feature dimension (attribute) and a specific value on the dimension from the first standard data set as a split point. This split point usually selects the value that increases the purity of the data set the most after splitting, but in this example, it is directly randomly selected for simplicity. Based on the selected dimension and split point, the data vectors in the first standard data set are divided into two subsets, namely two first-level groups (n=1 at this time). These two subsets will serve as direct child nodes of the current node (i.e., the first node). For each first-level group under the first node, repeat the above process: randomly select a new dimension from the remaining unused features, and select a split point on this dimension to continue to split the first-level group into two second-level groups (n=2 at this time). The recursion continues, and the process of selecting dimensions and split points is repeated in the subsets within the current group each time until the termination condition is reached. Termination conditions include: Each group contains only one data vector: This indicates that no effective segmentation can be performed because there is no more data to further distinguish. The preset number of segmentations is reached: This is a mechanism to prevent over-segmentation, ensure that the depth of the tree is within a reasonable range, and improve generalization ability. Once any subgroup meets the above termination conditions, the subgroup is no longer segmented and becomes a leaf node. The leaf node represents the final decision area, and the data vectors contained in it have similar characteristic attributes. During the entire recursive construction process, each node (including internal nodes and leaf nodes) should store its corresponding segmentation dimension, segmentation point information, and pointers (or indexes) to its child nodes. In addition, the leaf node also needs to store the index or direct data of all the first data vectors it contains for subsequent query and interpretation. This decision tree can be used for tasks such as classification and regression to help analyze patterns and laws in historical medical data.
[0043] Optionally, recursively constructing the first standard data according to a preset construction method to obtain storage data and storage data indexes includes: For each of the data nodes, the first standard data is recursively constructed according to the preset construction method to obtain the storage data index corresponding to the data node.
[0044] In one embodiment, a data node represents each data storage entity in a distributed data system, and they store data in a dispersed manner on different servers or devices in the system. This distributed storage method improves the accessibility of data and the overall fault tolerance of the system.
[0045] In a distributed storage system, there are multiple data nodes, and each data node has its own storage data index.
[0046] Optionally, performing data mining on all the stored data in the distributed data warehouse to obtain mining data includes: According to the stored data index, the approximate nearest neighbor of the second standard data is retrieved in the distributed data warehouse, wherein if there are multiple data warehouses, and the similarity between the type of the historical medical data in the multiple data warehouses and the type of the second standard data exceeds a first similarity threshold, the approximate nearest neighbors of the second standard data in the multiple data warehouses are searched in parallel; The approximate nearest neighbor is used as the mining data.
[0047] Since the stored data has multiple dimensions (images, text vectors), for high-dimensional data that needs to be retrieved, a distributed approximate nearest neighbor algorithm is used to build an index on each node. This step requires that the data be distributed to different nodes, and each node is responsible for index construction and query processing of a portion of the data to improve efficiency and scalability. In an embodiment of the present invention, the first similarity threshold indicates that the types between the historical medical data and the second standard data have a high degree of similarity. For example, the semantics of the historical medical data and the second standard data are calculated. When the similarity of the types represented by the semantics of the two exceeds the first similarity threshold, multiple data warehouses are searched in parallel for the approximate nearest neighbors of the second standard data. In other embodiments, the similarity can also be the similarity between the data types of the historical medical data and the second standard data. The first similarity threshold can be set according to the data type or the semantic type.
[0048] Using the constructed storage data index, quickly find the approximate nearest neighbor of the current data in the historical medical data, that is, the approximate nearest neighbor of the second standard data in the storage data. When there are multiple storage nodes with similar storage data, this step can be processed in parallel, and each node is responsible for processing a part of the query request. For example, if the data warehouses of the first unit and the second unit both have storage data similar to the second standard data, the nodes corresponding to the first unit and the nodes corresponding to the second unit are searched in parallel to speed up the retrieval efficiency. The approximate nearest neighbor is used as the mining data.
[0049] Alternatively, if Figure 3 As shown, the method of using the mined data that matches the second standard data as matching data by a preset method includes: Step S610: construct a co-occurrence matrix by using a co-occurrence analysis method, wherein the co-occurrence matrix is used to record the number of co-occurrences of the mining data and the second standard data.
[0050] Step S620: determining a first co-occurrence frequency and a first co-occurrence similarity between the mined data and the second standard data in the co-occurrence matrix.
[0051] Step S630: taking the mined data whose first co-occurrence frequency is higher than a frequency threshold and whose first co-occurrence similarity is higher than a second similarity threshold as the matching data.
[0052] The co-occurrence analysis method is used to construct a co-occurrence matrix, whose rows and columns represent different dimensions in the data set, and each element in the matrix represents the co-occurrence frequency or number of times between the corresponding items. For example, if item A and item B appear N times in the same context, the value of position (A, B) in the matrix is N. When processing large-scale mining data, sparse matrix storage can be used to reduce space complexity.
[0053] Based on the co-occurrence matrix, the relationship between the mined data and the second standard data is quantified. The first co-occurrence frequency is directly read from the matrix, that is, the frequency of co-occurrence of the mined data and the second standard data. The first co-occurrence similarity may need to be further calculated, such as measuring the strength of the association between the two through the Jaccard similarity coefficient, cosine similarity or Pearson correlation coefficient. This step aims to gain a deeper understanding of the association pattern between different data items. Two key thresholds are set: frequency threshold and second similarity threshold. Each mined data in the co-occurrence matrix is evaluated, and only when its co-occurrence frequency with the second standard data exceeds the frequency threshold and the co-occurrence similarity exceeds the second similarity threshold, the mined data will be marked as matching data. This step ensures that the selected matching data are not only frequently co-occurring, but also have a high correlation in semantics or behavioral patterns.
[0054] Optionally, after constructing the co-occurrence matrix by the co-occurrence analysis method, the method further includes: A second co-occurrence frequency and a second co-occurrence similarity of all the stored data and the second standard data are determined in the co-occurrence matrix.
[0055] The stored data having the second co-occurrence frequency higher than the frequency threshold and the second co-occurrence similarity higher than the second similarity threshold and not existing in the matching data is used as the optimization data.
[0056] The structure of the co-occurrence matrix is optimized based on the second co-occurrence frequency and the second co-occurrence similarity corresponding to the optimization data.
[0057] Based on the co-occurrence matrix, the relationship between all stored data items and the second standard data is retrieved again. The second co-occurrence frequency refers to the number of times the stored data co-occurs with the second standard data. It is used to retrieve similar data that is missed based on the mining data matching. Due to the screening settings of the mining data, the stored data may be ignored due to excessive strictness. These missed stored data are defined as optimized data, which may contain important potential association information. Co-occurrence similarity represents a measure of the strength of the relationship between a pair of words calculated based on the co-occurrence matrix. When a certain data correlation between the stored data and the second standard data is high, the co-occurrence frequency and co-occurrence similarity of the two are correspondingly high. Optionally, the co-occurrence similarity can be calculated based on an existing similarity algorithm, such as point mutual information, cosine similarity, etc.
[0058] The process is designed as an iterative cycle. After each optimization, the performance of the matrix is re-evaluated and the optimization strategy is adjusted based on the feedback until a satisfactory analysis result is achieved.
[0059] Through the above steps, the efficiency and accuracy of subsequent data analysis can be improved by optimizing the co-occurrence matrix structure, laying a solid foundation for in-depth insights into the complex connections behind the data.
[0060] Optionally, determining the difference between the matching data and the actual data according to the matching data includes: Parsing the matching data to obtain multi-dimensional data contained in the matching data, wherein the multi-dimensional data includes a user code, a doctor code, an evaluation result, and a target completion degree; Obtaining the factor similarity between each dimension of the multi-dimensional data through the Pearson correlation coefficient matrix; taking the multi-dimensional data whose factor similarity is greater than a third similarity threshold as target data; Determine the factors and eigenvalues of the target data by using the minimum residual method, and retain the factors with eigenvalues greater than 1 as target factors; Rotate the target factor to obtain a factor loading matrix; Determining, according to the factor loading matrix, multi-dimensional data in the second standard data that is lower than the first preset score as low-score data; Extracting multi-dimensional data with a score higher than a second preset score from the matching data as high-score data; The low-scoring data is optimized according to the high-scoring data until the score of the low-scoring data is higher than a third preset score, wherein the first preset score is less than or equal to the third preset score, and the third preset score is less than or equal to the second preset score.
[0061] Among them, the evaluation results include the objective evaluation of the diagnosis process, which is filled in manually; the target completion includes the prognosis evaluation of the diagnosed person.
[0062] In one embodiment, according to the factor analysis method, the association between the dimensions in the matching data is analyzed, and then the multi-dimensional data in the actual data is optimized according to the association pattern. First, the matching data is parsed, and the factor similarity of the multi-dimensional data in the matching data is calculated according to the Pearson correlation coefficient matrix. The multi-dimensional data with a similarity greater than the third similarity threshold is used as the target data, indicating that the data has a certain correlation with the data of other dimensions, while the multi-dimensional data that has no correlation with the data of other dimensions is not very helpful for optimization. The factors and eigenvalues of the target data are determined by the minimum residual method, and the factors with eigenvalues greater than 1 are used as target factors to optimize the number of factors.
[0063] In order to improve the interpretability of the factors, the target factors are rotated. Rotation methods include orthogonal rotation (such as Varimax) and oblique rotation (such as Oblique). The purpose of rotation is to make each target data have a high load on as few factors as possible, so that the factors are easier to interpret. The scores of the multidimensional data of the second standard data are evaluated according to the factor loading matrix, and the multidimensional data in the matching data are scored to obtain low-score data and high-score data respectively. The low-score data is optimized by the high-score data to optimize the actual data according to the historical medical data. The first preset score, the second preset score, and the third preset score respectively represent the preset score thresholds, which are used to compare with the absolute value (i.e., score) of the factor loading of the second standard data in the factor loading matrix. The first preset score is used to filter out the low-score data in the second standard data, the second preset score is used to filter out the high-score data in the matching data, and the third standard data is used as the minimum score standard for the second standard data after optimization. For example, when the absolute value of the factor loading of the second standard data is less than the first preset score, the second standard data is low-score data and needs to be optimized by the high-score data in the matching data to re-excavate the second standard data with a higher matching degree.
[0064] An embodiment of the present invention provides a data comprehensive analysis system, including: A first acquisition module, used to acquire historical medical data of each data node, pre-process the historical medical data, and obtain first standard data; A first construction module, used for recursively constructing the first standard data according to a preset construction method to obtain storage data and storage data index; A second construction module is used to construct at least one data warehouse for each of the data nodes according to the stored data and the stored data index, and use all the data warehouses as distributed data warehouses; A second acquisition module is used to acquire actual data, perform the preprocessing on the actual data, and obtain second standard data; A mining module, used for performing data mining on all the stored data in the distributed data warehouse to obtain mining data; A matching module, used to use the mined data that matches the second standard data as matching data by a preset method; The optimization module is used to determine the difference between the matching data and the actual data according to the matching data.
[0065] Optionally, the data comprehensive analysis system also includes an image input device, an audio input device, and a display device.
[0066] Another embodiment of the present invention provides an electronic device, including a memory and a processor; the memory is used to store a computer program; the processor is used to implement the data comprehensive analysis method as described above when executing the computer program.
[0067] A computer program is stored on a computer-readable storage medium provided in another embodiment of the present invention. When the computer program is executed by a processor, the data comprehensive analysis method as described above is implemented.
[0068] An electronic device that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0069] The electronic device includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM, and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.
[0070] A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on the respective computers and having a client-server relationship to each other.
[0071] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. In the present application, the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, which may be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present invention. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
[0072] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.
Claims
1. A method for comprehensive data analysis, It is characterized in that include: Acquire historical medical data of each data node, and pre-process the historical medical data to obtain first standard data; Recursively construct the first standard data according to a preset construction method to obtain storage data and storage data index; For each of the data nodes, at least one data warehouse is constructed according to the stored data and the stored data index, and all the data warehouses are used as distributed data warehouses; Acquiring actual data, and performing the preprocessing on the actual data to obtain second standard data; Performing data mining on all the stored data in the distributed data warehouse to obtain mining data; Using a preset method, the mined data that matches the second standard data is used as matching data; The difference between the matching data and the actual data is determined according to the matching data.
2. The data comprehensive analysis method according to claim 1, It is characterized in that The acquiring of historical medical data of each data node, preprocessing the historical medical data, and obtaining first standard data comprises: Cleaning and standardizing the historical medical data, and converting the data into a first data vector; The set of the first data vectors is used as the first standard data.
3. The data comprehensive analysis method according to claim 2, It is characterized in that The recursive construction of the first standard data according to a preset construction method to obtain storage data and storage data index comprises: Randomly select a dimension and a split point from the first standard data; According to the dimension and the segmentation point, the first data vector is segmented into two first-level groups to form a first node, wherein the first-level group is a direct child node of the first node; In each of the first-level groups under the first node, a new dimension and a new split point are randomly selected respectively, and each of the first-level groups is further split into two second-level groups; Select a new dimension and a new split point, split the secondary group until the termination condition is met, complete the recursive construction, use the first standard data after the recursive construction as the storage data, and use the structure composed of all groups and all nodes obtained by the recursive construction as the storage data index, wherein the termination condition includes: there is only one first data vector in each n-level group, or, the preset number of splits is reached.
4. The data comprehensive analysis method according to claim 1, It is characterized in that The performing data mining on all the stored data in the distributed data warehouse to obtain the mining data comprises: According to the stored data index, the approximate nearest neighbor of the second standard data is retrieved in the distributed data warehouse, wherein if there are multiple data warehouses, and the similarity between the type of the historical medical data in the multiple data warehouses and the type of the second standard data exceeds a first similarity threshold, the approximate nearest neighbors of the second standard data in the multiple data warehouses are searched in parallel; The approximate nearest neighbor is used as the mining data.
5. The data comprehensive analysis method according to claim 1, It is characterized in that The method of using the mined data that matches the second standard data as matching data by a preset method includes: Constructing a co-occurrence matrix by a co-occurrence analysis method, wherein the co-occurrence matrix is used to record the number of co-occurrences of the mining data and the second standard data; Determining a first co-occurrence frequency and a first co-occurrence similarity between the mined data and the second standard data in the co-occurrence matrix; The mined data whose first co-occurrence frequency is higher than a frequency threshold and whose first co-occurrence similarity is higher than a second similarity threshold is used as the matching data.
6. The data comprehensive analysis method according to claim 5, It is characterized in that After constructing the co-occurrence matrix by the co-occurrence analysis method, the method further includes: Determining, in the co-occurrence matrix, a second co-occurrence frequency and a second co-occurrence similarity of all the stored data and the second standard data; taking the stored data, which has the second co-occurrence frequency higher than the frequency threshold and the second co-occurrence similarity higher than the second similarity threshold and does not exist in the matching data, as optimization data; The structure of the co-occurrence matrix is optimized based on the second co-occurrence frequency and the second co-occurrence similarity corresponding to the optimization data.
7. The data comprehensive analysis method according to claim 1, It is characterized in that Determining the difference between the matching data and the actual data according to the matching data includes: Parsing the matching data to obtain multi-dimensional data contained in the matching data, wherein the multi-dimensional data includes a user code, a doctor code, an evaluation result, and a target completion degree; Obtaining the factor similarity between each dimension of the multi-dimensional data through the Pearson correlation coefficient matrix; taking the multi-dimensional data whose factor similarity is greater than a third similarity threshold as target data; Determine the factors and eigenvalues of the target data by using the minimum residual method, and retain the factors with eigenvalues greater than 1 as target factors; Rotate the target factor to obtain a factor loading matrix; Determining, according to the factor loading matrix, multi-dimensional data in the second standard data that is lower than the first preset score as low-score data; Extracting multi-dimensional data with a score higher than a second preset score from the matching data as high-score data; The low-scoring data is optimized according to the high-scoring data until the score of the low-scoring data is higher than a third preset score, wherein the first preset score is less than or equal to the third preset score, and the third preset score is less than or equal to the second preset score.
8. A data comprehensive analysis system, It is characterized in that include: A first acquisition module, used to acquire historical medical data of each data node, pre-process the historical medical data, and obtain first standard data; A first construction module, used for recursively constructing the first standard data according to a preset construction method to obtain storage data and storage data index; A second construction module is used to construct at least one data warehouse for each of the data nodes according to the stored data and the stored data index, and use all the data warehouses as distributed data warehouses; A second acquisition module is used to acquire actual data, perform the preprocessing on the actual data, and obtain second standard data; A mining module, used for performing data mining on all the stored data in the distributed data warehouse to obtain mining data; A matching module, used to use the mined data that matches the second standard data as matching data by a preset method; The optimization module is used to determine the difference between the matching data and the actual data according to the matching data.
9. An electronic device, It is characterized in that including memory and processor; The memory is used to store computer programs; The processor is used to implement the data comprehensive analysis method as described in any one of claims 1-7 when executing the computer program.
10. A readable storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the data comprehensive analysis method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Data warehouse information processing method, device, system and medium
CN109388637A
Clinical data mining analysis and aided decision-making method based on Internet integrated medical platform
CN109830303A
Multi-label classification method for obtaining customer intention labels through robot collection
CN112732871A
Employment platform comment mining method and system
CN113111187A
Population standard address matching method and system based on analytical data warehouse
CN117235102A
Cited By
Operation duration prediction and scheduling method based on machine learning
CN120183638A
