Medicine data anomaly detection and automatic acquisition method

Abstract

The invention provides a medical data anomaly detection and automatic acquisition method, and aims to overcome the defects of single anomaly detection accuracy, response speed and data acquisition strategy in the prior art. According to the method, time-frequency characteristics of medical data are extracted through wavelet transform, and a self-adaptive multi-scale anomaly detection model is constructed by using a long-short-term memory network, so that a time sequence dependency relationship of the data is learned and parameters are dynamically adjusted; the intelligent acquisition strategy optimizer automatically adjusts an acquisition strategy based on a reinforcement learning technology to adapt to new medical data changes. According to the invention, the accuracy and real-time performance of anomaly detection are improved, the data acquisition strategy is optimized, and the intelligent level of medical data management is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a method for detecting and automatically collecting abnormal medical data. Background Art

[0002] In the field of medical data management, traditional technologies have significant limitations in anomaly detection and data collection. These technologies are usually based on static thresholds and simple statistical analysis, lack a deep understanding of the temporal dependencies of data, and cannot adapt to dynamic changes in data, resulting in limited accuracy and real-time performance of anomaly detection. In addition, traditional data collection strategies lack intelligence and cannot dynamically adjust the collection frequency and scope based on anomaly detection results, limiting the ability to quickly respond to and handle abnormal situations. Summary of the invention

[0003] 1. Technical issues to be solved

[0004] In view of the deficiencies in the prior art, the present invention provides a method for detecting and automatically collecting anomalies in medical data, which solves the problems of insufficient accuracy, slow response speed and single data collection strategy in the existing anomaly detection methods for medical data.

[0005] (II) Technical solution

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for detecting and automatically collecting abnormal medical data, comprising:

[0007] S1. Process the collected medical data and use wavelet transform to extract the time-frequency characteristics of the medical time series data to provide input for anomaly detection;

[0008] S2. Combined with the extracted time-frequency features, a long short-term memory network is used to train an adaptive multi-scale anomaly detection model, which can learn the temporal dependencies of data and dynamically adjust the sensitivity parameters of anomaly detection according to the statistical characteristics of historical data and the changing trends of real-time data;

[0009] S3. Use the adaptive multi-scale anomaly detection model to perform anomaly detection on medical data and identify abnormal data points; analyze the output of the adaptive multi-scale anomaly detection model to determine the specific information of the abnormal data points and their impact;

[0010] S4. Based on the abnormal results and data value assessment, the intelligent collection strategy optimizer is used to automatically adjust the collection strategy. The intelligent collection strategy optimizer evaluates the collection frequency, range and accuracy of different collection strategies based on reinforcement learning;

[0011] S5. Execute the adjusted collection strategy, collect new data, and feed the new data back to the adaptive multi-scale anomaly detection model for model update and sensitivity parameter adjustment;

[0012] S6. Record the collected data and its anomaly detection results in the database to ensure the integrity and traceability of the data.

[0013] The collected medical data include drug sales records, inventory, and patient feedback. The collected data are preprocessed, including removing missing values ​​and duplicate records. For example, if there are multiple duplicate entries of the same drug in the drug sales record, the latest sales data is identified and retained, and the old duplicate entries are deleted. The data is standardized and the coding and measurement standards are unified. For example, the different names for the same drug in different regions are unified into a standard name to ensure data consistency. The wavelet transform algorithm is applied to extract the time-frequency characteristics of time series data. By selecting the wavelet function and the number of decomposition layers, each time series data is subjected to wavelet decomposition to decompose the data into approximate coefficients and detail coefficients of different scales. For example, for the time series data of drug inventory, the wavelet transform decomposes the data into approximate coefficients reflecting long-term trends and detail coefficients capturing short-term fluctuations. The approximate coefficients represent the pattern of seasonal changes in inventory, while the detail coefficients represent the rapid changes in inventory caused by a certain event. After completing the wavelet decomposition, these data are converted into feature vectors to provide time-frequency characteristics as input for the anomaly detection model.

[0014] A multi-layer LSTM network structure is designed to deeply explore the long-term dependencies in medical data. For example, considering that drug sales are affected by holidays, seasonal factors or market activities, LSTM learns these time series patterns. During the training phase, the time-frequency features of historical medical data are used as input, and supervised learning is used to allow LSTM to recognize the patterns of normal data. LSTM dynamically adjusts its sensitivity parameters according to the changing trend of real-time data. For example, when abnormal fluctuations in the data are detected, its attention to recent data is increased to capture anomalies more accurately. A fully connected layer is added after the LSTM network, which converts the output of LSTM into anomaly scores, which are used to quantify the degree to which each data point deviates from the normal pattern and improve the ability to identify anomalies.

[0015] The adaptive multi-scale anomaly detection model can conduct in-depth analysis of medical data and identify anomalies. The model receives preprocessed time series data and its time-frequency features, which are input into LSTM. LSTM analyzes each data point by learning the temporal dependencies in the data and outputs an anomaly score, which quantifies the deviation of the data point from the normal pattern. For example, if the daily sales volume of a drug suddenly surges, the magnitude of this surge will be captured by the LSTM network and a higher anomaly score will be given. A dynamic threshold is set based on statistical analysis of historical data. The threshold takes into account the statistical characteristics of the data and is adjusted according to business needs. For example, the normal fluctuation range of drug sales during holidays will be larger than usual, so the threshold It will also be adjusted accordingly. Data points exceeding the threshold are marked as anomalies, and the output anomaly scores are sorted to identify the data points with the highest scores. The contextual information of these data points, such as the time and place of occurrence, the drugs involved, etc., will be further used to identify abnormal patterns and trends. For example, if the inventory of a certain drug in a certain area suddenly drops and coincides with the time of a disease outbreak in the area, this trend indicates that there are problems in the supply chain or a public health event. The threshold assumes that under normal circumstances, the standard deviation of the daily sales of a certain drug is 10 boxes. The threshold is set to the average sales plus 2 times the standard deviation, that is, if the average sales is 100 boxes, then the threshold is 120 boxes. Any sales exceeding this threshold will be marked as a potential anomaly.

[0016] The intelligent collection strategy optimizer adjusts the collection strategy through reinforcement learning to adapt to data changes and improve the efficiency and accuracy of anomaly detection. First, a reinforcement learning agent is built to learn the optimal collection strategy by interacting with the medical data collection environment. The agent's decision is based on multiple parameters of the current environment, including the frequency, scope, accuracy of the collection, and historical anomaly detection results, which together form the basis for the agent's decision. The agent performs multiple possible strategy adjustments, such as increasing the frequency of data collection in a specific time period or expanding the scope of data collection. By designing a reward mechanism, the agent receives positive feedback when it successfully captures abnormal data, and negative feedback when it fails, thereby guiding the agent to learn effective collection strategies. Over time, the agent learns what actions to take in specific situations to maximize rewards through continuous trial and error. For example, if the agent finds that increasing the frequency of monitoring a specific drug inventory during the flu season can more effectively identify inventory shortages, it will automatically adjust its strategy in the future to deal with similar situations. Through continuous learning and adjustment, the optimizer identifies the most effective collection strategy in a specific situation and adjusts the actual collection strategy accordingly, ensuring that the collection strategy is closely related to the anomaly detection results and can respond to changes in the data in a timely manner.

[0017] The adjusted collection strategy will be executed, and the newly collected data will be transmitted back to the adaptive multi-scale anomaly detection model in real time for further learning of the model and adjustment of sensitivity parameters; new data is input into the model, and the model re-evaluates and adjusts its internal parameters according to the actual situation of these data points to better capture the temporal dependencies in the data; according to the statistical characteristics of the new data, the sensitivity parameters of anomaly detection are dynamically adjusted, including adjusting the threshold or redefining the calculation method of the anomaly score; for example, if the newly collected drug sales data shows that the sales volume of a certain drug has abnormally surged in a specific period of time, these data will be quickly input into the LSTM, and the LSTM will analyze these data points and re-evaluate its internal parameters, such as adjusting the number of neurons or the learning rate of the hidden layer, to more accurately capture the temporal dependencies in the data; according to the statistical characteristics of the new data, the sensitivity parameters of anomaly detection are dynamically adjusted. For example, if the new data shows that the threshold for abnormal sales volume needs to be increased from twice the average sales volume to three times, the LSTM will automatically adjust to reduce false alarms and improve detection accuracy.

[0018] The integration of the collected new data and anomaly detection results is implemented through the data processing functions in the programming language, such as in Python, using dictionaries or objects to encapsulate this information; specifically, a dictionary is created for each data point, which contains key-value pairs, such as timestamp corresponding to the timestamp, data_value corresponding to the data value, anomaly_flag corresponding to the Boolean value of whether it is abnormal, anomaly_score corresponding to the anomaly score, and version corresponding to the version number; for example, a record is represented as {"timestamp":"2024-12-11T10:00:00Z","data_value":120,"anomaly_flag":true,"anomaly_score":85,"version":"v1"}; these dictionaries are then converted into the format received by the database, such as insert commands in SQL statements or sent directly to the database through an API; after receiving these structured records, the database stores them in a table through its transaction processing capabilities; the unique version number of each record ensures that the original data state can be tracked and accessed even after data updates or model re-evaluation.

[0019] (III) Beneficial effects

[0020] The present invention provides a method for detecting and automatically collecting abnormal medical data, which has the following beneficial effects:

[0021] 1. The present invention improves the accuracy and real-time performance of anomaly detection by combining wavelet transform and LSTM adaptive multi-scale anomaly detection model, captures the temporal dependencies in medical data, and dynamically adjusts sensitivity parameters to adapt to data changes.

[0022] 2. The present invention improves the intelligence level of data collection strategy through an intelligent collection strategy optimizer, and automatically adjusts the collection frequency, range and accuracy according to the abnormal detection results to capture key information more effectively.

[0023] 3. The present invention improves the integrity and traceability of data management by integrating the collected data and its abnormality detection results and transmitting them to the database in real time. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0025] In the field of medical data management, the present invention significantly improves the accuracy and real-time performance of anomaly detection and optimizes data collection strategies through a system demonstration.

[0026] Collect medical data and perform data preprocessing to remove missing values ​​and duplicate records. For example, for multiple duplicate entries of the same drug in drug sales records, identify and retain the latest sales data and delete old duplicate entries. In addition, standardize the data and unify the coding and measurement standards. For example, unify the different names for the same drug in different regions into one standard name to ensure data consistency.

[0027] Next, the wavelet transform algorithm is applied to extract the time-frequency characteristics of the time series data. By selecting the appropriate wavelet function and decomposition level, for example, when analyzing drug sales data, if the sales volume suddenly increases or decreases at a certain point in time, the Haar wavelet can quickly identify this mutation; wavelet decomposition is performed on each time series data to decompose the data into approximate coefficients and detail coefficients of different scales; for example, for the time series data of drug inventory, the wavelet transform decomposes the data into approximate coefficients that reflect long-term trends and detail coefficients that capture short-term fluctuations; the approximate coefficients represent the pattern of seasonal changes in inventory, while the detail coefficients represent the rapid changes in inventory caused by a certain event; after completing the wavelet decomposition, the mean, standard deviation and square root of the sum of squares of each coefficient are calculated, and then these are combined into a feature vector, which captures the time-frequency characteristics of the data.

[0028] In the anomaly detection stage, a multi-layer LSTM network structure is designed to deeply explore the long-term dependencies in medical data. For example, considering that drug sales are affected by holidays, seasonal factors or market activities, LSTM learns these time series patterns. In the training stage, the time-frequency characteristics of historical medical data are used as input, and supervised learning is used to allow LSTM to identify the patterns of normal data. Then LSTM dynamically adjusts its sensitivity parameters according to the changing trend of real-time data. For example, when abnormal fluctuations in the data are detected, its attention to recent data is increased to capture anomalies more accurately. A fully connected layer is added after the LSTM network, which converts the output of LSTM into anomaly scores, which are used to quantify the degree to which each data point deviates from the normal pattern.

[0029] The adaptive multi-scale anomaly detection model can conduct in-depth analysis of medical data and identify anomalies. The model receives preprocessed time series data and its time-frequency features, which are input into LSTM. LSTM analyzes each data point by learning the temporal dependencies in the data and outputs an anomaly score, which quantifies the deviation of the data point from the normal pattern. For example, if the daily sales volume of a drug suddenly surges, the magnitude of this surge will be captured by the LSTM network and a higher anomaly score will be given. A dynamic threshold is set based on statistical analysis of historical data. The threshold takes into account the statistical characteristics of the data and is adjusted according to business needs. For example, the normal fluctuation range of drug sales during holidays will be larger than usual, so the threshold will be adjusted accordingly. Data points exceeding the threshold are marked as anomalies, and the output anomaly scores are sorted to identify the data points with the highest scores. The contextual information of these data points, such as the time, location, and drugs involved, will be further used to identify abnormal patterns and trends. For example, if the inventory of a certain drug in a certain area suddenly drops and coincides with the outbreak of a disease in that area, this trend indicates a problem with the supply chain or a public health event.

[0030] The intelligent collection strategy optimizer is used to adjust the collection strategy through reinforcement learning to adapt to the changes of new medical data. First, a reinforcement learning agent is built to learn the optimal collection strategy through interaction with the medical data collection environment. The decision of the reinforcement learning agent is based on multiple parameters of the current environment, including the frequency, scope, accuracy of the collection, and historical anomaly detection results. These parameters together form the basis for the decision of the reinforcement learning agent. The reinforcement learning agent performs a variety of possible strategy adjustments, such as increasing the frequency of data collection in a specific time period or expanding the scope of data collection. By designing a reward mechanism, the reinforcement learning agent obtains positive feedback when it successfully captures abnormal data, and negative feedback when it successfully captures abnormal data. If the agent fails to monitor the inventory of a particular drug during the flu season, it will automatically adjust its strategy to deal with similar situations in the future. Through continuous learning and adjustment, the intelligent collection strategy optimizer identifies the most effective collection strategy in a specific situation and adjusts the actual collection strategy accordingly, ensuring that the collection strategy is closely related to the anomaly detection results and can respond to changes in the data in a timely manner.

[0031] The new data collected and the anomaly detection results are integrated to form structured records. The integration process is implemented through the data processing functions in the programming language. For example, in Python, dictionaries or objects are used to encapsulate this information. Specifically, a dictionary is created for each data point, which contains key-value pairs, such as timestamp corresponding to the timestamp, data_value corresponding to the data value, anomaly_flag corresponding to the Boolean value of whether it is abnormal, anomaly_score corresponding to the anomaly score, and version corresponding to the version number; for example, a record is represented as {"timestamp":"2024-12-11T10:00:00Z","data_value":120,"anomaly_flag":true,"anomaly_score":85,"version":"v1"}; these dictionaries are then converted into the format received by the database, such as insert commands in SQL statements or sent directly to the database through an API; after receiving these structured records, the database stores them in a table through its transaction processing capabilities; the unique version number of each record ensures that the original data state can be tracked and accessed even after data updates or model re-evaluation.

[0032] The present invention not only improves the accuracy and real-time performance of anomaly detection, but also optimizes the data collection strategy through an intelligent collection strategy optimizer, so that the collection strategy can be dynamically adjusted according to the anomaly detection results.

[0033] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting and automatically collecting abnormal medical data, characterized in that: The following steps are involved: S1. Process the collected medical data and use wavelet transform to extract the time-frequency characteristics of the medical time series data to provide input for anomaly detection; S2. Combined with the extracted time-frequency features, a long short-term memory network is used to train an adaptive multi-scale anomaly detection model, which can learn the temporal dependencies of data and dynamically adjust the sensitivity parameters of anomaly detection according to the statistical characteristics of historical data and the changing trends of real-time data; S3. Use an adaptive multi-scale anomaly detection model to perform anomaly detection on medical data and identify abnormal data points; Analyze the output of the adaptive multi-scale anomaly detection model to determine the specific information of the abnormal data points and their impact; S4. Based on the abnormal results and data value assessment, the intelligent collection strategy optimizer is used to automatically adjust the collection strategy. The intelligent collection strategy optimizer evaluates the collection frequency, range and accuracy of different collection strategies based on reinforcement learning; S5. Execute the adjusted collection strategy, collect new data, and feed the new data back to the adaptive multi-scale anomaly detection model for model update and sensitivity parameter adjustment; S6. Record the collected data and its anomaly detection results in the database to ensure the integrity and traceability of the data.

2. The method for detecting and automatically collecting medical data anomalies according to claim 1, characterized in that: The collected medical data include drug sales records, inventory, and patient feedback. The collected data are preprocessed, including cleaning, removing missing values ​​and duplicate records, and standardizing the data format to unify the coding and measurement standards of all data; the wavelet transform algorithm is applied to extract the time-frequency characteristics of the time series data, and a wavelet function and decomposition layer are selected to perform wavelet decomposition on each time series data; the data is decomposed into approximate coefficients and detail coefficients of different scales by performing discrete wavelet transform on the data. The approximate coefficients represent the low-frequency trend information of the data, while the detail coefficients contain high-frequency fluctuations and abnormal information; after completing the wavelet decomposition, the obtained data is converted into feature vectors, which will contain the time-frequency characteristics of the original data and provide input for the subsequent anomaly detection model.

3. A method for detecting and automatically collecting medical data anomalies according to claim 2, characterized in that: The time-frequency features are used as input data to detect anomalies in medical data. An LSTM network structure is designed, which contains multiple LSTM layers. Each layer consists of multiple LSTM units to capture long-term dependencies in the data. During the training process, the time-frequency features of historical medical data are input into the LSTM network, and the pattern of normal data is learned through supervised learning; an adaptive mechanism is added to dynamically adjust the sensitivity parameters of the LSTM according to the changing trend of real-time data; a fully connected layer is added after the LSTM, and the output of this layer is used to calculate the anomaly score.

4. The method for detecting and automatically collecting medical data anomalies according to claim 1, characterized in that: The adaptive multi-scale anomaly detection model performs anomaly detection on medical data, inputs the processed time series data and its corresponding time-frequency features into LSTM, and LSTM analyzes the data based on the learned temporal dependency relationship; The output anomaly score indicates how much each data point deviates from the normal pattern; A threshold is set to determine abnormal data points, and data points with anomaly scores exceeding the threshold will be marked as abnormal; Thresholds are derived based on statistical analysis of historical data and are dynamically adjusted to adapt to data changes and business needs; the anomaly scores output by the model are sorted, the data points with the highest scores are identified, and the contextual information of these data points is further analyzed; through this analysis, patterns and trends of abnormal data points are identified.

5. The method for detecting and automatically collecting medical data anomalies according to claim 1, characterized in that: The intelligent acquisition strategy optimizer uses reinforcement learning technology to evaluate and adjust the acquisition strategy; through abnormal data points, the impact of these anomalies on the overall medical data and the potential risks are determined; based on these analyses, the intelligent acquisition strategy optimizer will evaluate whether the current acquisition frequency, range and accuracy are sufficient to capture these changes; the reinforcement learning algorithm of the intelligent acquisition strategy optimizer will simulate different acquisition strategies and evaluate their ability to detect abnormal data; so that it can learn which acquisition strategies are most effective in specific situations and adjust the actual acquisition strategy accordingly; This dynamic adjustment ensures that the acquisition strategy is closely aligned with the anomaly detection results and can respond to changes in the data in a timely manner.

6. The method for detecting and automatically collecting medical data anomalies according to claim 1, characterized in that: The adjusted collection strategy will be executed, and the newly collected data will be transmitted back to the adaptive multi-scale anomaly detection model in real time for further learning of the model and adjustment of sensitivity parameters; new data is input into the model, and the model re-evaluates and adjusts its internal parameters based on the actual conditions of these data points to better capture the temporal dependencies in the data; based on the statistical characteristics of the new data, the sensitivity parameters of anomaly detection are dynamically adjusted, including adjusting the threshold or redefining the calculation method of the anomaly score.

7. The method for detecting and automatically collecting medical data anomalies according to claim 1, characterized in that: The collected new data and anomaly detection results are integrated to form structured records, including timestamps, data values, anomaly markers, and outputs of detection models; these records are transmitted to the database in real time through the data interface, and the transaction processing capabilities of the database are used to ensure the consistency and integrity of the data; A unique version number is assigned to each record, so that the original data state can be tracked even if the data changes in subsequent updates of the adaptive multi-scale anomaly detection model.