A drug risk monitoring method based on multi-source data fusion

Abstract

The present invention discloses a drug risk monitoring method based on multi-source data fusion, which relates to the field of data analysis technology. The present invention solves the problems of data heterogeneity and information islands through multi-source data fusion and processing, provides a comprehensive and high-quality data foundation for drug risk monitoring, adopts a weighted collaborative filtering algorithm and a Bayesian network to accurately mine potential risk signals, realizes real-time dynamic monitoring and early warning of drug risks, and significantly improves the timeliness and reliability of monitoring; at the same time, it uses a real-time stream processing framework to incrementally update new data, dynamically adjusts risk assessment results, and automatically issues early warnings when risk values ​​exceed the standard, effectively resolving the contradiction between complex processes and the need for rapid early warnings, improving the efficiency and accuracy of drug safety supervision, and providing strong support for ensuring public drug safety.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a drug risk monitoring method based on multi-source data fusion. Background Art

[0002] Drug risk monitoring is a crucial research area in the pharmaceutical field, directly related to public drug safety and health protection. With the expansion of drug use, timely and accurate identification and assessment of drug risks has become key to maintaining the stability of the medical system and improving patients' quality of life. However, current drug risk monitoring methods often rely on a single data source, such as adverse reaction reports from medical institutions or production records from pharmaceutical companies. This single reliance results in narrow information coverage and limited ability to capture risk signals, making it difficult to fully reflect the risk characteristics of the entire drug chain, from production to use. In addition, when processing multi-source heterogeneous data, existing methods often lack effective integration methods, resulting in information silos, which significantly restricts the accuracy and timeliness of monitoring results.

[0003] From a technical perspective, the core challenge in the field of drug risk monitoring lies in how to efficiently integrate multi-source data and identify potential risk signals from them. Specifically, the first is the problem of data heterogeneity. Data such as electronic medical records of medical institutions, pharmacy sales data, and social media feedback vary greatly in structure and format, making cleaning and standardization difficult. Secondly, the fusion algorithm lacks accuracy. Traditional methods find it difficult to balance the credibility and relevance of different data sources, resulting in the fusion results potentially masking key risk signals. Finally, real-time requirements pose higher challenges to existing analysis models, and there is a contradiction between complex data processing processes and the need for rapid early warning. These unresolved technical factors make comprehensive monitoring of drug risks and timely early warning a problem. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, the present invention provides a drug risk monitoring method based on multi-source data fusion.

[0005] The technical solution of the present invention is achieved as follows:

[0006] A drug risk monitoring method based on multi-source data fusion includes the following steps:

[0007] Acquire multi-source data, such as medical institution electronic medical records, pharmacy sales records, and social media feedback. Structure the raw data using pre-set format conversion rules to generate a uniformly coded first dataset. Then, based on missing values ​​and outliers in the first dataset, use statistical interpolation and anomaly detection algorithms to clean and repair them, resulting in a pre-processed second dataset with consistent fields.

[0008] Obtain the source attributes of the second data set, calculate the sample size and update frequency of each data source, determine the credibility weight of each data source, obtain a third data set with weight labels, extract features from the third data set, and transform the heterogeneous features into a standardized fourth data set through a preset feature mapping table;

[0009] Based on the fourth data set, a weighted collaborative filtering algorithm is used to fuse multi-source features. When the weight of a data source is lower than a preset threshold, its influence on the fusion result is reduced, thereby obtaining a fifth data set containing potential risk signals.

[0010] Obtain the time series data from the fifth dataset, calculate the changing trends of each drug adverse reaction using a sliding window technique, and obtain the sixth dataset. Based on the abnormal trends in the sixth dataset, use a Bayesian network to perform probabilistic reasoning on risk signals, identify high-risk drugs and their quantitative assessment values, and obtain the seventh dataset.

[0011] Feature vectors of high-risk drugs are extracted from the seventh data set, and the newly added data is incrementally updated through the real-time stream processing framework to obtain a dynamically adjusted eighth data set. Based on the risk assessment values ​​in the eighth data set, a ninth data set containing drug names and risk levels is generated.

[0012] Furthermore, obtaining the second data set that has been preprocessed and has consistent fields specifically includes:

[0013] Obtain the missing value distribution from the first data set through statistical methods, fill the missing values ​​using mean interpolation to obtain a preliminary filled data set, and then use the isolation forest algorithm to identify abnormal data in the preliminary filled data set and determine the location of abnormal points;

[0014] The outlier features are obtained from the outlier points. The outlier data is repaired using the median replacement method to obtain a repaired dataset. Then, based on the field attributes of the repaired dataset, the field format is standardized to obtain a consistent format dataset. If outliers still exist in the consistent format dataset, a clustering algorithm is used to detect the outliers and determine whether further repair is required.

[0015] The normal data range in the clustering results is obtained, and the outliers beyond the range are removed by elimination method to obtain the preprocessed data set. The integrity of the preprocessed data set is checked to determine the field consistency and obtain the second data set.

[0016] Furthermore, a third data set with weight labels is obtained, specifically including:

[0017] The data source and sample size are extracted from the source attributes of the second data set. The distribution characteristics of the sample size are calculated using statistical methods to obtain the sample size index of each data source. The frequency of the data source is obtained based on its update frequency. When the frequency exceeds a preset threshold, it is determined to be a high-frequency data source and the update frequency classification result is obtained. Based on the sample size index and update frequency classification, a credibility score is obtained from a pre-established credibility assessment model to determine the credibility level of each data source;

[0018] Based on the credibility level and sample size indicators, a linear regression algorithm is used to calculate the weight value to obtain a preliminary weight tag. The preliminary weight tag and update frequency classification are obtained. When the credibility level does not match the frequency, the weight value is adjusted to obtain an optimized weight-tagged data set. The clustering algorithm is then used to divide the data source categories and determine the third data set with weight tags. The consistency of the calculation results is verified through the weight tags of the third data set, and the integrity of the tag generation process is judged to obtain the final third data set.

[0019] Furthermore, the heterogeneous features are converted into a standardized fourth data set through a preset feature mapping table, specifically including:

[0020] Parse the drug name, adverse reaction description, and timestamp from the third data set, use a text extraction algorithm to generate a preliminary feature set, match the heterogeneous features in the preliminary feature set through a preset feature mapping table to obtain a mapped feature set, and when there are missing values ​​in the mapped feature set, associate the adverse reaction description with the timestamp, use a content filling algorithm to generate a complete feature set, compare the complete feature set with the preset table, and use a format verification algorithm to determine whether all heterogeneous features have been converted into a standardized format to obtain a standardized feature set;

[0021] According to the standardized feature set, drug names and adverse reactions are grouped to generate a classification feature set. Based on the timestamps in the classification feature set, a time series distribution analysis algorithm is used to calculate the distribution pattern of adverse reactions under different drug names to obtain a distribution feature set. Then, an association analysis algorithm is used to calculate the association strength between drug names and adverse reactions to generate the fourth data set.

[0022] Furthermore, a fifth data set containing potential risk signals is obtained, specifically including:

[0023] The fourth dataset is used to obtain multi-source feature data. The Pearson correlation coefficient in the weighted collaborative filtering algorithm is used to calculate the weight value of each data source. The weight value of each data source is then extracted based on the weight set. When the weight of a data source is lower than the preset threshold, the fusion coefficient of the corresponding feature is reduced to obtain the adjusted weight set.

[0024] The adjusted weight set is used to perform weighted fusion processing on multi-source features to generate a fused feature set. Potential risk signals are extracted through the fused feature set to obtain an intermediate feature set containing risk markers.

[0025] Then, we obtain the intermediate feature set, use logistic regression analysis to analyze risk markers and screen significant risk features to obtain a significant risk feature set. Based on the significant risk feature set, we combine the weighted collaborative filtering algorithm to optimize the fusion results and generate a fifth data set containing potential risk signals.

[0026] The fifth data set is used to analyze the distribution characteristics of risk signals using anomaly detection tools to obtain a risk distribution data set.

[0027] Furthermore, a sixth data set is obtained, specifically including:

[0028] Obtain time series data from the fifth data set, use a preset sliding window technique to calculate the data point set within each time period, and obtain a preliminary change trend. Then, based on the preliminary change trend, calculate the sliding window mean and standard deviation of each adverse drug reaction to obtain a quantified change trend indicator;

[0029] Based on the quantified change trend index, determine whether the change trend index of a certain drug exceeds the preset threshold. If it exceeds, mark it as abnormal, and obtain a preliminary judgment result of the trend abnormality;

[0030] Based on the preliminary judgment results of trend anomalies, the corresponding drug list is extracted, and the abnormal drugs are grouped using a clustering algorithm to obtain a classified abnormal drug set. The time series characteristics of adverse reactions in each group are then obtained from the classified abnormal drug set. The differences in the changing trends between the groups are determined to obtain the detailed analysis results of the abnormal trend;

[0031] Based on the refined analysis results of the abnormal trend, a data set including time series features and a list of abnormal drugs is generated to obtain the sixth data set.

[0032] Furthermore, the seventh data set is obtained, which specifically includes:

[0033] The time series features of abnormal trends were extracted from the sixth data set. A Bayesian network was used to construct a probability model for risk signals. The Markov Chain Monte Carlo method was used to obtain initial probability distribution data. The conditional probability of each adverse drug reaction was calculated based on the initial probability distribution data. The initial strength of the risk signal was determined, and a signal strength distribution set was obtained.

[0034] Abnormal signals exceeding the preset threshold are extracted from the signal intensity distribution set, and the weighted average method is used to calculate the quantitative risk value of the drug to obtain a preliminary list of high-risk drugs. Then, the corresponding time series data is obtained, and statistical tools are used to analyze the significance of the change trend to obtain a set of quantitative assessment values;

[0035] The risk level of each drug is determined by a set of quantitative assessment values. When the quantitative assessment value exceeds the preset threshold, the drug is marked as high-risk, and a classified drug list is obtained. Based on the classified drug list, the characteristic vector of the change trend between groups is extracted, and the Bayesian network is used for secondary probability inference to obtain the optimized risk probability distribution.

[0036] The final quantitative assessment value of high-risk drugs is calculated using the optimized risk probability distribution. A data integration tool is used to generate the seventh data set to obtain complete risk assessment data. The distribution characteristics of abnormal trends are then obtained. The data consistency is verified using a circular comparison method to obtain a confirmed risk signal set.

[0037] The dynamic characteristics of high-risk drugs are extracted through the confirmed risk signal set, and the time series data are fused using the incremental update algorithm to obtain the adjusted risk assessment results.

[0038] Furthermore, a dynamically adjusted eighth data set is obtained, specifically including:

[0039] Extract feature vectors of high-risk drugs from the seventh data set to obtain an initial feature set. Then, use a real-time stream processing framework to analyze the newly added data to determine whether it contains high-risk drug information and obtain an updated data subset.

[0040] The initial feature set is fused through the incremental update algorithm to generate an adjusted feature vector set. When the adjusted feature vector set exceeds the preset threshold, the feature weights are updated through the dynamic adjustment mechanism to obtain the optimized feature set. Based on the optimized feature set, the data generation module is used to construct the eighth data set to complete the dynamic update of the data set.

[0041] Furthermore, a ninth data set containing drug names and risk levels is generated, specifically including:

[0042] Obtaining the risk assessment value from the eighth data set, determining whether it exceeds a preset threshold, obtaining an exceeding-standard record, and then extracting the name record of the exceeding-standard drug based on the exceeding-standard record to determine the trigger state;

[0043] Generate a temporary data set containing drug names through the trigger state, obtain the integrity of the temporary data set, use the random forest algorithm to process the temporary data set, extract risk level features, and obtain the level generation result;

[0044] The ninth data set is constructed using the level generation results and name records to determine data integrity. The risk levels in the ninth data set are then verified to be consistent with the assessment of the eighth data set to obtain a verification result.

[0045] The data association in the ninth data set is updated through the verification results, and the updated data set is determined. Then, combined with the exceeding-standard records, data including drug names and risk levels are generated.

[0046] Beneficial effects of the present invention:

[0047] This invention successfully solves the problem of data heterogeneity through the collection, structured processing, cleaning and repair, and standardized transformation of multi-source data, breaks down information silos, and achieves efficient fusion and unified standardization of multi-source data. It provides a comprehensive, high-quality data foundation for drug risk monitoring, enabling comprehensive coverage of risk characteristics across the entire drug supply chain and significantly improving the data coverage and accuracy of drug risk monitoring.

[0048] This invention uses advanced technologies such as weighted collaborative filtering algorithms and Bayesian networks to accurately integrate multi-source data features and mine potential risk signals. It also dynamically monitors adverse drug reaction trends through time series analysis. This process not only improves the accuracy of risk signal mining and avoids the masking of key risk signals, but also realizes real-time dynamic monitoring and early warning of drug risks, providing strong support for comprehensive monitoring and timely early warning of drug risks, and significantly improving the timeliness and reliability of drug risk monitoring.

[0049] Through the real-time stream processing framework, new data is incrementally updated, risk assessment results are dynamically adjusted, and the early warning mechanism is automatically triggered when the risk value exceeds the threshold. This realizes real-time dynamic update of risk assessment and efficient early warning, can promptly reflect changes in drug risks, improve the efficiency and accuracy of drug safety supervision, effectively resolve the contradiction between complex data processing processes and rapid early warning needs, provide strong technical support for drug safety supervision, and help ensure the safety of public drug use. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a drug risk monitoring method based on multi-source data fusion provided in this application;

[0051] Figure 2 A schematic diagram of a process for obtaining a fifth data set containing potential risk signals by a drug risk monitoring method based on multi-source data fusion provided in this application;

[0052] Figure 3 A flowchart of a seventh data set obtained by a drug risk monitoring method based on multi-source data fusion provided in this application. DETAILED DESCRIPTION

[0053] In order to make the purposes, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0054] like Figure 1-3 As shown, this embodiment provides a drug risk monitoring method based on multi-source data fusion, including the following steps:

[0055] S100, acquiring multi-source data such as medical institution electronic medical records, pharmacy sales records, and social media feedback, performing structured processing on the raw data using preset format conversion rules to obtain a uniformly coded first dataset, then using statistical interpolation and anomaly detection algorithms to clean and repair missing values ​​and outliers in the first dataset, thereby obtaining a preprocessed second dataset with consistent fields;

[0056] Furthermore, in step S100, obtaining a first uniformly coded data set specifically includes:

[0057] Electronic medical records and sales records are obtained through the system interfaces of medical institutions and pharmacies, and social feedback is collected through social media platforms to obtain multi-source data. Preset rules are then used to perform format conversion on the multi-source data to generate structured raw data; unified coding rules are applied to the raw data to obtain a first data set, and then the first data set is grouped through a clustering algorithm to obtain classified data. When there are outliers in the classified data, the outliers are filtered out through a preset threshold to obtain cleaned data. The feature distribution of each group is calculated based on the cleaned data to obtain a feature set. The feature set is analyzed using a decision tree algorithm to determine the association pattern between the data and obtain a pattern result.

[0058] Specifically, a decision tree algorithm is used to analyze feature sets, determine the correlation patterns between data, and obtain pattern results. These correlations are specifically manifested as intrinsic connections between different data sources (such as electronic medical records, pharmacy sales records, and social media feedback). For example, there is a positive correlation between the frequency of a certain disease in electronic medical records and the sales volume of the corresponding drug in pharmacies, or a correlation between negative feedback on a certain drug on social media and the adverse reaction records of the drug in electronic medical records. This correlation pattern can help medical institutions better understand the relationship between disease and treatment and optimize treatment plans; help pharmaceutical companies analyze market demand and drug effectiveness, and adjust R&D and sales strategies; and provide data support for public health departments to monitor drug safety and disease epidemic trends, thereby improving the overall decision-making efficiency and service quality of the medical industry.

[0059] Furthermore, in step S100, obtaining a second data set that has been preprocessed and has consistent fields specifically includes:

[0060] Obtain the missing value distribution from the first data set through statistical methods, fill the missing values ​​using mean interpolation to obtain a preliminary filled data set, and then use the isolation forest algorithm to identify abnormal data in the preliminary filled data set and determine the location of abnormal points;

[0061] The outlier features are obtained from the outlier points. The outlier data is repaired using the median replacement method to obtain a repaired dataset. Then, based on the field attributes of the repaired dataset, the field format is standardized to obtain a consistent format dataset. If outliers still exist in the consistent format dataset, a clustering algorithm is used to detect the outliers and determine whether further repair is required.

[0062] The normal data range in the clustering results is obtained, and the outliers beyond the range are removed by elimination method to obtain the preprocessed data set. The integrity of the preprocessed data set is checked to determine the field consistency and obtain the second data set.

[0063] S200: Based on the source attributes of the second data set, the sample size and update frequency of each data source are calculated, and the credibility weight of each data source is determined to obtain a third data set with a weight label. Features such as drug name, adverse reaction description, and timestamp are extracted from the third data set, and the heterogeneous features are converted into a standardized fourth data set using a preset feature mapping table.

[0064] Furthermore, in step S200, a third data set with weight labels is obtained, which specifically includes:

[0065] The data source and sample size are extracted from the source attributes of the second data set. The distribution characteristics of the sample size are calculated using statistical methods to obtain the sample size index of each data source. The frequency of the data source is obtained based on its update frequency. When the frequency exceeds a preset threshold, it is determined to be a high-frequency data source and the update frequency classification result is obtained. Based on the sample size index and update frequency classification, a credibility score is obtained from a pre-established credibility assessment model to determine the credibility level of each data source;

[0066] Based on the credibility level and sample size indicators, a linear regression algorithm is used to calculate the weight value to obtain a preliminary weight tag. The preliminary weight tag and update frequency classification are obtained. When the credibility level does not match the frequency, the weight value is adjusted to obtain an optimized weight-tagged data set. The clustering algorithm is then used to divide the data source categories and determine the third data set with weight tags. The consistency of the calculation results is verified through the weight tags of the third data set, and the integrity of the tag generation process is judged to obtain the final third data set.

[0067] Specifically, the process of obtaining preliminary weight labels involves constructing a linear regression model using the credibility level and sample size indicator as independent variables and the data source credibility score as the dependent variable. The linear regression model assumes that the credibility score is a linear combination of the credibility level and sample size indicator. By training this model, a weight value for each data source can be obtained. The weight value reflects the contribution of the credibility level and sample size indicator to the credibility of the data source. Finally, the calculated weight value is labeled on each data source, forming a third weighted dataset.

[0068] The matching rules between update frequency and credibility are based on the logical relationship between a data source's update frequency and credibility. Specifically, if a data source's update frequency exceeds a preset threshold, it is considered a high-frequency data source. Typically, the credibility of such a data source is enhanced, as high-frequency data sources can promptly reflect the latest developments, reducing the risk of data obsolescence and thus improving data credibility. Conversely, if a data source's update frequency is below a preset threshold, it is considered a low-frequency data source. The credibility of such a data source may require a comprehensive assessment using other metrics (such as sample size and historical data quality). The connection between update frequency and credibility lies in the fact that high-frequency data sources not only have advantages in timeliness but also provide more complete information, reducing the possibility of data loss and further enhancing data credibility. Therefore, update frequency is an important dimension in assessing data source credibility.

[0069] Regarding weight adjustment, a linear regression algorithm is first used to calculate a preliminary weight for each data source based on its credibility level and sample size. This preliminary weight tag is then matched and verified against the update frequency classification. If the credibility level matches the update frequency (for example, a high-frequency data source has a higher credibility level), the weight calculation is considered reasonable and no adjustment is required. If the credibility level and update frequency do not match (for example, a low-frequency data source has a higher credibility level), the weight value is adjusted according to pre-set rules to ensure the rationality of the weight tag. The adjusted weight tag forms the third weight-tagged dataset.

[0070] Furthermore, in step S200, the heterogeneous features are converted into a standardized fourth data set using a preset feature mapping table, specifically including:

[0071] Parse the drug name, adverse reaction description, and timestamp from the third data set, use a text extraction algorithm to generate a preliminary feature set, match the heterogeneous features in the preliminary feature set through a preset feature mapping table to obtain a mapped feature set, and when there are missing values ​​in the mapped feature set, associate the adverse reaction description with the timestamp, use a content filling algorithm to generate a complete feature set, compare the complete feature set with the preset table, and use a format verification algorithm to determine whether all heterogeneous features have been converted into a standardized format to obtain a standardized feature set;

[0072] According to the standardized feature set, drug names and adverse reactions are grouped to generate a classification feature set. Based on the timestamps in the classification feature set, a time series distribution analysis algorithm is used to calculate the distribution pattern of adverse reactions under different drug names to obtain a distribution feature set. Then, an association analysis algorithm is used to calculate the association strength between drug names and adverse reactions to generate the fourth data set.

[0073] For example, the drug name, adverse reaction description and timestamp are parsed from the third data set, and a regular expression-based text extraction algorithm is used to match structured fields such as "drug: aspirin", "adverse reaction: headache", and "time: 2023-05-12" to generate a preliminary feature set containing the drug name, adverse reaction description and timestamp; for heterogeneous features in the preliminary feature set, such as the presence of two expressions of the drug name "Aspirin" and "Aspirin", a standardized matching is performed through a preset feature mapping table, and "Aspirin" is mapped to "Aspirin". After the mapping is obtained feature set; if there are missing values ​​in the mapped feature set, such as a record missing the adverse reaction description, the adverse reaction descriptions with the same timestamp in other records are associated through the timestamp, and a content filling algorithm based on a time window is used to fill the missing content in a nearest neighbor matching manner to generate a complete feature set. By comparing the complete feature set with the preset table, a format verification algorithm based on a rule engine is used to check whether the field conforms to standard formats such as "drug name: string", "adverse reaction description: text", and "timestamp: YYYY-MM-DD", and to determine whether all heterogeneous features are converted to obtain a standardized feature set. Based on the standardized feature set, the K-means clustering algorithm was used to group drug names and adverse reactions, setting the number of clusters K to 5. Similarity was calculated based on the TF-IDF feature to generate a classification feature set. Using the timestamps in the classification feature set, a time series distribution analysis algorithm was used to calculate the monthly frequency distribution of adverse reactions associated with different drug names, generating a distribution feature set. Based on this distribution feature set, the Apriori association analysis algorithm was used to calculate the confidence level between drug names and adverse reactions, for example, the confidence level for "aspirin → headache" was 85%, generating the final dataset. The cosine similarity algorithm was used to calculate vector matching, setting a threshold of 0.8, to identify a feature subset for high-risk drugs. Based on this feature subset, the Flink stream processing framework's incremental update algorithm was used to integrate high-risk drug records with matching levels exceeding the threshold in new data in real time to generate the fourth dataset.

[0074] S300: Based on the fourth data set, a weighted collaborative filtering algorithm is used to fuse multi-source features. When the weight of a data source is lower than a preset threshold, its influence on the fusion result is reduced, thereby obtaining a fifth data set containing potential risk signals.

[0075] Furthermore, in step S300, a fifth data set containing potential risk signals is obtained, specifically including:

[0076] S310: Obtain multi-source feature data from the fourth data set, calculate the weight of each data source using the Pearson correlation coefficient in the weighted collaborative filtering algorithm, and obtain a weight set. Then, extract the weight value of each data source based on the weight set. If the weight of a data source is lower than a preset threshold, reduce the fusion coefficient of the corresponding feature to obtain an adjusted weight set.

[0077] S320, performing weighted fusion processing on the multi-source features using the adjusted weight set to generate a fused feature set, extracting potential risk signals through the fused feature set, and obtaining an intermediate feature set containing risk markers;

[0078] S330: Obtain an intermediate feature set again, use logistic regression analysis to analyze risk markers and screen significant risk features to obtain a significant risk feature set, optimize the fusion results based on the significant risk feature set and combine them with a weighted collaborative filtering algorithm to generate a fifth data set containing potential risk signals;

[0079] S340. Analyze the distribution characteristics of risk signals using anomaly detection tools through the fifth data set to obtain a risk distribution data set, and then use a clustering algorithm to divide the risk levels to obtain a graded risk data set; extract high-risk feature vectors through the graded risk data set, and fuse the newly added data with a real-time stream processing framework to obtain an updated feature set.

[0080] For example, multi-source feature data such as user ratings, purchase history, and browsing behavior are extracted from the fourth data set. The Pearson correlation coefficient in the weighted collaborative filtering algorithm is used to calculate the initial weight of each data source. A threshold of 0.3 is set. If the weight of social network data is 0.25, its fusion coefficient is reduced by 50%. The Pearson correlation coefficient determines the weight by measuring the linear correlation between two data sources. Its value range is between -1 and 1, and the closer the value is to 1 or -1, the stronger the correlation. The threshold is set to 0.3. If the weight of a data source (such as social network data) is lower than the threshold (such as 0.25), it is considered to have weak relevance and may be insufficiently reliable. Therefore, its fusion coefficient is reduced by 50%. This rule of reducing weight adjusts the contribution of the data source in the fusion process based on its reliability and relevance to ensure that the fused feature set more accurately reflects user preferences. The features are weighted and summed according to the adjusted weight set to generate a fused feature set that includes user preferences. The isolation forest algorithm is used to detect anomalous scoring patterns from the fused features, labeling potentially risky users to form an intermediate feature set. Logistic regression is used to analyze risk markers, and features with a p-value less than 0.05 are selected as significant risk features. The significant features are fed into an improved collaborative filtering model, and the weights are adjusted using the SGD optimizer. The output is a fifth dataset containing high-risk user labels. The LOF algorithm is used to calculate the anomaly score for each sample in the fifth dataset, generating a risk density distribution heatmap. K-means clustering (k=3) is used to classify the distributed data into high, medium, and low risk levels, and the cluster centers are extracted as feature vectors. When real-time streaming data arrives, the cosine similarity with the feature vector is calculated through a Flink window. If the cosine similarity exceeds 0.8, an incremental update of the feature set is triggered.

[0081] Specifically, the present invention not only takes into account the credibility of each data source through a weighted collaborative filtering algorithm, but also ensures that key risk signals will not be masked by low-quality data sources by dynamically adjusting the weights. This solves the problem that traditional methods often find it difficult to balance the credibility and correlation of different data sources when processing multi-source data, resulting in the fusion results possibly masking key risk signals. This method significantly improves the accuracy and reliability of the fusion results and can more effectively mine potential risk signals.

[0082] S400. Obtain the time series data in the fifth data set, calculate the changing trend of each adverse drug reaction using a sliding window technique, determine a list of drugs with abnormal trends, and obtain a sixth data set. Based on the abnormal trends in the sixth data set, use a Bayesian network to perform probabilistic reasoning on risk signals, identify high-risk drugs and their quantitative assessment values, and obtain a seventh data set.

[0083] Furthermore, in step S400, a sixth data set is obtained, which specifically includes:

[0084] Obtain time series data from the fifth data set, use a preset sliding window technique to calculate the data point set within each time period, and obtain a preliminary change trend. Then, based on the preliminary change trend, calculate the sliding window mean and standard deviation of each adverse drug reaction to obtain a quantified change trend indicator;

[0085] Based on the quantified change trend index, determine whether the change trend index of a certain drug exceeds the preset threshold. If it exceeds, mark it as abnormal, and obtain a preliminary judgment result of the trend abnormality;

[0086] Based on the preliminary judgment results of trend anomalies, the corresponding drug list is extracted, and the abnormal drugs are grouped using a clustering algorithm to obtain a classified abnormal drug set. The time series characteristics of adverse reactions in each group are then obtained from the classified abnormal drug set. The differences in the changing trends between the groups are determined to obtain the detailed analysis results of the abnormal trend;

[0087] Based on the results of the detailed analysis of abnormal trends, a data set containing time series characteristics and a list of abnormal drugs is generated to obtain a sixth data set. Based on the time series characteristics in the sixth data set, a statistical method is used to calculate the fluctuation amplitude of each abnormal drug to obtain a quantitative value of trend fluctuation.

[0088] Exemplarily, the time series data of adverse drug reactions are extracted from the fifth data set. A sliding window with a window size of 30 days is used, and the mean of the data points in each window is calculated with a step size of 7 days to obtain a preliminary change trend. Based on the preliminary change trend, the mean and standard deviation of each drug in the sliding window are calculated. If the standard deviation of a drug exceeds 2 times the historical mean, it is marked as abnormal. The abnormal drugs are grouped using the DBSCAN clustering algorithm, and the neighborhood parameter eps=0.5 and the minimum number of samples min_samples=3 are set to obtain a classified set of abnormal drugs. The time series characteristics of each group of drugs are extracted, and the average volatility within the group and the difference between the groups are calculated. If the difference between the groups exceeds 0.7, it is determined to be a significant difference. Based on the results of the abnormal trend analysis, a sixth data set containing the drug name, time window, mean, standard deviation and group label is generated.

[0089] Furthermore, in step S400, a seventh data set is obtained, which specifically includes:

[0090] S410. Extracting time series features of abnormal trends from the sixth data set, constructing a probability model of risk signals using a Bayesian network, obtaining initial probability distribution data using a Markov chain Monte Carlo method, calculating the conditional probability of each adverse drug reaction based on the initial probability distribution data, determining a preliminary strength of the risk signal, and obtaining a signal strength distribution set;

[0091] S420. Abnormal signals exceeding a preset threshold are extracted from the signal intensity distribution set, and a weighted average method is used to calculate the quantitative risk value of the drug to obtain a preliminary list of high-risk drugs. Then, corresponding time series data is obtained, and statistical tools are used to analyze the significance of the change trend to obtain a set of quantitative assessment values.

[0092] S430: Determine the risk level of each drug using a set of quantitative assessment values. When the quantitative assessment value exceeds a preset threshold, mark the drug as high-risk. Obtain a classified drug list. Extract feature vectors of inter-group change trends based on the classified drug list. Perform secondary probabilistic reasoning using a Bayesian network to obtain an optimized risk probability distribution.

[0093] S440. Calculate the final quantitative assessment value of the high-risk drug using the optimized risk probability distribution, use the data integration tool to generate the seventh data set, obtain complete risk assessment data, and then obtain the distribution characteristics of abnormal trends. Use the circular comparison method to verify data consistency to obtain a confirmed risk signal set.

[0094] S450. Extract the dynamic characteristics of high-risk drugs through the confirmed risk signal set, use the incremental update algorithm to fuse the time series data, and obtain the adjusted risk assessment results.

[0095] Exemplarily, the time series characteristics of abnormal trends are extracted through the sixth data set, and a probability model of risk signals is constructed using a Bayesian network. The prior probability is set to 0.3, and the Markov chain Monte Carlo method is used for parameter estimation to obtain initial probability distribution data; the conditional probability of adverse reactions of each drug is calculated based on the initial probability distribution data, and the Bayesian formula is used in combination with the adverse reaction frequency in historical data. For example, the P(ADR|Risk) of a certain drug A is 0.85 to determine the preliminary intensity of the risk signal and obtain a signal intensity distribution set; abnormal signals exceeding the preset threshold are extracted through the signal intensity distribution set, and the threshold is set to 2 times the standard deviation. The weighted average method is used to calculate the risk quantification value of the drug, and the weight distribution is 60% for time series volatility and 40% for adverse reaction frequency to obtain a preliminary high-risk drug list; the corresponding time series data are obtained based on the preliminary high-risk drug list, and the t-test is used to analyze the significance of the change trend, and the p-value <0.05 is set as the significance standard to obtain a quantitative evaluation value set; through the quantitative evaluation value The risk level of each drug is determined by the set. If the assessment value of a certain drug B exceeds the threshold of 7.5, it is marked as a high-risk drug, and a classified drug list is obtained. The characteristic vector of the change trend between groups is extracted based on the classified drug list. The abnormal drugs are divided into three groups using the K-means clustering algorithm. Each group is input into the Bayesian network for quadratic probabilistic inference with a learning rate of 0.01 to obtain an optimized risk probability distribution. The final quantitative assessment value of high-risk drugs is calculated based on the optimized risk probability distribution. The SQL database integration tool is used to generate the seventh data set with fields including drug ID, risk value, and timestamp to obtain complete risk assessment data. The distribution characteristics of the abnormal trend are obtained from the seventh data set. The hash check algorithm is used to compare the previous and next data versions. If the difference rate is less than 1%, it passes the verification and obtains the confirmed risk signal set. The dynamic characteristics of high-risk drugs are extracted from the confirmed risk signal set. The online learning algorithm is used to fuse real-time streaming data with a window size of 30 days and a sliding step size of 1 day to obtain the adjusted risk assessment results.

[0096] Specifically, the present invention uses a Bayesian network to perform probabilistic reasoning on risk signals, which can fully consider various uncertain factors and provide more accurate risk assessment results. In addition, the Bayesian network can also dynamically update the risk assessment model based on new data to ensure the timeliness and accuracy of the risk assessment. This solves the problem that traditional methods often lack effective handling of uncertainties in risk assessment, resulting in inaccurate risk assessment results. This method not only improves the scientific nature and reliability of risk assessment, but also provides strong support for timely early warning of drug risks.

[0097] S500. Extract feature vectors of high-risk drugs from the seventh data set, incrementally update the newly added data through the real-time stream processing framework to obtain a dynamically adjusted eighth data set, and determine whether the risk value of a certain drug exceeds a preset threshold based on the risk assessment value in the eighth data set. If it exceeds the threshold, the automatic early warning mechanism is triggered to generate a ninth data set containing the drug name and risk level.

[0098] Furthermore, in step S500, a dynamically adjusted eighth data set is obtained, which specifically includes:

[0099] Extract feature vectors of high-risk drugs from the seventh data set to obtain an initial feature set. Then, use a real-time stream processing framework to analyze the newly added data to determine whether it contains high-risk drug information and obtain an updated data subset.

[0100] The initial feature set is fused through the incremental update algorithm to generate an adjusted feature vector set. When the adjusted feature vector set exceeds the preset threshold, the feature weights are updated through the dynamic adjustment mechanism to obtain the optimized feature set. Based on the optimized feature set, the data generation module is used to construct the eighth data set to complete the dynamic update of the data set.

[0101] For example, feature vectors for high-risk drugs are extracted from the seventh dataset. Dimensionality reduction is performed using the principal component analysis (PCA) algorithm, and the first 10 principal components are extracted as the initial feature set. The new data is analyzed using the real-time stream processing framework Apache Kafka. A rule engine determines whether the number of adverse reaction reports for a drug exceeds 50, generating an updated data subset. For this updated data subset, the initial feature set is fused using the incremental update algorithm Online PCA to generate an adjusted feature vector set. If the variance contribution of a feature in the adjusted feature vector set exceeds 15%, the feature weights are updated using a dynamic adjustment mechanism, and the weight parameters are optimized using gradient descent to obtain an optimized feature set. Based on the optimized feature set, the eighth dataset is constructed using the data generation module. Dynamic updates of the dataset are performed using the HDFS storage format. After obtaining the eighth dataset, the risk identification process is used to detect the changing trends of high-risk drugs. The time series analysis algorithm ARIMA is used to predict the risk distribution for the next 30 days, generating a risk distribution result.

[0102] Furthermore, in step S500, a ninth data set including drug names and risk levels is generated, specifically including:

[0103] Obtaining the risk assessment value from the eighth data set, determining whether it exceeds a preset threshold, obtaining an exceeding-standard record, and then extracting the name record of the exceeding-standard drug based on the exceeding-standard record to determine the trigger state;

[0104] Generate a temporary data set containing drug names through the trigger state, obtain the integrity of the temporary data set, use the random forest algorithm to process the temporary data set, extract risk level features, and obtain the level generation result;

[0105] The ninth data set is constructed using the level generation results and name records to determine data integrity. The risk levels in the ninth data set are then verified to be consistent with the assessment of the eighth data set to obtain a verification result.

[0106] The data association in the ninth data set is updated through the verification results, and the updated data set is determined. Then, combined with the exceeding-standard records, data including drug names and risk levels are generated.

[0107] Exemplarily, drug risk assessment values ​​are extracted from the eighth data set. If the risk value of a drug exceeds the preset threshold of 0.8, it is marked as an over-standard record. According to the drug ID field in the over-standard record, the drug information table is matched to obtain the corresponding drug name, a name record is generated and the trigger state is set to "warning", the name record is stored in a temporary data set, and the temporary data set is checked for null values ​​or duplicates to ensure data integrity. The temporary data set is analyzed using a random forest algorithm, the number of decision trees is set to 100, the maximum depth is 10, the risk level features are extracted by feature importance sorting, and three-level classification results of high, medium and low are output; the classification results are merged with the drug name records to construct a ninth data set, and the field matching and missing value ratio are checked. The risk levels in the ninth data set are compared with the original assessment values ​​of the eighth data set. If the original assessment values ​​of high-risk drugs are all greater than 0.8, the verification is passed; based on the verification results, the drug-level mapping relationship in the ninth data set is corrected, and the data association index is updated. Integrate the updated data set with the exceedance records to generate a structured data table containing the drug name (such as "Drug A") and the risk level (such as "High Risk"). Use the K-means clustering algorithm to process the final data set, set the number of clusters to 3, divide the level intervals based on the risk value distribution, and output the grading results.

[0108] While the specific embodiments of the present invention have been described in detail above, they are intended only as examples, and the present invention is not limited to the specific embodiments described above. Those skilled in the art should understand that the above embodiments and descriptions are merely illustrative of the principles of the present invention, and that various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A drug risk monitoring method based on multi-source data fusion, characterized by: The steps include: Acquire multi-source data and perform structured processing on the raw data using pre-set format conversion rules to obtain a uniformly coded first dataset. Then, based on missing values ​​and outliers in the first dataset, use statistical interpolation and anomaly detection algorithms to clean and repair them, resulting in a pre-processed second dataset with consistent fields. Obtain the source attributes of the second data set, calculate the sample size and update frequency of each data source, determine the credibility weight of each data source, obtain a third data set with weight labels, extract features from the third data set, and transform the heterogeneous features into a standardized fourth data set through a preset feature mapping table; Multi-source feature data is obtained through the fourth data set. The Pearson correlation coefficient in the weighted collaborative filtering algorithm is used to calculate the weight value of each data source to obtain a weight set. The weight value of each data source is then extracted based on the weight set. When the weight of a data source is lower than the preset threshold, the fusion coefficient of the corresponding feature is reduced to obtain an adjusted weight set. The adjusted weight set is used to perform weighted fusion processing on multi-source features to generate a fused feature set. Potential risk signals are extracted through the fused feature set to obtain an intermediate feature set containing risk markers. Then, we obtain the intermediate feature set, use logistic regression analysis to analyze risk markers and screen significant risk features to obtain a significant risk feature set. Based on the significant risk feature set, we combine the weighted collaborative filtering algorithm to optimize the fusion results and generate a fifth data set containing potential risk signals. Analyze the distribution characteristics of risk signals using anomaly detection tools on the fifth data set to obtain a risk distribution data set; Obtain the time series data from the fifth dataset, calculate the changing trends of each drug adverse reaction using a sliding window technique, and obtain the sixth dataset. Based on the abnormal trends in the sixth dataset, use a Bayesian network to perform probabilistic reasoning on risk signals, identify high-risk drugs and their quantitative assessment values, and obtain the seventh dataset. Feature vectors of high-risk drugs are extracted from the seventh data set, and the newly added data is incrementally updated through the real-time stream processing framework to obtain a dynamically adjusted eighth data set. Based on the risk assessment values ​​in the eighth data set, a ninth data set containing drug names and risk levels is generated.

2. The drug risk monitoring method based on multi-source data fusion according to claim 1, characterized in that: The obtaining of the pre-processed second data set with consistent fields specifically includes: Obtain the missing value distribution from the first data set through statistical methods, fill the missing values ​​using mean interpolation to obtain a preliminary filled data set, and then use the isolation forest algorithm to identify abnormal data in the preliminary filled data set and determine the location of abnormal points; The outlier features are obtained from the outlier points. The outlier data is repaired using the median replacement method to obtain a repaired dataset. Then, based on the field attributes of the repaired dataset, the field format is standardized to obtain a consistent format dataset. If outliers still exist in the consistent format dataset, a clustering algorithm is used to detect the outliers and determine whether further repair is required. The normal data range in the clustering results is obtained, and the outliers beyond the range are removed by elimination method to obtain the preprocessed data set. The integrity of the preprocessed data set is checked to determine the field consistency and obtain the second data set.

3. The drug risk monitoring method based on multi-source data fusion according to claim 1, characterized in that: The obtaining of the third data set with weight labels specifically includes: The data source and sample size are extracted from the source attributes of the second data set. The distribution characteristics of the sample size are calculated using statistical methods to obtain the sample size index of each data source. The frequency of the data source is obtained based on its update frequency. When the frequency exceeds a preset threshold, it is determined to be a high-frequency data source and the update frequency classification result is obtained. Based on the sample size index and update frequency classification, a credibility score is obtained from a pre-established credibility assessment model to determine the credibility level of each data source; Based on the credibility level and sample size indicators, a linear regression algorithm is used to calculate the weight value to obtain a preliminary weight tag. The preliminary weight tag and update frequency classification are obtained. When the credibility level does not match the frequency, the weight value is adjusted to obtain an optimized weight-tagged data set. The clustering algorithm is then used to divide the data source categories and determine the third data set with weight tags. The consistency of the calculation results is verified through the weight tags of the third data set, and the integrity of the tag generation process is judged to obtain the final third data set.

4. The drug risk monitoring method based on multi-source data fusion according to claim 1, characterized in that: The converting of heterogeneous features into a standardized fourth data set using a preset feature mapping table specifically includes: Parse the drug name, adverse reaction description, and timestamp from the third data set, use a text extraction algorithm to generate a preliminary feature set, match the heterogeneous features in the preliminary feature set through a preset feature mapping table to obtain a mapped feature set, and when there are missing values ​​in the mapped feature set, associate the adverse reaction description with the timestamp, use a content filling algorithm to generate a complete feature set, compare the complete feature set with the preset table, and use a format verification algorithm to determine whether all heterogeneous features have been converted into a standardized format to obtain a standardized feature set; According to the standardized feature set, drug names and adverse reactions are grouped to generate a classification feature set. Based on the timestamps in the classification feature set, a time series distribution analysis algorithm is used to calculate the distribution pattern of adverse reactions under different drug names to obtain a distribution feature set. Then, an association analysis algorithm is used to calculate the association strength between drug names and adverse reactions to generate the fourth data set.

5. The drug risk monitoring method based on multi-source data fusion according to claim 1, characterized in that: The obtaining of the sixth data set specifically includes: Obtain time series data from the fifth data set, use a preset sliding window technique to calculate the data point set within each time period, and obtain a preliminary change trend. Then, based on the preliminary change trend, calculate the sliding window mean and standard deviation of each adverse drug reaction to obtain a quantified change trend indicator; Based on the quantified change trend index, determine whether the change trend index of a certain drug exceeds the preset threshold. If it exceeds, mark it as abnormal, and obtain a preliminary judgment result of the trend abnormality; Based on the preliminary judgment results of trend anomalies, the corresponding drug list is extracted, and the abnormal drugs are grouped using a clustering algorithm to obtain a classified abnormal drug set. The time series characteristics of adverse reactions in each group are then obtained from the classified abnormal drug set. The differences in the changing trends between the groups are determined to obtain the detailed analysis results of the abnormal trend; Based on the refined analysis results of the abnormal trend, a data set including time series features and a list of abnormal drugs is generated to obtain the sixth data set.

6. The drug risk monitoring method based on multi-source data fusion according to claim 1, characterized in that: The obtaining of the seventh data set specifically includes: The time series features of abnormal trends were extracted from the sixth data set. A Bayesian network was used to construct a probability model for risk signals. The Markov Chain Monte Carlo method was used to obtain initial probability distribution data. The conditional probability of each adverse drug reaction was calculated based on the initial probability distribution data. The initial strength of the risk signal was determined, and a signal strength distribution set was obtained. Abnormal signals exceeding the preset threshold are extracted from the signal intensity distribution set, and the weighted average method is used to calculate the quantitative risk value of the drug to obtain a preliminary list of high-risk drugs. Then, the corresponding time series data is obtained, and statistical tools are used to analyze the significance of the change trend to obtain a set of quantitative assessment values; The risk level of each drug is determined by a set of quantitative assessment values. When the quantitative assessment value exceeds the preset threshold, the drug is marked as high-risk, and a classified drug list is obtained. Based on the classified drug list, the characteristic vector of the change trend between groups is extracted, and the Bayesian network is used for secondary probability inference to obtain the optimized risk probability distribution. The final quantitative assessment value of high-risk drugs is calculated using the optimized risk probability distribution. A data integration tool is used to generate the seventh data set to obtain complete risk assessment data. The distribution characteristics of abnormal trends are then obtained. The data consistency is verified using a circular comparison method to obtain a confirmed risk signal set. The dynamic characteristics of high-risk drugs are extracted through the confirmed risk signal set, and the time series data are fused using the incremental update algorithm to obtain the adjusted risk assessment results.

7. The drug risk monitoring method based on multi-source data fusion according to claim 1, characterized in that: The dynamically adjusted eighth data set specifically includes: Extract feature vectors of high-risk drugs from the seventh data set to obtain an initial feature set. Then, use a real-time stream processing framework to analyze the newly added data to determine whether it contains high-risk drug information and obtain an updated data subset. The initial feature set is fused through the incremental update algorithm to generate an adjusted feature vector set. When the adjusted feature vector set exceeds the preset threshold, the feature weights are updated through the dynamic adjustment mechanism to obtain the optimized feature set. Based on the optimized feature set, the data generation module is used to construct the eighth data set to complete the dynamic update of the data set.

8. The drug risk monitoring method based on multi-source data fusion according to claim 1, characterized in that: The generating of the ninth data set including the drug name and risk level specifically includes: Obtaining the risk assessment value from the eighth data set, determining whether it exceeds a preset threshold, obtaining an exceeding-standard record, and then extracting the name record of the exceeding-standard drug based on the exceeding-standard record to determine the trigger state; Generate a temporary data set containing drug names through the trigger state, obtain the integrity of the temporary data set, use the random forest algorithm to process the temporary data set, extract risk level features, and obtain the level generation result; The ninth data set is constructed using the level generation results and name records to determine data integrity. The risk levels in the ninth data set are then verified to be consistent with the assessment of the eighth data set to obtain a verification result. The data association in the ninth data set is updated through the verification results, and the updated data set is determined. Then, combined with the exceeding-standard records, data including drug names and risk levels are generated.

Images