Pathogen transmission rapid early warning and traceability analysis method based on multi-source data fusion

Abstract

The invention discloses a pathogen transmission rapid early warning and traceability analysis method based on multi-source data fusion. The method comprises the steps of S1, collecting multi-source data such as drug sales, network behaviors, social media, outpatient diagnosis and traffic time and space; s2, through cleaning and standardized preprocessing, social text disease features are extracted by adopting BERT; s3, mining a comprehensive weak signal through single-source anomaly detection and multi-source correlation analysis; s4, fusing the features by using an Attention-LSTM model, and outputting a regional risk index; s5, training an early warning model based on historical data, and setting a three-level threshold to trigger early warning; and S6, positioning a propagation starting point in combination with the spatio-temporal data, and constructing a propagation chain through a graph neural network. The early warning is advanced by 3-7 days, the traceability precision reaches the community level, and the prevention and control precision is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of pathogen early warning and tracing, in particular to a pathogen transmission rapid early warning and tracing analysis method based on multi-source data fusion. BACKGROUND

[0002] Currently, the early warning and tracing of pathogen transmission mainly rely on the confirmed case reports of official medical systems, traditional epidemiological investigations and the like, and there are the following significant deficiencies:

[0003] The existing technologies focus on the internal data of medical systems, and ignore the weak signals contained in non-medical scenarios such as drug sales, network searches and social media. Such signals often appear before the confirmed cases and are the key clues for early warning. Multi-source data has problems such as format heterogeneity, large dimension difference and strong noise interference. The existing technologies lack an efficient fusion mechanism and cannot integrate the scattered weak signals into comprehensive features with early warning value. Traditional tracing relies on manual epidemiological investigations, which is time-consuming and labor-intensive and is easily affected by memory bias and incomplete information, making it difficult to quickly locate the transmission starting point and transmission chain and providing support for precise management.

[0004] Therefore, the present application is proposed. SUMMARY

[0005] To solve the above technical problems, the basic idea of the technical solution of the present application is:

[0006] The pathogen transmission rapid early warning and tracing analysis method based on multi-source data fusion comprises the following steps:

[0007] Step S1: Multi-source data acquisition, acquiring drug sales data, third-party detection agency pathogen detection data, network behavior data, social media data and supplementary data, wherein the supplementary data includes laboratory detection results of "flu-like symptoms" in community clinics, medical records and public transportation space-time data;

[0008] Step S2: Multi-source data preprocessing, performing data cleaning, data standardization and feature extraction on the acquired heterogeneous data, wherein the feature extraction includes extracting the disease keyword frequency and disease concentration of social media text by using the BERT pre-training model;

[0009] Step S3: Weak signal mining, screening out comprehensive weak signals through single-source data anomaly detection and multi-source signal association analysis;

[0010] Step S4: Multi-source data fusion, performing feature fusion on the comprehensive weak signals and supplementary data by using an Attention-LSTM model based on an attention mechanism, and outputting a regional comprehensive risk index;

[0011] Step S5: Early warning model construction and warning triggering. The early warning model is trained based on historical pathogen transmission data, and three-level warning thresholds are set and corresponding warnings are triggered.

[0012] Step S6: Source tracing analysis, combining multi-source spatiotemporal data to locate the core starting point of propagation, and using graph neural networks to construct a propagation chain map.

[0013] In a preferred embodiment of the present invention, in step S1, the drug sales data includes the sales volume, sales growth rate, and inventory change data of antiviral, antipyretic, and antitussive drugs in a specific region and at a specific time period; the network behavior data includes the search volume, search growth rate, and spatiotemporal distribution of searchers of keywords related to pathogen infection symptoms in a specific region; and the social media data includes symptom description text with spatiotemporal tags and the discussion popularity of symptom-related topics.

[0014] In a preferred embodiment of the present invention, in step S2, the data cleaning adopts a strategy that combines outlier removal and missing value completion. Missing value completion adopts the interpolation method of the average of the previous 3 days or the filling method of the average of the same type of data in the same region. The data standardization includes calculating the regional sales growth index and the regional search popularity value.

[0015] In a preferred embodiment of the present invention, in step S3, the single-source data anomaly detection adopts... According to the principle or sliding window detection method, if the growth rate index of drug sales in a certain region is greater than... If the search popularity value increases for three consecutive days with a daily growth rate of >50%, and the concentration of social media symptoms is >2 times the average of the past 14 days, then these are respectively identified as abnormal signals for drug sales, abnormal signals for searches, and abnormal signals for social media symptoms. The multi-source signal correlation analysis constructs a signal correlation matrix. If the temporal overlap of abnormal signals from different sources is ≤2 days and the spatial overlap is ≥0.7, then they are integrated into a comprehensive weak signal.

[0016] In a preferred embodiment of the present invention, in step S4, the attention mechanism layer of the Attention-LSTM model assigns a weight of 0.35 to drug sales data, a weight of 0.3 to search data, a weight of 0.25 to social data, and a weight of 0.1 to traffic data. The LSTM fusion layer captures the temporal change trend of each source feature and outputs a comprehensive risk index, wherein the comprehensive risk index ranges from 0 to 1.

[0017] As a preferred embodiment of the present application, in step S5, the early warning model adopts an XGBoost gradient boosting tree or a CNN-LSTM model, the training input is a comprehensive risk index, and the training label is "whether there is an official confirmed case in the region within the next 7 days"; the three-level early warning threshold includes: a comprehensive risk index ∈ [0.6, 0.8) triggers a general early warning, ∈ [0.8, 0.9) triggers an attention early warning, and ≥ 0.9 triggers an emergency early warning.

[0018] As a preferred embodiment of the present application, in step S6, the propagation core starting point positioning is confirmed by screening the publisher's resident area of the earliest disease description text, combined with the abnormal time of drug sales and the time of outpatient treatment in the area; the propagation chain construction adopts a graph neural network, the node is an area, the edge is the personnel flow intensity between areas, and the edge weight is the time difference of disease abnormal signals between two areas. The high-frequency propagation path is mined by a shortest path algorithm.

[0019] As a preferred embodiment of the present application, in step S1, the keyword search volume of the network behavior data is obtained by a search engine API, and a time series decomposition method is used to extract a trend component and a seasonal component, wherein the trend component is used to calculate a search growth rate.

[0020] As a preferred embodiment of the present application, according to claim 4, in step S3, the sliding window detection method adopts a moving average line with a window size of 7 days, and the abnormal detection threshold is 2 times the standard deviation of the moving average line.

[0021] As a preferred embodiment of the present application, according to claim 7, in step S6, the graph neural network adopts a graph attention network (GAT) for node embedding, and the edge weight in the propagation chain graph is normalized as wherein, and are the disease abnormal signal timestamps of area i and area j, respectively; the shortest path algorithm adopts a Dijkstra algorithm, and the path cost is defined as the reciprocal of the edge weight, and the calculation formula is as follows: The minimum cost path is mined as a high-frequency propagation path.

[0022] Compared with the prior art, the present application has the following beneficial effects:

[0023] The application can intervene in prevention and control in the early stage of pathogen transmission and reduce the transmission range by mining weak signals such as drug sales, search and social media, and the early warning time is 3-7 days earlier than the official diagnosis case report; the weak signal identification accuracy is improved by effectively excluding the misjudgment of single data fluctuation through single-source anomaly detection and multi-source correlation analysis; the fusion model based on the attention mechanism can automatically allocate feature weights, avoiding the subjectivity of manually setting weights; combined with the spatiotemporal data and the GNN algorithm, the transmission starting point can be located within 24 hours after the early warning trigger, the key nodes of the transmission chain are clear, and the tracing accuracy can reach the community level, providing decision support for precise prevention and control.

[0024] The specific embodiments of the application will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0025] In the drawings:

[0026] Figure 1 The flowchart of the pathogen transmission rapid early warning and tracing analysis method based on multi-source data fusion. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the embodiments of the application more clear, the technical scheme in the embodiments will be clearly and completely described below with reference to the drawings of the embodiments of the application, and the following embodiments are used to illustrate the application.

[0028] The pathogen transmission rapid early warning and tracing analysis method based on multi-source data fusion comprises the following steps:

[0029] Step S1: multi-source data acquisition, collecting drug sales data, network behavior data, social media data and supplementary data, the supplementary data comprising 'flu-like symptoms' medical records of community clinics and public transportation spatiotemporal data;

[0030] Step S2: multi-source data preprocessing, performing data cleaning, data standardization and feature extraction on the collected heterogeneous data, the feature extraction comprising extracting disease keyword frequency and disease concentration from social media text using a BERT pre-training model;

[0031] Step S3: weak signal mining, screening out comprehensive weak signals through single-source data anomaly detection and multi-source signal correlation analysis;

[0032] Step S4: multi-source data fusion, performing feature fusion on the comprehensive weak signals and the supplementary data using an Attention-LSTM model based on the attention mechanism, and outputting a regional comprehensive risk index;

[0033] Step S5: early warning model construction and early warning trigger, training the early warning model based on historical pathogen transmission data, setting three-level early warning thresholds and triggering corresponding early warnings;

[0034] Step S6: Traceability analysis, combined with multi-source spatio-temporal data to locate the propagation core starting point, and graph neural network is used to construct the propagation chain graph.

[0035] Further, in step S1, the drug sales data includes the sales quantity, sales growth rate, and inventory change data of antiviral, antipyretic, and cough medicines in a specific region and period; the network behavior data includes the search volume, search growth rate, and spatio-temporal distribution of searchers of keywords related to the symptoms of pathogen infection in a specific region; and the social media data includes texts describing the symptoms with spatio-temporal tags and the discussion heat of topics related to the symptoms.

[0036] Further, in step S2, the data cleaning adopts a strategy combining outlier rejection and missing value completion, and the missing value completion adopts a 3-day average interpolation method or a same-region and same-type data average filling method; and the data standardization includes calculating the regional sales growth rate index and the regional search heat value.

[0037] Further, in step S3, the single-source data anomaly detection adopts the principle or sliding window detection method, and if the drug sales growth rate index of a region is greater than , the search heat value has continuously increased for 3 days and the single-day growth rate is greater than 50%, and the social media symptom concentration is greater than twice the average value in the recent 14 days, it is determined as a drug sales anomaly signal, a search anomaly signal, and a social symptom anomaly signal, respectively; the multi-source signal correlation analysis constructs a signal correlation matrix, and if the time overlap degree of different source anomaly signals is less than or equal to 2 days and the spatial overlap degree is greater than or equal to 0.7, it is integrated as a comprehensive weak signal.

[0038] In the principle, for a data sequence , the mean , and the standard deviation .

[0039] Anomaly signal determination: if + , it is considered as an anomaly.

[0040] Further, in step S4, the attention mechanism layer of the Attention-LSTM model assigns a weight of 0.35 to the drug sales data, a weight of 0.3 to the search data, a weight of 0.25 to the social data, and a weight of 0.1 to the traffic data, the LSTM fusion layer captures the time sequence change trend of each source feature and outputs a comprehensive risk index, and the value range of the comprehensive risk index is 0-1.

[0041] Further, in step S5, the early warning model adopts an XGBoost gradient boosting tree or a CNN-LSTM model, the training input is a comprehensive risk index, and the training label is whether an official confirmed case occurs in the region within the next 7 days; the three-level early warning threshold includes: a comprehensive risk index ∈ [0.6, 0.8) triggers a general early warning, ∈ [0.8, 0.9) triggers an attention early warning, and ≥ 0.9 triggers an emergency early warning.

[0042] Further, in step S6, the propagation core starting point positioning is confirmed by screening the publisher's resident area of the earliest disease description text, combined with the abnormal time of drug sales and the time of outpatient visits in the area; the propagation chain construction adopts a graph neural network, the nodes are regions, the edges are the intensity of personnel flow between regions, the edge weight is the time difference of disease abnormal signals between two regions, and the high-frequency propagation path is mined by a shortest path algorithm.

[0043] Further, in step S1, the keyword search volume of the network behavior data is obtained by a search engine API, and a time series decomposition method is used to extract a trend component and a seasonal component, wherein the trend component is used to calculate the search growth rate.

[0044] Further, in step S3, the sliding window detection method adopts a moving average line with a window size of 7 days, and the abnormal detection threshold is 2 times the standard deviation of the moving average line.

[0045] The calculation formula of the sliding window detection threshold is: wherein, = , = .

[0046] Further, in step S6, the graph neural network adopts a graph attention network (GAT) for node embedding, and the edge weight in the propagation chain graph is normalized as wherein, and are the disease abnormal signal timestamps of regions i and j, respectively; the shortest path algorithm adopts a Dijkstra algorithm, and the path cost is defined as the reciprocal of the edge weight, and the calculation formula is as follows: The minimum cost path is mined as a high-frequency propagation path.

[0047] Embodiment 1

[0048] The specific embodiments of the present application are described in detail below in combination with the influenza pathogen propagation monitoring examples of Shanghai from November to December 2023. Those skilled in the art can implement the present application based on the description, but it should not be understood as limiting the scope of the application.

[0049] Data collection stage, drug sales data: Collect daily sales data of 200 chain pharmacies in 30 streets, including sales quantity, sales growth and inventory changes, including sales quantity, sales growth and inventory changes, a total of 60 days (November 1 to December 30) of data.

[0050] Network behavior data: Collect search data of 15 core keywords related to influenza symptoms in the city through Baidu search API, collect search data by hour, and extract trend components using STL decomposition method to calculate search growth.

[0051] Social media data: Collect disease description texts with time and space tags through Weibo and WeChat public number API, a total of 120,000 valid texts, extract publication time, positioning information and symptom description, and collect related topic heat data.

[0052] Supplementary data: Collect influenza-like symptom visit records of community health service centers in 30 streets, as well as passenger flow data of subways, buses and railways, including daily passenger flow of each line and passenger origin / destination distribution.

[0053] Data preprocessing stage: Data cleaning, using box plot method to remove outliers in drug sales data, specifically data with sales quantity exceeding 3 times the average value of the past 30 days, and filling in missing data by the average value of the first 3 days; For social media data, irrelevant texts are removed by keyword matching, and 85,000 valid texts are retained.

[0054] Data standardization: Use min-max standardization to map drug sales growth to the [0,1] interval, and use Z-score standardization to convert search volume to search heat value.

[0055] Feature extraction: Use BERT pre-training model to process social media texts and extract daily disease keyword frequency and disease concentration in each street.

[0056] Weak signal mining stage: Single-source anomaly detection: Apply the 3σ principle (mean μ=0.22, standard deviation σ=0.11, anomaly threshold 0.55) to drug sales data, and find that the sales growth index of A, B and C streets on November 25 exceeds the threshold; For search data, use sliding window detection method, find that the search heat of A and B streets increased by more than 50% from November 26 to 28; For social media data, use sliding window detection method, find that the disease concentration of A, B and D streets on November 27 exceeds twice the average value of the past 14 days.

[0057] Multi-source signal correlation analysis: Construct signal correlation matrix, calculate time overlap and spatial overlap. The time overlap of A street is 0.8, and the spatial overlap is 1.0, which is integrated into a weak signal. The time overlap of B street is 0.75, and the spatial overlap is 1.0, which is also integrated into a weak signal. C and D streets are not integrated because they only have single signals.

[0058] Multi-source data fusion stage: The weak signals of A and B streets and the supplementary data are input into the Attention-LSTM model, and the weighted fusion is performed according to the weight distribution scheme (0.35 for drug sales data, 0.3 for search data, 0.25 for social data, and 0.1 for traffic data). The comprehensive risk index is output after the LSTM layer processing: A street is 0.88, and B street is 0.82.

[0059] Early warning trigger stage: XGBoost model (trained based on 2022 influenza epidemic data, AUC value 0.93) is used for early warning determination. According to the three-level early warning threshold, the comprehensive risk index of A street and B street belongs to the range of [0.8, 0.9), triggering attention warning, and pushing the early warning information to the city CDC, suggesting to start community investigation and public place disinfection.

[0060] Traceability analysis stage: Transmission core starting point positioning: Identify the earliest disease description text from social media data (published on November 20, with a permanent area of A street), combined with drug sales and outpatient abnormal time (A street is earlier than B street), confirm A street as the transmission core starting point.

[0061] Transmission chain map construction: Build a graph neural network with 30 streets as nodes to calculate the edge weight between regions (based on the time difference of abnormal signals, such as the weight between A and B streets is 0.5). Use Dijkstra algorithm to mine high-frequency transmission paths, the path A→B→E has the lowest cost (4.5), and is determined as the high-frequency transmission path.

[0062] Table 1: Weak signal detection method and abnormal judgment standard

[0063] Data source Detection method Key parameters and thresholds Detection results (example) Drug sales 3σ principle Mean μ = 0.22, standard deviation σ = 0.11, abnormal threshold μ + 3σ = 0.55 On November 25, the growth rate index of A, B, and C streets exceeded the threshold Web search Sliding window method Window size: 7 days; single-day growth rate threshold: > 50% From November 26 to 28, the heat value of A and B streets exceeded the threshold for 3 consecutive days Social media Sliding window method Window size: 14 days; multiple threshold: > 2 times the mean On November 27, the symptom concentration of A, B, and D streets exceeded the threshold

[0064] Table 2: Multi-source signal correlation and fusion model

[0065] Project Content description Association determination condition Time overlap = intersection duration / union duration; spatial overlap = signal coverage area intersection A street association result Drug (11 / 25), search (11 / 26-28), and social (11 / 27) signals; time overlap = 0.8, spatial overlap = 1.0 → integrated into a comprehensive weak signal B street association result Time overlap = 0.75, spatial overlap = 1.0 → integrated into a comprehensive weak signal C / D street result Only single signal → not integrated Fusion model Attention-LSTM model Weight distribution Drug sales (0.35), search data (0.30), social data (0.25), and traffic data (0.10) Fusion output A street comprehensive risk index: 0.88; B street comprehensive risk index: 0.82

[0066] Table 3: Early warning trigger and traceability analysis

[0067] Stage Method / model Key input / parameters Output and conclusion Early warning trigger XGBoost classification model AUC = 0.93; threshold: [0.8, 0.9) for attention warning A street (0.88) and B street (0.82) both triggered attention warning Core starting point positioning Time sequence analysis The earliest social media text (11 / 20, A street) and the time of abnormal signals in each street Confirming A street as the core starting point of transmission Transmission chain construction Graph neural network Nodes: 30 streets; edge weights: 1 / (1 + abnormal signal time difference) A → B edge weight = 0.5; A → D edge weight = 0.25 High-frequency path mining Dijkstra algorithm Calculate path cost (sum of inverse weights) Path A → B → E has the lowest cost (4.5), determined as the high-frequency transmission path

[0068] Implementation effect verification

[0069] Table 4: Implementation effect verification and performance indicators

[0070] Verification index Implementation results of the invention Comparison benchmark / explanation Early warning time 5 days (A street warning on November 25 vs. first confirmed case on November 30) Compared with official confirmed case reporting time Weak signal identification accuracy 92% Verified by subsequent confirmed cases Accuracy of tracing Street level Accurate positioning to specific streets Accuracy of transmission chain mining 88% The predicted path A → B → E is consistent with the actual order of confirmed cases Effect of epidemic prevention and control Transmission range reduced by 30% Compared with the same period last year Cost-effectiveness of prevention and control Prevention and control cost reduced by 25% Compared with the same period last year

[0071] Through the follow-up monitoring data of the city CDC, the first confirmed case of influenza occurred in A Street on November 30, and confirmed cases occurred in B Street and E Street on December 2 and 4, respectively, which is highly consistent with the early warning results and transmission chain analysis of the application. The early warning time of the application is 5 days earlier than the official report, the weak signal recognition accuracy is 92%, the tracing accuracy reaches the street level, the transmission chain mining accuracy is 88%, effectively reduces the epidemic transmission range by 30%, and reduces the prevention and control cost by 25%.

Claims

1. A method for rapid early warning and source tracing analysis of pathogen transmission based on multi-source data fusion, characterized in that, Includes the following steps: Step S1: Multi-source data collection, including drug sales data, online behavior data, social media data, and supplementary data; Step S2: Multi-source data preprocessing, which involves data cleaning, data standardization and feature extraction of the collected heterogeneous data. The feature extraction includes using a BERT pre-trained model to extract the frequency of disease keywords and the concentration of disease symptoms from social media texts. Step S3: Weak signal mining, through single-source data anomaly detection and multi-source signal correlation analysis, to screen out comprehensive weak signals; Step S4: Multi-source data fusion. The Attention-LSTM model based on the attention mechanism is used to fuse features of the comprehensive weak signal and supplementary data, and output the comprehensive regional risk index. Step S5: Early warning model construction and warning triggering. The early warning model is trained based on historical pathogen transmission data, and three-level warning thresholds are set and corresponding warnings are triggered. Step S6: Source tracing analysis, combining multi-source spatiotemporal data to locate the core starting point of propagation, and using graph neural networks to construct a propagation chain map.

2. The method for rapid early warning and source tracing analysis of pathogen transmission based on multi-source data fusion according to claim 1, characterized in that, In step S1, the drug sales data includes the sales volume, sales growth rate, and inventory change data of influenza treatment drugs in a specific region and at a specific time period; the online behavior data includes the search volume, search growth rate, and spatiotemporal distribution of searchers for keywords related to pathogen infection symptoms in a specific region; and the social media data includes symptom description text with spatiotemporal tags and the discussion popularity of symptom-related topics.

3. The method for rapid early warning and source tracing analysis of pathogen transmission based on multi-source data fusion according to claim 1, characterized in that, In step S2, the data cleaning adopts a strategy that combines outlier removal and missing value completion. Missing value completion adopts the interpolation method of the average of the previous 3 days or the filling method of the average of the same type of data in the same region. The data standardization includes calculating the regional sales growth index and the regional search popularity value.

4. The method for rapid early warning and source tracing analysis of pathogen transmission based on multi-source data fusion according to claim 1, characterized in that, In step S3, the single-source data anomaly detection adopts... The principle or sliding window detection method.

5. The method for rapid early warning and source tracing analysis of pathogen transmission based on multi-source data fusion according to claim 1, characterized in that, In step S4, the attention mechanism layer of the Attention-LSTM model assigns a weight of 0.35 to drug sales data, a weight of 0.3 to search data, a weight of 0.25 to social data, and a weight of 0.1 to traffic data. The LSTM fusion layer captures the temporal change trend of each source feature and outputs a comprehensive risk index, the comprehensive risk index of which ranges from 0 to 1.

6. The method for rapid early warning and source tracing analysis of pathogen transmission based on multi-source data fusion according to claim 1, characterized in that, In step S5, the early warning model adopts an XGBoost gradient boosting tree or a CNN-LSTM model, and the training input is a comprehensive risk index.

7. The method for rapid early warning and source tracing analysis of pathogen transmission based on multi-source data fusion according to claim 1, characterized in that, In step S6, the core origin of the propagation is located by screening the area where the publisher of the earliest symptom description text resides, combined with the abnormal drug sales time and outpatient visit time in that area; the propagation chain is constructed using a graph neural network, where nodes are areas, edges are the intensity of personnel flow between areas, and edge weights are the time difference of the abnormal symptom signals between two areas, and high-frequency propagation paths are mined through the shortest path algorithm.

8. The method for rapid early warning and source tracing analysis of pathogen transmission based on multi-source data fusion according to claim 1, characterized in that, In step S1, the keyword search volume of the network behavior data is obtained through the search engine API, and the trend component and seasonal component are extracted using the time series decomposition method, wherein the trend component is used to calculate the search growth rate.

9. The method for rapid early warning and source tracing analysis of pathogen transmission based on multi-source data fusion according to claim 1, characterized in that, In step S3, the sliding window detection method uses a 7-day moving average with an anomaly detection threshold of twice the standard deviation of the moving average.

10. The method for rapid early warning and source tracing analysis of pathogen transmission based on multi-source data fusion according to claim 1, characterized in that, In step S6, the graph neural network uses a graph attention network for node embedding, and the edge weights in the propagation chain graph are normalized to... in, and These are the timestamps of the abnormal disease signals in regions i and j, respectively; the shortest path algorithm uses Dijkstra's algorithm, and the path cost is defined as the reciprocal of the edge weight, calculated as follows: The path with the lowest cost is selected as the high-frequency propagation path.

Images