Flu trend prediction method and device based on multimodal data

Abstract

The invention discloses a method and device for predicting influenza trends based on multimodal data, belonging to the technical field of combining epidemic monitoring with artificial intelligence, comprising: extracting three types of comprehensive feature vectors corresponding to Baidu index mode, meteorological data mode, and graph data mode; performing weighted summation on the comprehensive feature vectors corresponding to each mode based on attention weights calculated by an attention mechanism to obtain a total comprehensive feature vector; performing deep feature fusion on the total comprehensive feature vector using a prediction model including a multi-head self-attention mechanism, a feedforward neural network, a multimodal interaction layer, and multi-level feature fusion, and predicting an influenza positive rate based on the fused features to obtain an influenza trend, thereby improving the accuracy and reliability of influenza trend prediction.

Description

Technical Field

[0001] The present invention belongs to the technical field of combining epidemic monitoring and artificial intelligence, and specifically relates to a method and device for predicting influenza trends based on multimodal data. Background Art

[0002] Seasonal influenza poses a persistent and serious threat to public health. Since influenza A virus has a wide host range and can infect wild birds, poultry, pigs, horses, humans and other animals, it is unrealistic to completely eliminate influenza virus from the perspective of natural ecological distribution, and influenza surveillance in each region is of great significance to public health in other regions and even the world.

[0003] Traditional influenza surveillance systems that rely on case reports often have difficulty detecting influenza in a timely manner due to delays in medical treatment, laboratory confirmation, and reporting procedures. These delays hinder early warning and rapid response to influenza. To address this problem, researchers have explored methods for early influenza detection using alternative data sources, such as analyzing over-the-counter drug sales and school absence data. However, these methods still have limitations in the timeliness and coverage of the data.

[0004] With the development of big data technology, using Internet search engine data to monitor flu trends has become a new method. Using Baidu Index to predict flu trends can obtain search data in a timely manner after flu-related events occur, providing real-time dynamics of flu, such as a flu trend prediction method based on Baidu Index and machine learning disclosed in a patent application with publication number CN116313134A.

[0005] Using population mobility to monitor influenza trends has also become a new method, for example, a method for predicting the trend of influenza incidence in a city is disclosed in a patent application with publication number CN114388137A.

[0006] However, the above-mentioned single data source still has shortcomings in the accuracy of influenza prediction. How to combine data from multiple sources to make influenza predictions to improve prediction accuracy has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] In view of the above, the purpose of the present invention is to provide a method and device for influenza trend prediction based on multimodal data, which integrates Baidu index data, meteorological data and heat map data to more comprehensively understand the transmission characteristics of influenza and improve the accuracy and reliability of influenza trend prediction.

[0008] To achieve the above-mentioned purpose of the invention, an embodiment provides a method for predicting influenza trends based on multimodal data, comprising the following steps:

[0009] Collect and pre-process Baidu index data of influenza-related keywords, meteorological data, and population-related graph data;

[0010] Extract the feature vector of Baidu Index for each keyword, and fuse the feature vectors of Baidu Index for all keywords to obtain a comprehensive feature vector corresponding to the Baidu Index modality;

[0011] The wind direction data in the meteorological data is vectorized and the numerical data is standardized before being spliced, and the feature vector of the splicing result is extracted as the comprehensive feature vector corresponding to the meteorological data mode;

[0012] Extract the feature vector of each type of graph in the graph data and fuse them to obtain the comprehensive feature vector corresponding to the graph data modality;

[0013] The comprehensive feature vectors corresponding to each modality are weighted based on the attention weights calculated by the attention mechanism and then fused to obtain the total comprehensive feature vector;

[0014] A prediction model including a multi-head self-attention mechanism, a feedforward neural network, a multimodal interaction layer, and multi-level feature fusion is used to perform deep feature fusion on the total comprehensive feature vector and predict the influenza positivity rate based on the fused features to obtain the influenza trend.

[0015] Preferably, an LSTM model with an attention mechanism is used to extract the feature vector of the Baidu Index of each keyword, including:

[0016] The first implicit state of each keyword is extracted through the LSTM model, the first attention score is calculated according to the first implicit state, the first attention weight is calculated according to the first attention score, and the first implicit state is weighted and summed according to the first attention weight to obtain the characteristic vector of the Baidu Index of each keyword.

[0017] Preferably, a CNN-LSTM model with an attention mechanism is used to extract a feature vector of the splicing result, including:

[0018] CNN is used to extract the initial feature vector of the splicing result and input it into LSTM. The second implicit state of the initial feature vector is extracted in LSTM, and the second attention score is calculated according to the second implicit state. The second attention weight is calculated according to the second attention score, and the second implicit state is weighted and summed according to the second attention weight to obtain the feature vector of the splicing result.

[0019] Preferably, a ResNet model is used to extract feature vectors of each type of graph in the graph data, including:

[0020] The graph data includes three types of graphs: population heat map, traffic flow map, and population density map. Each type of graph is input into the ResNet model, and the feature vector of each type of graph is obtained through forward reasoning calculation.

[0021] Preferably, a prediction model including a multi-head self-attention mechanism, a feedforward neural network, a multimodal interaction layer, and a multi-level feature fusion is used to perform deep feature fusion on the total comprehensive feature vector and predict the influenza positive rate based on the fused features, including:

[0022] Map the total comprehensive feature vector to a high-dimensional space and generate the feature vector of the high-dimensional space Perform position coding and add position coding The eigenvector of The result obtained by multi-head self-attention calculation and the input feature vector Perform residual connection and normalize to get features ;

[0023] The characteristics The output features obtained by the feedforward neural network are Perform residual connection and normalize to get features ;

[0024] Through the multimodal interaction layer, the features Each modal feature extracted from the self-attention is self-attentioned, and the self-attention output features are fused and layer normalized to obtain the fused features ;

[0025] use Encoder pair fusion features After encoding, the influenza positivity rate was predicted.

[0026] Preferably, the self-attention output feature fusion is performed by feature splicing, that is, the self-attention output features of the modal features of all modalities are spliced ​​to achieve fusion;

[0027] Fusion features After encoding, a fully connected layer is used to predict the influenza positivity rate.

[0028] Preferably, the method further comprises: dividing the collected data into a synchronous group, a leading group, and a lagging group,

[0029] The influenza positivity rate was predicted for each group, and the weighted sum of the predicted influenza positivity rates of the three groups was taken as the final influenza positivity rate.

[0030] To achieve the above-mentioned purpose of the invention, the embodiment further provides a flu trend prediction device based on multimodal data, comprising:

[0031] A data collection module, which is used to collect and pre-process Baidu index data of influenza-related keywords, meteorological data, and population-related graph data;

[0032] The first feature extraction module is used to extract the feature vector of the Baidu Index of each keyword, and fuse the feature vectors of the Baidu Index of all keywords to obtain a comprehensive feature vector corresponding to the Baidu Index modality;

[0033] A second feature extraction module is used to vectorize the wind direction data in the meteorological data and normalize the numerical data, and then splice them, and extract the feature vector of the splicing result as the comprehensive feature vector corresponding to the meteorological data mode;

[0034] The third feature extraction module is used to extract the feature vector of each type of graph in the graph data and fuse them to obtain a comprehensive feature vector corresponding to the graph data modality;

[0035] The feature initial fusion module is used to weight the comprehensive feature vectors corresponding to each modality based on the attention weights calculated by the attention mechanism and then fuse them to obtain the total comprehensive feature vector;

[0036] The influenza prediction module is used to perform deep feature fusion on the total comprehensive feature vector using a prediction model that includes a multi-head self-attention mechanism, a feedforward neural network, a multimodal interaction layer, and multi-level feature fusion, and predict the influenza positivity rate based on the fused features to obtain the influenza trend.

[0037] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned influenza trend prediction method based on multimodal data.

[0038] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the above-mentioned influenza trend prediction method based on multimodal data.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention improves the accuracy of influenza trend prediction by integrating multimodal data sources. It is difficult for a single data source to fully reflect the complexity and dynamic changes of influenza, while Baidu Index, meteorological data and graph data provide rich influenza-related information from different perspectives. As an Internet search engine data, Baidu Index can reflect the public's attention to influenza-related symptoms and information in real time, and has a high degree of real-time performance. Meteorological data includes multiple dimensions such as temperature and humidity. These meteorological factors have a significant impact on the spread and prevalence of influenza viruses. Graph data include population heat maps, traffic flow maps and population density maps. These graph information reflects the mobility and aggregation of the crowd, which has an important impact on the risk of influenza transmission. By integrating these multimodal data, the present invention can more comprehensively understand the transmission characteristics of influenza, thereby improving the accuracy and reliability of predictions.

[0041] The present invention can realize real-time dynamic monitoring of influenza and improve early warning capabilities. Traditional influenza monitoring systems often have data delays due to their reliance on case reports and laboratory confirmations, making it difficult to timely reflect the latest trends of influenza. The Baidu Index has the characteristics of real-time updates, and can instantly obtain search data after influenza-related events occur, providing real-time dynamic information on influenza. By combining meteorological data and heat map information, the real-time nature and spatial coverage of the data can be further improved, thereby realizing real-time dynamic monitoring of influenza, timely capturing early signals of influenza, improving early warning capabilities, and providing a scientific basis for the timely adjustment of public health prevention and control measures.

[0042] The present invention also promotes the application of big data technology in the field of public health. With the development of the Internet and information technology, big data has broad application prospects in public health monitoring and prediction. Baidu Index, meteorological data and heat map information are typical representatives of big data. Through the multimodal data fusion technology of the present invention, the advantages of big data can be fully utilized to improve the accuracy and efficiency of influenza trend prediction. The present invention not only provides new technical means for influenza prevention and control, but also explores new paths for the application of big data technology in the field of public health. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 is a flow chart of a method for predicting influenza trends based on multimodal data provided in an embodiment;

[0045] Figure 2is a flowchart of a flu trend prediction process based on multimodal data provided by an embodiment;

[0046] Figure 3 Schematic diagram of the structure of a flu trend prediction device based on multimodal data provided in an embodiment. DETAILED DESCRIPTION

[0047] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0048] The inventive concept of the present invention is: the existing influenza monitoring system mainly relies on traditional case reports and laboratory confirmation, and has problems such as data delay and limited coverage, making it difficult to capture early signals of influenza in a timely manner. At the same time, the existing single data source is still insufficient in the accuracy of influenza prediction. In order to solve this problem, the present invention proposes a multimodal-based influenza trend prediction method and device, which integrates Baidu Index data, meteorological data and graph data to achieve more accurate and real-time influenza trend prediction.

[0049] like Figure 1 and Figure 2 As shown, the influenza trend prediction method based on multimodal data provided by the embodiment includes the following steps:

[0050] S1, collects and preprocesses Baidu index data of influenza-related keywords, meteorological data, and population-related graph data.

[0051] In the embodiment, the influenza positive rate ILI% data of a certain city is obtained, and the time axis of the data is determined. Keywords related to influenza are analyzed, and keywords are determined by reading and analyzing literature and communicating with experts in the field of influenza, and Baidu index data within the corresponding time axis of a certain city is obtained based on these keywords.

[0052] Determine the indicators included in the meteorological data, and obtain the meteorological data of multiple dimensional indicators of a certain city according to the corresponding time axis, including temperature, humidity, precipitation, wind speed, wind direction, air pressure, UV index, and air quality, among which temperature, humidity, precipitation, wind speed, air pressure, UV index, and air quality are numerical data, and wind direction is vector data with direction. These dimensional indicators are closely related to the spread and prevalence of influenza.

[0053] Obtain population-related graph data for a city based on the corresponding time axis, including population heat maps, traffic flow maps, and population density Figure 3 Class diagrams, these graph data reflect the mobility and aggregation of the population, which has an important impact on the risk of influenza transmission.

[0054] The collected data is preprocessed, mainly involving missing value processing and missing data completion for Baidu Index data and meteorological data. Special processing of Baidu Index data is also performed, including:

[0055] First, we used the Baidu Index time series data of influenza-related keywords, merged the synonyms in these keywords into compound words, and merged the data. Then, we conducted a Pearson correlation analysis between the influenza positive rate ILI% data and the Baidu Index of influenza-related keywords. Then, we conducted a first round of preliminary screening of influenza-related keywords, and eliminated the influenza-related keywords with weak correlation, that is, the Baidu Index with a correlation coefficient lower than ±0.4.

[0056] Then, considering the incubation period of the disease before the outbreak of influenza and the subsequent development of symptoms, the Baidu Index of influenza-related keywords was subjected to a distributed lag nonlinear model lag correlation analysis, and the distributed lag nonlinear model DLNM was used to analyze the lag correlation of the Baidu Index of influenza-related keywords. Through the lag correlation analysis, and based on the lag effect and relative risk calculation, influenza-related keywords were classified into synchronous keywords (such as fever, cough, cough, cold, etc.), leading keywords (such as influenza prevention, influenza vaccine, etc.) and lagging keywords (such as cold medicine, etc.). According to the calculation, the leading keywords in the range of 1-7 days have a significant impact on the influenza positive rate with a lag of 6 days. The lagging keywords in the range of 1-20 days have a significant impact on the influenza positive rate with a lag of 10 days.

[0057] According to the time axis and influenza positive rate ILI% data, three time axis registered data sets are prepared, which are:

[0058] Synchronous group, in which the Baidu index, weather data, and graph data of the synchronized keywords are on the same time axis, that is, the Baidu index, weather data, and graph data of the synchronized keywords on the same day correspond to each other;

[0059] The preceding group, which corresponds to the Baidu index of the preceding keyword of the current day, the meteorological data of the time axis delayed by 6 days, and the graph data of the time axis delayed by 6 days;

[0060] The lagged group includes the current day’s lagged keyword Baidu index, the meteorological data 10 days behind the time axis, and the corresponding graph data 10 days behind the time axis.

[0061] The preprocessed Baidu index data, meteorological data, and population-related graph data are all event sequences.

[0062] S2, extract the feature vector of Baidu Index for each keyword, and fuse the feature vectors of Baidu Index for all keywords to obtain a comprehensive feature vector corresponding to the Baidu Index modality.

[0063] In the embodiment, the Baidu index of each keyword in each group , j represents the index of the keyword, t represents the index of the day in the timeline, and the LSTM model with the attention mechanism is used to extract the feature vector of the Baidu Index of each keyword, including:

[0064] Build a separate LSTM model for each keyword ( ), and transform the Baidu Index into the first hidden state through the LSTM model ;

[0065] In order to specialize the feature extraction of Baidu Index and capture the important information in the Baidu Index time series, the attention mechanism is introduced into LSTM. First, according to the first hidden state Calculating the first attention score , and then based on the first attention score Calculate the first attention weight , the first attention score can be converted into Converted to the first attention weight , s is the time index, and finally based on the first attention weight For the first hidden state Perform weighted summation to obtain the feature vector of Baidu Index for each keyword ;

[0066] in and They are the weight matrix, the parameters learned from the training data, and the first attention score at time step t. represents the first hidden state at time step t The importance of the first attention weight at all time steps The sum is 1.

[0067] After obtaining the feature vectors of the Baidu Index of all keywords, the comprehensive feature vector corresponding to the Baidu Index modality is obtained by fusion. , which can be obtained by splicing , The total number of keywords.

[0068] S3, the wind direction data in the meteorological data are vectorized and the numerical data are standardized, and then spliced, and the feature vector of the splicing result is extracted as the comprehensive feature vector corresponding to the meteorological data mode.

[0069] In the embodiment, the meteorological data includes numerical data and structured data, wherein the structured data includes wind direction data. The wind direction data needs to be vectorized and converted into sine and cosine representations to avoid the periodicity problem of the angle data. Convert to a two-dimensional vector [ ],in , ;

[0070] For the numerical data in the meteorological data, since they are not in the same dimension as the Baidu Index, these meteorological values ​​need to be standardized so that their mean is 0 and their standard deviation is 1. The standardized meteorological data is expressed as ,in is the index of each indicator in the meteorological data;

[0071] The standardized numerical data and the vectorized wind direction data are spliced ​​to obtain the splicing result. , is the number of indicators in the meteorological data.

[0072] Use CNN-LSTM model with attention mechanism to extract splicing results The feature vectors include:

[0073] CNN is used to extract the initial feature vector of the splicing result and input it into LSTM, and the second hidden state of the initial feature vector is extracted in LSTM. , according to the second implicit state Calculating the Second Attention Score , according to the second attention score Calculate the second attention weight , according to the second attention weight For the second hidden state Perform weighted summation to obtain the feature vector of the splicing result As the comprehensive feature vector corresponding to the meteorological data mode .

[0074] S4, extract the feature vector of each type of graph in the graph data and fuse them to obtain the comprehensive feature vector corresponding to the graph data modality.

[0075] In the embodiment, firstly, the population heat map included in the graph data is , Traffic flow map , and population density maps Normalize the image data to the range of [0,1] and resize the image to the same scale to obtain the processed image. .

[0076] Then use the ResNet model to extract The eigenvector of , and fuse them to get the comprehensive feature vector corresponding to the graph data modality , the comprehensive feature vector can be obtained by splicing .

[0077] S5, the comprehensive feature vector corresponding to each modality is weighted based on the attention weight calculated by the attention mechanism and then fused to obtain the total comprehensive feature vector.

[0078] In the embodiment, when the comprehensive feature vectors corresponding to each modality are weighted and summed based on the attention mechanism to obtain the total comprehensive feature vector, the comprehensive feature vectors corresponding to each Baidu index modality, meteorological data modality, and image data modality are first calculated. , , Attention score , , ; Then use the Softmax function to , , Converted into attention weights respectively , , ; Then the comprehensive feature vector , , According to their respective attention weights , , Weighted features , , ; Finally, the feature , , At each time step Fusion is performed to obtain the total comprehensive feature vector of the synchronization group , Similarly, the total comprehensive eigenvectors of the preceding group and the succeeding group are similar.

[0079] S6. Use a prediction model that includes a multi-head self-attention mechanism, a feedforward neural network, a multimodal interaction layer, and multi-level feature fusion to perform deep feature fusion on the total comprehensive feature vector and predict the influenza positivity rate based on the fused features to obtain the influenza trend.

[0080] In the embodiment, in order to better predict influenza, a prediction model structure combining a multimodal Transformer architecture and a multi-level feature fusion mechanism is used for prediction.

[0081] The total comprehensive feature vector of the synchronization group obtained in step S5 The shape of is (T, d), where T represents the number of time steps and d represents the feature dimension. In the prediction model, the comprehensive feature vector Mapping to high-dimensional space through linear transformation, obtaining features in high-dimensional space , whose shape is (T, ) is the model dimension for input into the transformer;

[0082] is the eigenvector of the high-dimensional space Position encoding is performed to retain the temporal information of the sequence. The formula is: , , where pos is the position index, k is the dimension index, and the position encoding matrix The shape is (T, );

[0083] Adding positional encoding The eigenvector of , the feature vector After multi-head self-attention, specifically:

[0084] Linear changes generate queries, keys, and values, where , , , , and is the weight matrix, with the shape of ( , ), Represents dimension;

[0085] according to , , Calculate the attention weight Attention(Q,K,V)=softmax( )V;

[0086] MultiHead(Q,K,V)=Concat( , … ) ,in is the weight matrix for each head i, is the output weight matrix;

[0087] Compare the result MultiHead(Q,K,V) obtained by multi-head attention calculation with the input feature vector Perform residual connection and normalization Get Features , the operation formula is:

[0088] ;

[0089] Then the feature Through the feedforward neural network FFN, the output features of the feedforward neural network are combined with the input features Make residual connection and normalize Get Features , operation formula: .

[0090] MultiHead(Q,K,V)=Concat( , … ) ,in is the weight matrix for each head i, is the output weight matrix;

[0091] Compare the result MultiHead(Q,K,V) obtained by multi-head attention calculation with the input feature vector Perform residual connection and normalization Get Features , the operation formula is:

[0092] ;

[0093] Then the feature Through the feedforward neural network FFN, the output features of the feedforward neural network are combined with the input features Make residual connection and normalize Get Features , operation formula: .

[0094] The self-attention output features of the three modal features Fusion, and then the fused features Perform layer normalization Get fusion features , the formula is as follows:

[0095]

[0096] ;

[0097] in, Represents a splicing operation, and Represent the weight matrix and bias vector of the layer normalization respectively; finally, use Encoder pair fusion features After encoding, the influenza positivity rate is predicted. When predicting the influenza positivity rate, a fully connected layer is used, that is, the encoded features Input to the fully connected layer to predict the influenza positive rate Since the goal of the present invention is to predict the influenza positive rate, It is a one-dimensional time series, and each time step t corresponds to a predicted value of influenza positivity rate.

[0098] Through the above detailed operation process, the prediction model designed by the present invention can not only capture the key information in each modal data, but also better capture the relationship between each modality through the interaction layer and multi-level feature fusion, thereby improving the accuracy and robustness of influenza positivity rate prediction.

[0099] In the embodiment, corresponding prediction models are constructed for the synchronization group, the leading group, and the lagging group respectively. , , The influenza positivity rate was calculated and the prediction results were , , ;

[0100] Perform cross-validation on each prediction model, calculate its prediction performance (based on mean square error MSE), and assign weights inversely according to performance indicators to ensure that the weights meet According to the determined weights, the prediction results of each model are weighted averaged. The weighted average formula is as follows: + , the final weights are determined by cross validation. =0.425, =0.328, =0.247, so when the three groups are combined for prediction, the prediction formula is + .

[0101] Through the above detailed operation process, the prediction models constructed by the synchronization group, the leading group and the lagging group are combined to generate a comprehensive prediction result using the weighted average method. Determining weights by cross-validation or empirically assigning weights can balance the fused model between different data groups, thereby improving the accuracy and robustness of influenza positivity prediction.

[0102] like Figure 3As shown, the influenza trend prediction device 30 based on multimodal data provided by the embodiment includes a data acquisition module 31, a first feature extraction module 32, a second feature extraction module 33, a third feature extraction module 34, a feature initial fusion module 35, and an influenza prediction module 36, wherein the data acquisition module 31 is used to collect and pre-process the Baidu index data of influenza-related keywords, meteorological data, and population-related graph data; the first feature extraction module 32 is used to extract the feature vector of the Baidu index of each keyword, and fuse the feature vectors of the Baidu index of all keywords to obtain a comprehensive feature vector corresponding to the Baidu index modality; the second feature extraction module 33 is used to vectorize and quantize the wind direction data in the meteorological data. The value data is standardized and then spliced, and the feature vector of the splicing result is extracted as the comprehensive feature vector corresponding to the meteorological data modality; the third feature extraction module 34 is used to extract the feature vector of each type of graph in the graph data and fuse them to obtain the comprehensive feature vector corresponding to the graph data modality; the feature initial fusion module 35 is used to weight the comprehensive feature vector corresponding to each modality based on the attention weight calculated by the attention mechanism and then fuse them to obtain the total comprehensive feature vector; the influenza prediction module 36 is used to use a prediction model including a multi-head self-attention mechanism, a feedforward neural network, a multi-modal interaction layer, and multi-level feature fusion to perform deep feature fusion on the total comprehensive feature vector and predict the influenza positivity rate based on the fused features to obtain the influenza trend.

[0103] It should be noted that the influenza trend prediction device based on multimodal data provided in the above embodiment should be illustrated by the division of the above functional modules when performing influenza trend prediction. The above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the influenza trend prediction device based on multimodal data provided in the above embodiment and the influenza trend prediction construction method embodiment based on multimodal data belong to the same concept. The specific implementation process is detailed in the influenza trend prediction method embodiment based on multimodal data, which will not be repeated here.

[0104] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein executable code is stored in the memory, and when the one or more processors execute the executable code, the method for predicting influenza trends based on multimodal data is implemented, specifically including the following steps:

[0105] S1, collects and preprocesses Baidu index data of influenza-related keywords, meteorological data, and population-related graph data;

[0106] S2, extract the feature vector of Baidu Index of each keyword, and fuse the feature vectors of Baidu Index of all keywords to obtain the comprehensive feature vector corresponding to Baidu Index modality;

[0107] S3, vectorizing the wind direction data in the meteorological data and normalizing the numerical data, and then splicing them, and extracting the feature vector of the splicing result as the comprehensive feature vector corresponding to the meteorological data mode;

[0108] S4, extract the feature vector of each type of graph in the graph data and fuse them to obtain the comprehensive feature vector corresponding to the graph data modality;

[0109] S5, weighting the comprehensive feature vectors corresponding to each modality based on the attention weights calculated by the attention mechanism and then fusing them to obtain the total comprehensive feature vector;

[0110] S6, using a prediction model including a multi-head self-attention mechanism, a feedforward neural network, a multimodal interaction layer, and multi-level feature fusion, performs deep feature fusion on the total comprehensive feature vector and predicts the influenza positivity rate based on the fused features to obtain the influenza trend.

[0111] The computing device provided in the embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the influenza trend prediction method based on multimodal data described in S1-S6 above. Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0112] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned influenza trend prediction method based on multimodal data is implemented, which specifically includes the following steps:

[0113] S1, collects and preprocesses Baidu index data of influenza-related keywords, meteorological data, and population-related graph data;

[0114] S2, extract the feature vector of Baidu Index of each keyword, and fuse the feature vectors of Baidu Index of all keywords to obtain the comprehensive feature vector corresponding to Baidu Index modality;

[0115] S3, vectorizing the wind direction data in the meteorological data and normalizing the numerical data, and then splicing them, and extracting the feature vector of the splicing result as the comprehensive feature vector corresponding to the meteorological data mode;

[0116] S4, extract the feature vector of each type of graph in the graph data and fuse them to obtain the comprehensive feature vector corresponding to the graph data modality;

[0117] S5, weighting the comprehensive feature vectors corresponding to each modality based on the attention weights calculated by the attention mechanism and then fusing them to obtain the total comprehensive feature vector;

[0118] S6, using a prediction model including a multi-head self-attention mechanism, a feedforward neural network, a multimodal interaction layer, and multi-level feature fusion, performs deep feature fusion on the total comprehensive feature vector and predicts the influenza positivity rate based on the fused features to obtain the influenza trend.

[0119] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0120] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for predicting influenza trends based on multimodal data, characterized in that: The following steps are involved: Collect and pre-process Baidu index data of influenza-related keywords, meteorological data, and population-related graph data; Extract the feature vector of Baidu Index of each keyword, fuse the feature vectors of Baidu Index of all keywords to obtain the comprehensive feature vector corresponding to Baidu Index modality, and specifically use the LSTM model with attention mechanism to extract the feature vector of Baidu Index of each keyword, including: extract the first implicit state of each keyword through LSTM model, calculate the first attention score according to the first implicit state, calculate the first attention weight according to the first attention score, and perform weighted summation of the first implicit state according to the first attention weight to obtain the feature vector of Baidu Index of each keyword; The wind direction data in the meteorological data are vectorized and the numerical data are standardized before being spliced, and the feature vector of the splicing result is extracted as the comprehensive feature vector corresponding to the meteorological data modality. Specifically, a CNN-LSTM model with an attention mechanism is used to extract the feature vector of the splicing result, including: using CNN to extract the initial feature vector of the splicing result and input it into LSTM, extracting the second implicit state of the initial feature vector in LSTM, calculating the second attention score according to the second implicit state, calculating the second attention weight according to the second attention score, and performing weighted summation on the second implicit state according to the second attention weight to obtain the feature vector of the splicing result; Extract the feature vector of each type of graph in the graph data and fuse them to obtain the comprehensive feature vector corresponding to the graph data modality. Specifically, use the ResNet model to extract the feature vector of each type of graph in the graph data, including: the graph data contains three types of graphs, namely population heat map, traffic flow map, and population density map. Input each type of graph into the ResNet model, and calculate the feature vector of each type of graph through forward reasoning. The comprehensive feature vectors corresponding to each modality are weighted based on the attention weights calculated by the attention mechanism and then fused to obtain the total comprehensive feature vector; A prediction model including a multi-head self-attention mechanism, a feedforward neural network, a multimodal interaction layer, and multi-level feature fusion is used to perform deep feature fusion on the total comprehensive feature vector and predict the influenza positivity rate based on the fused features to obtain the influenza trend, including: mapping the total comprehensive feature vector to a high-dimensional space, and performing position encoding on the feature vector of the high-dimensional space, performing residual connection and normalization on the result of the feature vector with the position encoding after multi-head self-attention calculation to obtain the feature; performing residual connection and normalization on the output feature obtained by the feedforward neural network to obtain the feature; performing self-attention on each modal feature extracted from the feature through a multimodal interaction layer, and performing layer normalization after fusing the self-attention output features to obtain the fused feature; and using an encoder to encode the fused feature and then predict the influenza positivity rate.

2. The influenza trend prediction method based on multimodal data according to claim 1, characterized in that: The self-attention output features are fused by feature splicing, that is, the self-attention output features of the modal features of all modalities are spliced ​​to achieve fusion; After encoding the fused features, a fully connected layer is used to predict the influenza positivity rate.

3. The influenza trend prediction method based on multimodal data according to claim 1, characterized in that: It also includes: dividing the collected data into a synchronous group, a leading group, and a lagging group, The influenza positivity rate was predicted for each group, and the weighted sum of the predicted influenza positivity rates of the three groups was taken as the final influenza positivity rate.

4. A flu trend prediction device based on multimodal data, characterized in that: include: A data collection module, which is used to collect and pre-process Baidu index data of influenza-related keywords, meteorological data, and population-related graph data; A first feature extraction module is used to extract the feature vector of the Baidu Index of each keyword, and fuse the feature vectors of the Baidu Index of all keywords to obtain a comprehensive feature vector corresponding to the Baidu Index modality. Specifically, an LSTM model with an attention mechanism is used to extract the feature vector of the Baidu Index of each keyword, including: extracting the first implicit state of each keyword through the LSTM model, calculating the first attention score according to the first implicit state, calculating the first attention weight according to the first attention score, and performing weighted summation of the first implicit state according to the first attention weight to obtain the feature vector of the Baidu Index of each keyword; A second feature extraction module is used to vectorize the wind direction data in the meteorological data and normalize the numerical data before splicing, and extract the feature vector of the splicing result as the comprehensive feature vector corresponding to the meteorological data modality. Specifically, a CNN-LSTM model with an attention mechanism is used to extract the feature vector of the splicing result, including: using CNN to extract the initial feature vector of the splicing result and input it into the LSTM, extracting the second implicit state of the initial feature vector in the LSTM, calculating the second attention score according to the second implicit state, calculating the second attention weight according to the second attention score, and performing weighted summation on the second implicit state according to the second attention weight to obtain the feature vector of the splicing result; The third feature extraction module is used to extract the feature vector of each type of graph in the graph data and fuse them to obtain the comprehensive feature vector corresponding to the graph data modality. Specifically, the ResNet model is used to extract the feature vector of each type of graph in the graph data, including: the graph data includes three types of graphs, namely, population heat map, traffic flow map, and population density map. Each type of graph is input into the ResNet model, and the feature vector of each type of graph is obtained by forward reasoning calculation; The feature initial fusion module is used to weight the comprehensive feature vectors corresponding to each modality based on the attention weights calculated by the attention mechanism and then fuse them to obtain the total comprehensive feature vector; The influenza prediction module is used to use a prediction model including a multi-head self-attention mechanism, a feedforward neural network, a multimodal interaction layer, and multi-level feature fusion to perform deep feature fusion on the total comprehensive feature vector and predict the influenza positive rate based on the fused features to obtain the influenza trend, including: mapping the total comprehensive feature vector to a high-dimensional space, and performing position encoding on the feature vector of the high-dimensional space, adding the position-encoded feature vector, performing residual connection with the input feature vector after multi-head self-attention calculation and normalizing to obtain the feature; performing residual connection with the input output feature obtained by the feedforward neural network and normalizing to obtain the feature; performing self-attention on each modal feature extracted from the feature through a multimodal interaction layer, and performing layer normalization after fusing the self-attention output features to obtain the fused feature; and using an encoder to encode the fused feature and then predict the influenza positive rate.

5. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the influenza trend prediction method based on multimodal data described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the influenza trend prediction method based on multimodal data described in any one of claims 1 to 3 is implemented.

Images