A news dissemination method based on big data

Abstract

This invention discloses a big data-based news dissemination method, which relates to the field of artificial intelligence technology. The method includes collecting news data and preprocessing it to generate a structured news dataset; constructing a dynamic hypergraph to perform structural modeling on the structured news dataset, capturing the behavioral intensity associations between users and news and the semantic associations between news and topics through the connections between nodes and hyperedges; dynamically updating nodes and hyperedges using Apache Kafka and removing expired interaction records; performing multi-layer aggregation and nonlinear transformation on the node features of the updated dynamic hypergraph using a hypergraph neural network model; and analyzing the spatiotemporal dynamic characteristics of news dissemination feature vectors using a deep learning model to generate an optimized dissemination strategy. By constructing a dynamic hypergraph based on the structured news dataset, the present invention enhances the accuracy and adaptability of news dissemination feature extraction.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a news dissemination method based on big data. Background Art

[0002] With the increasing complexity of the Internet information ecosystem, news dissemination has evolved from traditional content distribution to personalized recommendations based on user behavior. However, existing technologies generally use collaborative filtering algorithms and matrix decomposition methods to construct user-news association models, predicting users' potential interests based on their historical behavior data. In recent years, hypergraph theory has been incorporated into the field of social network analysis. The high-order correlation characteristics of hypergraph theory can effectively model nonlinear relationships between multimodal data. Some studies have attempted to apply static hypergraphs to news topic clustering and user group segmentation. In terms of real-time data processing, distributed messaging systems have also realized the dynamic collection and transmission of data streams, providing news dissemination systems with incremental update capabilities with millisecond-level delays.

[0003] The shortcomings of current static hypergraph technology in the field of news dissemination are mainly reflected in the lack of timeliness of dynamic interactions: existing technologies mostly adopt a batch processing mechanism with a fixed time window, which leads to delayed updates of user behavior intensity correlations and cannot synchronize the semantic drift rate of news topics; typical methods adopted by existing technologies, such as Hypergraph Neural Network, mostly rely on offline full data to rebuild the hypergraph topology, and although the news feature extraction algorithm based on TF-IDF can capture static semantics, it is difficult to eliminate the interference of expired interaction records on the current user feature vector. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a news dissemination method based on big data to solve the shortcomings of current static hypergraph technology in the field of news dissemination.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a news dissemination method based on big data, which includes collecting news data and preprocessing it to generate a structured news data set, wherein the structured news data set includes hashed user identifiers, user feature vectors, interaction records, news content feature vectors, and topic feature vectors; constructing a dynamic hypergraph to perform structured modeling on the structured news data set, capturing the behavioral intensity association between users and news and the semantic association between news and topics through the connection between nodes and hyperedges, and using Apache Kafka to dynamically update nodes and hyperedges and remove expired interaction records; using a hypergraph neural network model to perform multi-layer aggregation and nonlinear transformation on the node features of the updated dynamic hypergraph to generate a news dissemination feature vector; and analyzing the spatiotemporal dynamic characteristics of the news dissemination feature vector through a deep learning model to generate an optimized dissemination strategy.

[0008] As a preferred solution of the big data-based news dissemination method described in the present invention, the construction of a dynamic hypergraph to perform structured modeling of a structured news data set refers to reading structured news data from the structured news data set, defining user nodes, news nodes and topic nodes, and creating user-news hyperedges and news-topic hyperedges to form a dynamic hypergraph.

[0009] As a preferred solution of the big data-based news dissemination method described in the present invention, the method of capturing the behavioral strength association between users and news refers to extracting interaction records from structured news data, creating user-news hyperedges, connecting user nodes and news nodes, setting hyperedge weights based on interaction types, and if there are multiple user-news interaction records, accumulating the interaction weights through aggregation operations to update the weight of a single user-news hyperedge;

[0010] The semantic association between news and topics refers to obtaining news content feature vectors and topic feature vectors from structured news data, using the fully connected layer of the BERT model to reduce the dimension of the news content feature vector, and calculating the cosine similarity between the reduced news content feature vector and the topic feature vector through Python's built-in calculation function. If the cosine similarity is greater than the similarity threshold, a news-topic hyperedge is created to connect the news node and the topic node. The cosine similarity is used as the news-topic hyperedge weight to reflect the semantic association between the news and the topic.

[0011] As a preferred solution of the big data-based news dissemination method described in the present invention, the dynamic updating of nodes and hyperedges using Apache Kafka refers to obtaining new structured news data through Apache Kafka, updating the structured news dataset, adding user nodes, news nodes and topic nodes, updating the user-news hyperedge weight based on the new interaction records, and updating the news-topic hyperedge based on the cosine similarity calculated based on the news content feature vector and the topic feature vector to form an updated dynamic hypergraph.

[0012] As a preferred solution of the big data-based news dissemination method of the present invention, wherein: the node features of the updated dynamic hypergraph are aggregated in multiple layers using the hypergraph neural network model, and the specific steps are:

[0013] Loading the user feature vector of the user node, the news content feature vector of the news node, and the topic feature vector of the topic node from the dynamic hypergraph;

[0014] The fully connected layer of the BERT model is used to reduce the dimensionality of user feature vectors and news content feature vectors;

[0015] The user feature vector and news content feature vector after dimensionality reduction are weighted and fused with the topic feature vector without dimensionality reduction to form a unified node feature vector;

[0016] Based on the dynamic hypergraph, a hypergraph neural network model is constructed and multiple convolutional layers are set;

[0017] The first layer forms the initial aggregated feature vector by weightedly aggregating the unified node feature vectors of neighboring nodes through user-news hyperedges and news-topic hyperedges;

[0018] The second layer forms an intermediate aggregated feature vector by weightedly aggregating the initial aggregated feature vectors of the neighborhood nodes through the user-news hyperedge and the news-topic hyperedge;

[0019] The third layer forms the final aggregated feature vector by weightedly aggregating the intermediate aggregated feature vectors of the neighborhood nodes through the user-news hyperedge and the news-topic hyperedge.

[0020] As a preferred solution of the news dissemination method based on big data of the present invention, the nonlinear transformation comprises the following specific steps:

[0021] Apply the ReLU activation function to the initial aggregated feature vector generated by the first layer of convolution for nonlinear mapping to generate the initial enhanced feature vector;

[0022] Apply the ReLU activation function to the intermediate aggregated feature vector generated by the second layer of convolution for nonlinear mapping to generate an intermediate enhanced feature vector;

[0023] The final aggregated feature vector generated by the third layer of convolution is subjected to nonlinear mapping using the ReLU activation function to generate a news propagation feature vector.

[0024] As a preferred solution of the news dissemination method based on big data described in the present invention, wherein: the spatiotemporal dynamic characteristics of the news dissemination feature vector are analyzed by a deep learning model, and the specific steps are:

[0025] Get news propagation feature vectors from Apache Kafka;

[0026] The news propagation feature vector is processed through the attention mechanism layer of the deep learning model to generate attention weights;

[0027] Adjust the news propagation feature vector based on the attention weight to form a weighted news propagation feature vector;

[0028] The weighted news propagation feature vector is processed by time stepping through the LSTM layer of the deep learning model to capture the propagation dynamic characteristics of the weighted news propagation feature vector;

[0029] The gating mechanism of the LSTM layer is used to update the hidden state of the LSTM layer to generate the spatiotemporal dynamic characteristic representation of the news propagation feature vector.

[0030] As a preferred solution of the big data-based news dissemination method described in the present invention, the optimized dissemination strategy includes news identification, dissemination priority, dissemination channel and dissemination frequency.

[0031] In a second aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the big data-based news dissemination method as described in the first aspect of the present invention.

[0032] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the big data-based news dissemination method as described in the first aspect of the present invention.

[0033] The beneficial effects of the present invention are as follows: the present invention constructs a dynamic hypergraph based on a structured news dataset, defines user nodes, news nodes and topic nodes, and uses interaction records and cosine similarity to create user-news hyperedges and news-topic hyperedges. At the same time, Apache Kafka dynamically updates nodes and hyperedges and removes expired interaction records, thereby realizing dynamic structured modeling of user-news behavior intensity association and news-topic semantic association; capturing multidimensional complex relationships in the form of a hypergraph and maintaining real-time performance is suitable for analyzing dynamic changes in user behavior and news semantics, ultimately achieving the effect of improving the comprehensiveness and timeliness of relationship modeling, and enhancing the accuracy and adaptability of news dissemination feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 The flowchart of the news dissemination method based on big data.

[0036] Figure 2 This is a flowchart of the big data news dissemination architecture.

[0037] Figure 3 Schematic diagram for modeling dynamic hypergraphs.

[0038] Figure 4 Flowchart of feature aggregation in hypergraph neural network. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0041] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0042] Reference Figures 1 to 4 This embodiment provides a news dissemination method based on big data, comprising the following steps:

[0043] S1. Collect news data and preprocess it to generate a structured news dataset.

[0044] Furthermore, news data includes news text content, user behavior data, and social media interaction data, as follows:

[0045] Obtain news text content (news title, news text, news keywords and news release time) from news websites;

[0046] Obtain social media interaction data (post content, forwarding history, comments, and hashtags) from social media platforms;

[0047] Obtain user behavior data (user ID, news ID, interaction type (click, like, comment, share), timestamp, and browsing time) from news applications;

[0048] We developed a crawler using the Scrapy framework and customized collection rules for each type of news data (news text content, user behavior data, and social media interaction data) to ensure real-time news data. The collection rule was set to a 1-second frequency, which balanced real-time performance with server load (referring to the upper limit of requests per second set by social media platform APIs).

[0049] Configure three news storage topics (news_content, social_interaction, and user_behavior) through Apache Kafka (distributed message queue);

[0050] Among them, the news_content topic stores news text content;

[0051] The social_interaction topic stores social media interaction data;

[0052] The user_behavior topic stores user behavior data;

[0053] Scrapy's crawler writes the collected news data into three news storage topics in Apache Kafka. Apache Kafka consumers read the news data from the three news storage topics and store it in the MongoDB distributed database. Apache Kafka is a distributed message queue used for efficient data transmission and is divided into producer and consumer roles. Scrapy's crawler acts as a producer and writes the collected news data into the three news storage topics in Apache Kafka. Consumers are preconfigured applications (for example, Apache Kafka clients implemented in Python) that are responsible for reading news data from the three news storage topics and storing it in the MongoDB distributed database.

[0054] Preprocessing tasks are performed on news text content, user behavior data, and social media interaction data, as follows:

[0055] Use Python's hashlib hash calculation library to calculate the MD5 hash value of the news text content, compare the hash value of the current news text content with the stored hash value set (stored in the MongoDB distributed database), remove duplicate news text content records, and set the duplication rate to 5% (based on the reprint rate of news aggregation platforms, such as news websites);

[0056] Encode news titles, news text, post content, and comments in UTF-8 to ensure consistent character formats and compatibility with multilingual characters;

[0057] Set default values ​​for empty fields in news text content, social media interaction data, and user behavior data: for example, fill in "unknown" for news keywords, "0" for comments, and "0.00" for browsing time.

[0058] Use Python's re library to remove HTML tags from news titles, news text, post content, and comments, filter special characters (such as emoticons), and generate plain text data;

[0059] Using the TF-IDF algorithm from the scikit-learn machine learning library, we extracted the top 10 keywords with high TF-IDF values ​​from the news headlines and news texts (the top 10 are sorted in descending order of TF-IDF values ​​based on information retrieval standards) and generated a news content feature vector (300 dimensions) as follows:

[0060] Get the cleaned news title and news body (from text cleaning) from the MongoDB distributed database;

[0061] Use scikit-learn's TfidfVectorizer (TF-IDF vectorization tool) to merge the news title and news body into a document, calculate the TF-IDF value, and select the top 10 keywords with the highest TF-IDF value;

[0062] The keywords are mapped into a 300-dimensional sparse vector. The first 10 dimensions are filled with the TF-IDF values ​​of the keywords and the remaining 290 dimensions are filled with zeros to generate the news content feature vector.

[0063] The news title and news body are encoded using the pre-trained BERT (language) model to form a news content feature vector (300 dimensions), as follows:

[0064] Get news titles and news text from the MongoDB distributed database;

[0065] Merge the news title and news body into a complete text by string concatenation to capture semantic information;

[0066] Use the built-in tokenizer of the BERT model to tokenize the complete text (for example, "AI," "technology," and "breakthrough") and generate a token ID sequence and attention mask.

[0067] The token ID sequence and attention mask are fed into the encoder of the BERT model, and a 768-dimensional CLS vector is generated through forward propagation to capture the overall semantics of the news title and news body.

[0068] Through the fully connected layer, linear transformation is applied to the 768-dimensional CLS vector to generate the news content feature vector.

[0069] The news content feature vectors generated by the TF-IDF algorithm and the BERT model are merged into the final news content feature vector through weighted averaging (weights of 0.7:0.3).

[0070] Collect the user's recent click records (e.g., 100 times) and use a pre-trained text classification model (based on the BERT model, classified into 10 categories of user interest distribution, such as technology, sports, and entertainment) to generate a user feature vector (128 dimensions). The details are as follows:

[0071] Get the news title and text of the click record, use the BERT model encoder to generate a 768-dimensional CLS vector;

[0072] The 768-dimensional CLS vector is input into the fully connected layer and a 10-dimensional probability vector is output through softmax.

[0073] Use NumPy (a numerical computing library) to average the 10-dimensional probability vectors of 100 clicks to generate a 10-dimensional user interest distribution;

[0074] The user interest distribution is linearly transformed through the fully connected layer to generate the user feature vector.

[0075] Use the Word2Vec (word embedding) model to segment the topic tags of social media interaction data and generate topic feature vectors (64 dimensions) as follows:

[0076] Get hashtags from social media interaction data from the MongoDB distributed database;

[0077] Use the Jieba word segmentation tool to segment the topic tags (for example, "AI technology" is divided into "AI" and "technology");

[0078] The input layer defines the word segmentation of the topic tag as a one-hot vector;

[0079] The hidden layer projects the one-hot vector into a 64-dimensional space through the embedding matrix to generate a 64-dimensional hidden layer vector;

[0080] The output layer uses the context words to predict the context word probability distribution of the 64-dimensional hidden layer vector, optimizes the embedding matrix through backpropagation, and generates a 64-dimensional word embedding vector;

[0081] Load the pre-trained Word2Vec (word embedding) model;

[0082] If the topic tag contains a single word, load the pre-trained Word2Vec model and use the 64-dimensional word embedding vector as the topic feature vector;

[0083] If the topic tag contains multiple words (such as "AI" and "technology"), load the pre-trained Word2Vec model, take the average of the 64-dimensional word embedding vectors, and generate the topic feature vector.

[0084] Apply differential privacy algorithms to user behavior data to protect the privacy of user behavior data, as follows:

[0085] Obtain user behavior data from the MongoDB distributed database;

[0086] Normalize the click records of user behavior data (min-max normalization) to generate standardized user behavior data;

[0087] Use Python's diffprivlib differential privacy library to generate Laplace noise (privacy parameter ε = 0.1, which controls the privacy protection strength of user behavior data click records. Smaller values ​​mean stronger protection. Sensitivity is 1.0. ε and sensitivity are set based on the differential privacy standard).

[0088] Add Laplace noise to the click records of normalized user behavior data (for example, the normalized click count of 0.5 becomes 0.5+noise);

[0089] Use SHA-256 hashing to anonymize the user ID of user behavior data and generate an irreversible hashed user ID;

[0090] Output anonymized user behavior data (including hashed user IDs and noised click records) and store it in the MongoDB distributed database.

[0091] Use PySpark (a distributed computing framework) to load news text content, social media interaction data, and user behavior data from the MongoDB distributed database to generate a PySpark DataFrame. Use PySpark to parallelize preprocessing tasks (SHA-256 hashing, BERT classification, TF-IDF, and Word2Vec inference) to aggregate and generate a structured news dataset (including hashed user identifiers, user feature vectors, interaction records, news content feature vectors, and topic feature vectors) and write it to the MongoDB distributed database.

[0092] Among them, PySpark DataFrame is a distributed data structure in PySpark, which contains hashed user ID, user feature vector, interaction record (hashed user ID, news ID, interaction type: click, like, comment, share, timestamp), news content feature vector and topic feature vector.

[0093] S2. Build a dynamic hypergraph to perform structured modeling on the structured news dataset. Through the connection between nodes and hyperedges, capture the behavioral intensity association between users and news and the semantic association between news and topics. Use Apache Kafka to dynamically update nodes and hyperedges and remove expired interaction records.

[0094] Furthermore, we read the structured news dataset from the MongoDB distributed database;

[0095] Construct a dynamic hypergraph based on structured news dataset;

[0096] Define the node types of the dynamic hypergraph as user nodes, news nodes, and topic nodes;

[0097] Associate user nodes with hashed user IDs in the structured news dataset;

[0098] Associate news nodes with news identifiers in the structured news dataset;

[0099] Associating topic nodes with topic tags in structured news datasets;

[0100] Use PySpark join to match node attributes, matching hashed user IDs with user feature vectors, news IDs with news content feature vectors, and topic tags with topic feature vectors. Use the matched feature vectors (user feature vector, news content feature vector, and topic feature vector) as node attributes.

[0101] Initialize the graph object in NetworkX (graph analysis library), write user nodes, news nodes, and topic nodes into the graph object in NetworkX, and build the node set of the hypergraph;

[0102] For each interaction record extracted from the PySpark DataFrame, a user-news hyperedge is created in the NetworkX graph object, connecting the user node (hashed user ID) with the news node (news ID). The hyperedge weight is set according to the interaction type (0.5 for clicks, 1.0 for likes, 1.5 for comments, and 2.0 for shares. The hyperedge weight is predefined and reflects the interaction strength). If the same user-news group already has a hyperedge or there are multiple new interaction records, the PySpark groupBy operation is used to group the user and news, and the sum operation is used to add the weight values ​​of the multiple interactions (the weight value refers to the numerical value mapped by the interaction type for each interaction (for example, 1 for likes, 2 for comments, and 3 for shares) to update the hyperedge weight, thereby generating or updating a single user-news hyperedge. The hyperedge weight represents the total interaction strength of the user on the news.

[0103] Through the fully connected layer of the BERT model, the original 300-dimensional news content feature vector is reduced to a 64-dimensional news content feature vector;

[0104] Using Python's common mathematical operations (standard mathematical operations built into Python, such as addition, multiplication, division, and square root), we calculated the cosine similarity between the reduced news content feature vectors and the topic feature vectors generated by the Word2Vec model, generating the semantic similarity for each news-topic pair.

[0105] If the cosine similarity between news and topics is greater than 0.7 (a similarity threshold commonly used in information retrieval, indicating high semantic relevance), a news-topic hyperedge is created in the NetworkX graph object, connecting the news node and the topic node. The cosine similarity is used as the weight of the news-topic hyperedge to indicate the strength of the semantic association.

[0106] Store user nodes, news nodes, topic nodes, user-news hyperedges, and news-topic hyperedges into the initialized NetworkX graph object to form a dynamic hypergraph.

[0107] Use Apache Kafka consumers to subscribe to news_content, social_interaction, and user_behavior topics to obtain new news data;

[0108] Repeat the preprocessing task to generate new hashed user IDs, user feature vectors, news content feature vectors, and topic feature vectors, and update the structured news dataset.

[0109] If a new hashed user ID is added to the updated structured news dataset, a user node is added to the graph object in NetworkX, with the new hashed user ID as the unique identifier and bound to the generated user feature vector;

[0110] If a new news identifier is added to the updated structured news dataset, a news node is added to the graph object in NetworkX, with the new news identifier as the unique identifier and bound to the generated news content feature vector;

[0111] If a new topic tag is added to the updated structured news dataset, a topic node is added to the graph object in NetworkX, with the new topic tag as the unique identifier and bound to the generated topic feature vector;

[0112] Based on the new interaction records in the updated structured news dataset, the hyperedge weight is updated to generate or update a single user-news hyperedge;

[0113] For the new news text content in the updated structured news dataset, the fully connected layer of the BERT model is used to reduce the original 300-dimensional news content feature vector to a 64-dimensional news content feature vector. The cosine similarity between the reduced news content feature vector and the topic feature vector generated by the Word2Vec model is calculated. If the cosine similarity is greater than 0.7 (the threshold in the information retrieval field), a news-to-topic hyperedge is added to the graph object in NetworkX, connecting the news node and the topic node, and the cosine similarity is used as the news-to-topic hyperedge weight.

[0114] Use PySpark to incrementally process the updated structured news dataset, repeatedly adding new nodes and new hyperedges to update the graph object in NetworkX.

[0115] Set a 7-day time window (a common cycle that balances news timeliness and data volume) and remove interaction records with timestamps older than 7 days from the interaction log;

[0116] Based on the removed interaction records, delete the corresponding user-news hyperedge from the graph object in NetworkX;

[0117] If a news node or topic node is no longer connected to other nodes because all user-news hyperedges and news-topic hyperedges have been removed, it becomes an isolated node and is deleted from the NetworkX graph object. An isolated node is a news node or topic node that is no longer connected to user nodes, news nodes, or topic nodes in the dynamic hypergraph through user-news hyperedges and news-topic hyperedges.

[0118] Use PySpark's scheduled tasks (e.g., daily execution) to remove interaction records older than 7 days, the corresponding user-news hyperedges, and isolated nodes, and regularly update the graph object in NetworkX.

[0119] Export the graph object in NetworkX to JSON format, including node data and hyperedge data, and store it in the MongoDB distributed database as a graph structure data collection;

[0120] Use NetworkX (graph analysis library) to count the number of nodes and hyperedges in the NetworkX graph object, perform graph analysis using the NetworkX graph object, calculate the average hyperedge weight, and generate metadata about the size and connection strength of the hypergraph (number of nodes to number of hyperedges).

[0121] The generated metadata is stored in the metadata collection of the MongoDB distributed database for monitoring the size and connection quality of the dynamic hypergraph.

[0122] S3. Use the hypergraph neural network model to perform multi-layer aggregation and nonlinear transformation on the node features of the updated dynamic hypergraph to generate a news propagation feature vector.

[0123] Furthermore, a graph structure data set of a dynamic hypergraph is read from a MongoDB distributed database;

[0124] Load the pre-trained hypergraph neural network model and set up three convolutional layers. Each convolutional layer aggregates node and neighborhood information through hyperedge weighting.

[0125] The input of the hypergraph neural network model is the node feature vector, the hyperedge connection relationship and the hyperedge weight as the convolutional layer aggregation weight;

[0126] The fully connected layer of the BERT model takes the user feature vector as input, and after matrix multiplication and ReLU activation, it is reduced to a 64-dimensional user feature vector, which is aligned with the topic feature vector dimension of the topic node;

[0127] The fully connected layer of the BERT model takes the news content feature vector as input, and after matrix multiplication and ReLU activation, it is reduced to a 64-dimensional news content feature vector, which is aligned with the topic feature vector of the topic node.

[0128] The topic feature vector of the unreduced topic node, the user feature vector after dimensionality reduction, and the news content feature vector are weighted and fused to form a unified node feature vector, which serves as the input of the hypergraph neural network model. The topic feature vector of the topic node remains unchanged and has the same dimension as the user feature vector and news content feature vector after dimensionality reduction.

[0129] The first convolutional layer finds neighboring nodes based on the user-news hyperedge and news-topic hyperedge and weightedly aggregates the uniform-dimensional node feature vectors of the neighboring nodes (neighboring nodes are also nodes in the dynamic hypergraph and therefore also have the uniform node feature vectors generated above). This generates an initial aggregated feature vector, which is then transformed nonlinearly using ReLU activation to generate an initial enhanced feature vector to enhance the expressive power of the initial aggregated feature vector. Dropout is then applied to prevent overfitting of the hypergraph neural network model, generating an intermediate feature vector.

[0130] The second convolutional layer finds neighboring nodes based on the user-news hyperedge and news-topic hyperedge, and weights and aggregates the 64-dimensional intermediate feature vectors of the neighboring nodes to generate an intermediate aggregated feature vector. ReLU activation is applied to the intermediate aggregated feature vector for nonlinear transformation to generate an intermediate enhanced feature vector, which enhances the expressiveness of the second-layer aggregated feature vector. Dropout is then applied to prevent overfitting of the hypergraph neural network model, generating an updated intermediate feature vector.

[0131] The third convolutional layer finds neighboring nodes based on the user-news hyperedge and news-topic hyperedge, and weightedly aggregates the updated 64-dimensional intermediate feature vectors of the neighboring nodes to generate the third-layer aggregated feature vector. ReLU activation is then applied to the third-layer aggregated feature vector to perform a nonlinear transformation to enhance its expressiveness. No dropout processing is performed to generate the final news propagation feature vector (i.e., the final aggregated feature vector).

[0132] Add a time dimension to the news propagation feature vector by using the timestamp of the interaction record to generate a news propagation feature vector with time information;

[0133] The news propagation feature vectors with time information are stored in the MongoDB distributed database as a news propagation feature vector set, which is updated every hour, consistent with the update frequency of the dynamic hypergraph.

[0134] Use Apache Kafka consumers to subscribe to the news_content, social_interaction, and user_behavior topics to obtain new news data (based on news text content, social media interaction data, and user behavior data, generate new user nodes, news nodes, topic nodes, user-news hyperedges, and news-topic hyperedges);

[0135] If the dynamic hypergraph is updated due to new news data, the steps of updating the news propagation feature vector with time information are repeated;

[0136] If the dynamic hypergraph is removed due to expired interaction records, the steps of deleting the news nodes, topic nodes, and user-news and news-topic hyperedges of the dynamic hypergraph are repeated to remove the corresponding news propagation feature vectors with time information, maintaining the data consistency between the news propagation feature vector set and the dynamic hypergraph;

[0137] Use a PySpark scheduled task (executed daily) to check the news propagation feature vector set in the MongoDB distributed database (including news propagation feature vectors with time information) and remove news propagation feature vectors with timestamps older than 7 days to maintain consistency with the timeliness of the dynamic hypergraph.

[0138] Use Python's general calculation methods to calculate the size of the news dissemination feature vector set (the number of news dissemination feature vectors of news nodes) and the average norm of the news dissemination feature vectors (calculate the Euclidean norm of the news dissemination feature vectors of news nodes and then take the average), generate metadata for the news dissemination feature vectors, and store them in the metadata collection of the MongoDB distributed database;

[0139] The output is a news dissemination feature vector with time information, which is stored in the MongoDB distributed database and transmitted to the next step via Apache Kafka to generate an optimized dissemination strategy.

[0140] It should be noted that the hypergraph neural network model is an existing model and is well-known in this field. The basic architecture is implemented through multi-layer convolution. Each layer of convolution weights and aggregates node feature vectors based on the hyperedge connection relationship to generate a more expressive feature representation. It is widely used in the field of communication analysis.

[0141] S4. Analyze the spatiotemporal dynamic characteristics of news dissemination feature vectors through deep learning models to generate optimized dissemination strategies.

[0142] Furthermore, we receive news propagation feature vectors with time information from Apache Kafka;

[0143] The news propagation feature vector with time information is loaded into a pre-trained deep learning model, which includes an attention mechanism layer, an LSTM layer, and a fully connected layer, to analyze the spatiotemporal dynamic characteristics of the news propagation feature vector;

[0144] The attention mechanism layer inputs the news dissemination feature vector into the fully connected layer and generates an attention score through matrix multiplication. The attention score is then normalized using the Softmax function to generate an attention weight (indicating the relative importance of each dimension in the news dissemination feature vector to the dissemination strategy, ranging from 0 to 1 and summing to 1). Next, the news dissemination feature vector is element-wise multiplied by the attention weight to generate a weighted news dissemination feature vector.

[0145] The LSTM layer is set to a hidden dimension of 128 and a time step of 24 (representing the spread dynamics within 24 hours, based on the spread period). Then, the weighted news spread feature vector is used as input, and the time step is processed using the timestamp (a time series within 24 hours). The LSTM gating mechanism (input gate, forget gate, output gate) is used to update the hidden state of the LSTM layer to generate a time series feature vector (with a dimension of 128, representing the spatiotemporal dynamic characteristics of news spread within 24 hours).

[0146] The fully connected layer receives the time series feature vector and maps it to a three-dimensional propagation strategy vector (representing propagation priority, propagation channel, and propagation frequency, a total of three dimensions) through matrix multiplication. Then, the three-dimensional propagation strategy vector is normalized (range 0 to 1, sum to 1) through the Softmax function to generate the final propagation strategy vector (representing the normalized propagation strategy).

[0147] The communication strategy vector is associated with the news identifier through index matching to generate an optimized communication strategy in the format of (news identifier, communication priority, communication channel and communication frequency) to guide news communication practice.

[0148] It should be noted that the attention mechanism layer involves the work of the fully connected layer because the fully connected layer generates attention scores through matrix mapping to measure the importance of the dimensions of the news dissemination feature vector. However, the core work of the attention mechanism layer is not limited to the fully connected layer, but also includes normalization processing and weighted calculation, which together achieve weighted enhancement of the dimensions of the news dissemination feature vector.

[0149] This embodiment also provides a computer device suitable for the case of a news dissemination method based on big data, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the news dissemination method based on big data proposed in the above embodiment.

[0150] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0151] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the news dissemination method based on big data proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0152] In summary, the present invention constructs a dynamic hypergraph based on a structured news dataset, defines user nodes, news nodes, and topic nodes, and uses interaction records and cosine similarity to create user-news hyperedges and news-topic hyperedges. At the same time, Apache Kafka dynamically updates nodes and hyperedges and removes expired interaction records, thereby realizing dynamic structured modeling of user-news behavior intensity association and news-topic semantic association; capturing multidimensional complex relationships in the form of a hypergraph and maintaining real-time performance is suitable for analyzing dynamic changes in user behavior and news semantics, ultimately achieving the effect of improving the comprehensiveness and timeliness of relationship modeling, and enhancing the accuracy and adaptability of news dissemination feature extraction.

[0153] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A news dissemination method based on big data, characterized by: include, Collecting news data and preprocessing it to generate a structured news data set, wherein the structured news data set includes hashed user identifiers, user feature vectors, interaction records, news content feature vectors, and topic feature vectors; We construct a dynamic hypergraph to model the structured news dataset. By connecting nodes and hyperedges, we capture the behavioral intensity of user-news interactions and the semantic associations between news and topics. We leverage Apache Kafka to dynamically update nodes and hyperedges and remove expired interaction records. The hypergraph neural network model is used to perform multi-layer aggregation and nonlinear transformation on the node features of the updated dynamic hypergraph to generate the news propagation feature vector; Analyze the spatiotemporal dynamic characteristics of news dissemination feature vectors through deep learning models to generate optimized dissemination strategies; The construction of a dynamic hypergraph to perform structured modeling on a structured news dataset refers to reading structured news data from the structured news dataset, defining user nodes, news nodes, and topic nodes, and creating user-news hyperedges and news-topic hyperedges to form a dynamic hypergraph; The specific steps of analyzing the spatiotemporal dynamic characteristics of news propagation feature vectors through deep learning models are as follows: Get news propagation feature vectors with time information from Apache Kafka; The news propagation feature vector is processed through the attention mechanism layer of the deep learning model to generate attention weights; Adjust the news propagation feature vector based on the attention weight to form a weighted news propagation feature vector; The LSTM layer of the deep learning model is used to perform time step processing on the weighted news propagation feature vector to capture the propagation dynamic characteristics of the weighted news propagation feature vector; The gating mechanism of the LSTM layer is used to update the hidden state of the LSTM layer to generate the spatiotemporal dynamic characteristic representation of the news propagation feature vector.

2. The big data-based news dissemination method according to claim 1, characterized in that: Capturing the behavioral strength association between users and news refers to extracting interaction records from structured news data, creating user-news hyperedges, connecting user nodes and news nodes, setting hyperedge weights based on interaction types, and if there are multiple user-news interaction records, accumulating the interaction weights through aggregation operations to update the weight of a single user-news hyperedge; The semantic association between news and topics refers to obtaining news content feature vectors and topic feature vectors from structured news data, using the fully connected layer of the BERT model to reduce the dimension of the news content feature vector, and calculating the cosine similarity between the reduced news content feature vector and the topic feature vector through Python's built-in calculation function. If the cosine similarity is greater than the similarity threshold, a news-topic hyperedge is created to connect the news node and the topic node. The cosine similarity is used as the news-topic hyperedge weight to reflect the semantic association between the news and the topic.

3. The big data-based news dissemination method according to claim 1, wherein: The dynamic updating of nodes and hyperedges using Apache Kafka refers to obtaining new structured news data through Apache Kafka, updating the structured news dataset, adding user nodes, news nodes, and topic nodes, updating the user-news hyperedge weight based on new interaction records, and updating the news-topic hyperedge based on the cosine similarity calculated based on the news content feature vector and the topic feature vector to form an updated dynamic hypergraph.

4. The big data-based news dissemination method according to claim 1, wherein: The hypergraph neural network model is used to perform multi-layer aggregation on the node features of the updated dynamic hypergraph. The specific steps are: Loading the user feature vector of the user node, the news content feature vector of the news node, and the topic feature vector of the topic node from the dynamic hypergraph; The fully connected layer of the BERT model is used to reduce the dimensionality of user feature vectors and news content feature vectors; The user feature vector and news content feature vector after dimensionality reduction are weighted and fused with the topic feature vector without dimensionality reduction to form a unified node feature vector; Based on the dynamic hypergraph, a hypergraph neural network model is constructed and multiple convolutional layers are set; The first layer forms the initial aggregated feature vector by weightedly aggregating the unified node feature vectors of neighboring nodes through user-news hyperedges and news-topic hyperedges; The second layer forms an intermediate aggregated feature vector by weightedly aggregating the initial aggregated feature vectors of the neighborhood nodes through the user-news hyperedge and the news-topic hyperedge; The third layer forms the final aggregated feature vector by weightedly aggregating the intermediate aggregated feature vectors of the neighborhood nodes through the user-news hyperedge and the news-topic hyperedge.

5. The big data-based news dissemination method according to claim 1, wherein: The nonlinear transformation specifically comprises the following steps: Apply the ReLU activation function to the initial aggregated feature vector generated by the first layer of convolution for nonlinear mapping to generate the initial enhanced feature vector; Apply the ReLU activation function to the intermediate aggregated feature vector generated by the second layer of convolution for nonlinear mapping to generate an intermediate enhanced feature vector; The final aggregated feature vector generated by the third layer of convolution is subjected to nonlinear mapping using the ReLU activation function to generate a news propagation feature vector.

6. The big data-based news dissemination method according to claim 1, wherein: The optimized communication strategy includes news identification, communication priority, communication channels and communication frequency.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the big data-based news dissemination method described in any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the big data-based news dissemination method described in any one of claims 1 to 6 are implemented.

Images