News information publishing platform based on big data intelligent mining

Through intelligent big data mining technology, combined with multi-source data collection, in-depth semantic analysis and personalized push, the shortcomings of traditional news information publishing systems in data collection and recommendation effects have been solved, and more accurate and personalized news recommendations have been achieved, improving The social value of user experience and news.

CN118964739AInactive Publication Date: 2024-11-15GUANGZHOU YANGYUE BOZHONG INFORMATION TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411064308.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The traditional news and information publishing system has room for improvement in the breadth and depth of data collection, the accuracy of content analysis, and the effect of personalized recommendations, which is difficult to meet the timeliness, accuracy and personalized needs of users.

Method used

A news information publishing platform based on intelligent mining of big data has been designed, including multi-source data acquisition module, deep semantic analysis module and personalized push module. Through multi-source data acquisition, deep semantic analysis and personalized push, it provides accurate news. recommend.

Benefits of technology

It improves the breadth and depth of news data collection, provides more accurate personalized recommendation effects, enhances the communication influence and social value of news, and improves the accuracy of user experience and recommendations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964739A_ABST
    Figure CN118964739A_ABST
Patent Text Reader

Abstract

The invention relates to a news information publishing platform based on big data intelligent mining, and the platform comprises a multi-source data collection module which is used for obtaining related news data and user behavior data of various data sources, integrating heterogeneous data of different channels, and generating a unified news data set and a user behavior data set; the deep semantic analysis module is used for performing deep semantic analysis on the unified news data set to generate a deep semantic analysis result, including subject classification, emotional tendency judgment, key character time extraction and semantic similarity calculation; and the personalized pushing module is used for extracting features according to the user behavior data set to construct a user portrait, monitoring new behavior data of the user in real time, updating the user portrait, providing a highly personalized news recommendation list in combination with a deep semantic analysis result, establishing a user feedback channel and adjusting a recommendation strategy. According to the platform, the news data acquisition breadth and depth can be improved, and a more accurate personalized recommendation effect of news contents is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention mainly relates to the field of information technology, and in particular to a news information publishing platform based on big data intelligent mining. Background Art

[0002] With the popularization of Internet technology and mobile devices, the speed of information dissemination has increased exponentially, but traditional news information publishing systems are facing many challenges. On the one hand, information overload makes it difficult for users to filter out truly valuable news; on the other hand, the timeliness, accuracy and personalization of news are difficult to meet the growing needs of users. Existing news publishing systems have room for improvement in terms of the breadth and depth of data collection, the accuracy of content analysis, and the effectiveness of personalized recommendations. Summary of the invention

[0003] To solve the above problems, the present invention provides a news information publishing platform based on big data intelligent mining, which can improve the breadth and depth of news data collection and achieve more accurate personalized recommendation effect of news content.

[0004] To achieve the above object, the technical solution adopted by the present invention is:

[0005] A news information publishing platform based on big data intelligent mining, comprising: a multi-source data acquisition module, a deep semantic analysis module and a personalized push module; the multi-source data acquisition module, the deep semantic analysis module and the personalized push module are communicatively connected;

[0006] The multi-source data acquisition module is used to obtain relevant news data and user behavior data from various data sources, and integrate heterogeneous data from different channels through data cleaning and fusion technology to generate a unified news data set and user behavior data set;

[0007] The deep semantic analysis module is used to perform deep semantic analysis on the integrated unified news data set to generate deep semantic analysis results, including topic classification, sentiment tendency judgment, key person time extraction and semantic similarity calculation;

[0008] The personalized push module is used to extract features based on the collected user behavior data set to build a user portrait, monitor the user's new behavior data in real time and update the user portrait, provide a highly personalized news recommendation list based on the deep semantic analysis results, establish a user feedback channel and adjust the recommendation strategy.

[0009] Furthermore, the multi-source data acquisition module includes: a data acquisition unit and a data processing unit.

[0010] in,

[0011] A data collection unit is used to obtain news related information from a data source that provides an API, and formulate relevant call rules and frequency limits according to the API document. The data source includes: mainstream news websites, social media platforms, professional forums, blogs, and video sharing platforms; the news related information includes title, text, release time, location, and author;

[0012] The data processing unit is used to clean the collected news-related information, remove noise data, duplicate data and incomplete data, convert data from different data sources into a unified format, and formulate relevant data standards, including field naming, data type and value range; integrate similar data from multiple data sources based on familiarity.

[0013] Furthermore, the deep semantic analysis module is used to perform deep semantic analysis on the collected news text, which specifically includes the following steps:

[0014] S1. Using news text as input, the preset BERT model is used to extract features and perform preliminary semantic understanding on the news text, and identify entity information in the news text, including key persons, organizations, places, and times;

[0015] S2. According to the preset news topic classification model, the classification rules are constructed by the news text features to determine the topic category to which the news belongs;

[0016] S3, construct a dictionary containing sentiment words and corresponding sentiment polarity to assist in determining the sentiment tendency of the text;

[0017] S4. Calculate the similarity scores between different news texts based on preset standards to provide data support for subsequent personalized recommendation deduplication processing and public opinion analysis.

[0018] Further, in step S1, the steps of building the preset BERT model are as follows:

[0019] S11, annotate the key people, organizations, places and times in the large amount of news text data collected and divide the data set into training set, verification set and test set according to 7:2:1;

[0020] S12. Preprocess the news text, including cleaning special characters, word segmentation, converting the news text into an input format acceptable to the BERT model, adding special tags and converting words in the news text into indexes in the model vocabulary;

[0021] S13, passing the processed input news text to the BERT model for forward propagation to obtain the hidden state of the model output;

[0022] S14. Add a fully connected layer on top of the output of the BERT model as a classification layer for entity recognition.

[0023] S15. Train the model using the training set, define the loss function and optimizer, calculate the loss in each training batch and update the model parameters through back propagation, evaluate the performance of the trained model on the validation set, and use accuracy, recall, and F1 value to evaluate the effect of entity recognition.

[0024] Further, in step S2, the steps of building the preset news topic classification model are specifically as follows:

[0025] S21, collect historical annotation data sets, the data sets contain news on different topics;

[0026] S22. Use text segmentation technology to split news text into independent words or phrases, and remove stop words to reduce the number of features during model training;

[0027] S23. The TF-IDF model is used to calculate the weight of each word and is input into a preset random forest model as a feature value, and the topic classification described in the corresponding news text is output.

[0028] Further, in step S23, the preset random forest model includes the following building steps:

[0029] S231, data preparation: prepare a historical annotated data set including features and labels, wherein the features are weights of each word in the historical annotated data set, and the labels include the following topics: politics, economy, sports, society, technology, and entertainment;

[0030] S232, data set division: divide the data set into a training set and a test set, using 70% of the data samples in the data set as the training set and 30% of the samples as the test set;

[0031] S233, Random Sampling: Randomly extract a certain number of data samples from the training set and construct multiple different training sets with replacement;

[0032] S234, constructing a decision tree: for each randomly sampled training set, randomly selecting a number of feature variables to construct a CART decision tree model;

[0033] S235, Ensemble Decision Tree: Integrate multiple decision trees into a random forest model and use the average prediction of multiple decision trees to make the final prediction;

[0034] S236. Model evaluation. Use evaluation metrics to evaluate the classification performance of the random forest model on the test set.

[0035] Further, in step S3, the construction of a dictionary containing sentiment words and corresponding sentiment polarities includes the following steps:

[0036] S31, obtaining various news texts in the historical annotation data set and user comments in the relevant news comment area;

[0037] S32, filter out high-frequency words in the text and manually annotate them based on the characteristics of the news;

[0038] S33, integrating the existing general sentiment dictionary with the words extracted and annotated from the news data, and removing duplicates;

[0039] S34. The constructed dictionary is cross-validated using news texts of different types and topics and updated regularly.

[0040] Further, in step S4, the similarity scores between different news texts are calculated according to a preset standard, including the following steps:

[0041] S41. Build a vocabulary covering all words in the unified news dataset;

[0042] S42, converting the news text into a single vector and calculating the TF-IDF feature vector;

[0043] S43, obtaining similarity scores between different texts by combining the preset standard with the similarity, wherein the similarity calculation formula is:

[0044]

[0045] Among them, a i 、b i are the i-th element in the feature vectors of text A and B respectively, are the means of the eigenvectors, n represents the dimension of the eigenvectors A and B, i.e. the number of elements, is the standard deviation of the eigenvector A, is the standard deviation of the eigenvector B.

[0046] Furthermore, the personalized push module extracts features based on the user's behavior data to construct a user portrait, specifically including the following steps:

[0047] Collect relevant user behavior data, including browsing history, reading time, interactive activity, interest keyword weights, and topic attention weights;

[0048] Clean and preprocess the collected data to remove duplicate and invalid data records and handle missing values;

[0049] Extract relevant features from the preprocessed data, including article browsing frequency, average reading time, interactive activity, and interest keyword weights;

[0050] The calculation formula for article browsing frequency is as follows:

[0051]

[0052] Among them, F represents the frequency of article browsing, N i Indicates the number of times a user browses articles in a specific category, N c Indicates the total number of views by the user;

[0053] The average reading time is calculated as follows:

[0054]

[0055] Among them, T represents the average reading time, M represents the number of times a user reads articles of a specific category, and T j Indicates the time it takes a user to read an article of a specific category for the jth word;

[0056] The calculation formula for interactive activity is as follows:

[0057]

[0058] Among them, A represents the interactive activity, N t 、N r 、N s Respectively represent the number of likes, comments and shares, N b Indicates the total number of articles viewed by users;

[0059] The calculation formula for interest keyword weight is as follows:

[0060]

[0061] Among them, W represents the weight of the interest keyword, L is the number of keywords, and F k represents the number of times the kth keyword appears, C k Indicates the importance coefficient of the kth keyword;

[0062] The extracted features are integrated to construct a user portrait, which is represented by a vector. Each feature is a dimension of the vector. The user portrait is updated regularly based on the accumulation of user behavior data.

[0063] Furthermore, it also includes a public opinion monitoring module, which is used to monitor user feedback and public opinion dynamics after news release in real time, evaluate the dissemination effect and influence of news through deep semantic analysis results, set relevant thresholds to timely adjust the recommendation strategy and news release strategy, and specifically includes the following steps:

[0064] Access to the public opinion system to enable real-time monitoring of multiple data sources;

[0065] Based on the results of deep semantic analysis, the sentiment tendency of news texts and the sentiment classification of user comments are obtained to evaluate the user's sentiment tendency towards news;

[0066] According to historical annotated data sets and industry standards, set relevant thresholds for user feedback volume and sentiment ratio, and regularly evaluate and adjust the thresholds;

[0067] Adjust the personalized recommendation algorithm according to relevant thresholds, and adjust the news push time according to public opinion trends and user active time.

[0068] The beneficial effects of the present invention are:

[0069] The present invention ensures safe and stable acquisition of rich and diverse news data through a deep multi-source data collection module, so as to provide users with a wider range of news choices to meet diverse interest needs. The combination of the breadth of multi-source data collection and the deep semantic analysis module enables the present invention to more comprehensively grasp the user's interest preferences and behavior patterns, not limited to the superficial browsing history, and deeply understand the user's attention to different topics, emotions and key elements, so as to provide more accurate recommendations.

[0070] The news of the present invention can reach the target audience more effectively through the personalized push module, improve the exposure and reading volume of news, enhance the dissemination influence of news, and improve the social value and influence of news. The public opinion monitoring module tracks the dissemination effect of news and user feedback in real time, provides a dynamic adjustment basis for the recommendation system, can timely discover changes in user interests and trends in hot topics, quickly adjust the recommendation strategy, and ensure that the recommended content is consistent with user needs. Through the collaboration between the modules, the present invention greatly improves the user experience and enhances the accuracy of news recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of a news information publishing platform based on big data intelligent mining of the present invention. DETAILED DESCRIPTION

[0072] See also Figure 1 As shown, the present invention provides a news information publishing platform based on big data intelligent mining, which can meet the diversified needs for news information, improve the accuracy of content analysis of news information, and improve the effect and value of personalized recommendation.

[0073] The news information publishing platform based on big data intelligent mining includes: a multi-source data acquisition module, a deep semantic analysis module and a personalized push module; the multi-source data acquisition module, the deep semantic analysis module and the personalized push module are communicatively connected;

[0074] The multi-source data acquisition module is used to obtain relevant news data and user behavior data from various data sources, and integrate heterogeneous data from different channels through data cleaning and fusion technology to generate a unified news data set and user behavior data set;

[0075] Furthermore, the multi-source data acquisition module includes: a data acquisition unit and a data processing unit, wherein:

[0076] A data collection unit is used to obtain news related information from a data source that provides an API, and formulate relevant call rules and frequency limits according to the API document. The data source includes: mainstream news websites, social media platforms, professional forums, blogs, and video sharing platforms; the news related information includes title, text, release time, location, and author;

[0077] The data processing unit is used to clean the collected news-related information, remove noise data, duplicate data and incomplete data, convert data from different data sources into a unified format, and formulate relevant data standards, including field naming, data type and value range; integrate similar data from multiple data sources based on familiarity.

[0078] Specifically, sort out and classify various data sources that provide APIs, obtain interface specifications, data formats, access permissions and other information based on the API documents of each data source, and formulate calling rules based on the provisions of the API documents, including the amount of data obtained for each request, the time interval between requests, etc., to ensure that data is obtained under the premise of compliance and does not affect the normal service of the data source.

[0079] After receiving the news-related data obtained by the data collection unit, the data processing unit performs data cleaning operations, removes noise data, including garbled characters, invalid characters, etc., through data filtering algorithms, and uses data comparison algorithms to identify and delete duplicate data records to ensure the uniqueness of the data. Check the integrity of the data and mark or delete data records that are missing key fields (such as titles, text, etc.). Convert the diverse data formats from different data sources into a preset standard format.

[0080] Furthermore, the deep semantic analysis module is used to perform deep semantic analysis on the integrated unified news data set to generate deep semantic analysis results, including topic classification, sentiment tendency judgment, key person time extraction and semantic similarity calculation;

[0081] The deep semantic analysis module is used to perform deep semantic analysis on the collected news text, and specifically includes the following steps:

[0082] S1. Using news text as input, the preset BERT model is used to extract features and perform preliminary semantic understanding on the news text, and identify entity information in the news text, including key persons, organizations, places, and times;

[0083] Furthermore, in step S1, the steps of building the preset BERT model are specifically as follows:

[0084] S11. Collect a large amount of news text data, annotate key people, organizations, places and time, and divide the data set into training set, verification set and test set according to 7:2:1;

[0085] S12. Preprocess the news text, including cleaning special characters, word segmentation, and converting the news text into the input format received by the BERT model, adding special tags and converting the words in the news text into indexes in the model vocabulary;

[0086] S13, passing the processed input news text to the BERT model for forward propagation to obtain the hidden state of the model output;

[0087] S14. Add a fully connected layer on top of the output of the BERT model as a classification layer for entity recognition.

[0088] S15. Train the model using the training set, define the loss function and optimizer, calculate the loss in each training batch and update the model parameters through back propagation, evaluate the performance of the trained model on the validation set, and use accuracy, recall, and F1 value to evaluate the effect of entity recognition.

[0089] Specifically, the data collected by the multi-source data collection module is obtained and annotated, and the annotated data set is divided into a training set, a validation set, and a test set in a ratio of 7:2:1. A suitable word segmentation tool, such as Jieba, is selected to segment the news text into words or terms, and special tags (such as [CLS] and [SEP]) are added according to the requirements of the model, and the words in the news text are converted into indexes in the model vocabulary so that the model can process them.

[0090] The processed news text is input into the model, which encodes and calculates the input to obtain the hidden states of the output. These hidden states contain the semantic representation of the text. A fully connected layer is connected on top of the BERT model input as a classification layer, which is used to classify and identify entities based on the hidden states.

[0091] The model is trained using the data in the training set. In this embodiment, the loss function uses the cross entropy function to measure the difference between the model prediction and the true label. The Adam optimizer is used to update the model parameters by calculating the gradient of the loss for directional propagation. The above process is repeated in each training batch to gradually optimize the performance of the model. The performance of the trained model is evaluated using the validation set and the test set, and the accuracy, recall rate, and F1 value are calculated to evaluate the effect of entity recognition.

[0092] Through the language understanding ability and pre-trained knowledge of the BERT model, various entities in news texts can be accurately identified, reducing errors and omissions. Feature extraction and preliminary semantic understanding help to better grasp the overall meaning and context of news texts, providing more valuable information for follow-up.

[0093] S2. According to the preset news topic classification model, the classification rules are constructed by the news text features to determine the topic category to which the news belongs;

[0094] Furthermore, in step S2, the steps of building the preset news topic classification model are specifically as follows:

[0095] S21, collect historical annotation data sets, the data sets contain news on different topics;

[0096] S22. Use text segmentation technology to split news text into independent words or phrases, and remove stop words to reduce the number of features during model training;

[0097] S23. The TF-IDF model is used to calculate the weight of each word and is input into a preset random forest model as a feature value, and the topic classification described in the corresponding news text is output.

[0098] Specifically, the TF-IDF model is used to calculate the weight of each word in the text as the feature value. For a given text set, the term frequency (TF) and inverse document frequency (IDF) are calculated. For example, for a specific document d and a word t, the formula for calculating the term frequency is:

[0099]

[0100] Among them, TF t,d represents the frequency of word t in document d, n t,d represents the number of times word t appears in document d, Σ t′∈ d n t′,d Represents the total number of times all words appear in document d; for example, if word t appears 10 times in document d, and the total number of times all words appear in document d is 100, then n t,d =10,∑ t′∈d nt′,d =100, then TF t,d =0.1.

[0101] The calculation formula for inverse document frequency is:

[0102]

[0103] Among them, IDF t represents the inverse document frequency of word t, N is the total number of documents in the corpus, df t is the number of documents containing word t. For example, there are 1000 documents in the entire text collection, and there are 100 documents containing word t, that is, N = 1000, df t =100, then IDF t =log10=1, then we can calculate TF-IDF t,d =0.1x1=0.1;

[0104] Calculate the TF-IDF value for each word in document d in this way to get the weight of each word in the document, highlighting the words that appear frequently in the document and are relatively uncommon in the entire text collection. Represent the news text as a vector composed of the TF-IDF value of word t. The vectors of all news texts are combined into a feature matrix, where each row represents a news text and each column represents a word. Input the constructed feature matrix into the preset random forest model to output the topic classification to which the news text belongs.

[0105] Furthermore, in step S23, the preset random forest model includes the following building steps:

[0106] S231, data preparation: prepare a historical annotated data set including features and labels, wherein the features are weights of each word in the historical annotated data set, and the labels include the following topics: politics, economy, sports, society, technology, and entertainment;

[0107] S232, data set division, divide the data set into a training set and a test set, using 70% of the data samples in the data set as the training set and 30% of the samples as the test set;

[0108] S233, Random Sampling: Randomly extract a certain number of data samples from the training set and construct multiple different training sets with replacement;

[0109] S234, constructing a decision tree: for each randomly sampled training set, randomly selecting a number of feature variables to construct a CART decision tree model;

[0110] S235. Integrated Decision Tree: Integrate multiple constructed decision trees into a random forest model, and make the final prediction using the predicted average value of multiple decision trees;

[0111] S236. Model Evaluation: Evaluate the classification performance of the random forest model on the test set using evaluation metrics.

[0112] It should be noted that when collecting a large number of historical labeled data sets, since the data is based on historical periods, the themes and keywords of the news are known. Decompose the historical labeled data set into independent words or phrases and remove common stop words, such as "de", "shi", "zai", etc., to reduce the number of features during model training.

[0113] It should be noted that CART is a decision tree algorithm that uses the Gini coefficient to evaluate the quality of different features for dividing the data set, so as to select the optimal splitting feature. By integrating multiple CART decision tree algorithms, the bias and variance of a single decision tree can be reduced. News text data may have complex features and noises, and CART decision trees are prone to overfitting. The random forest integrates multiple decision trees. Due to random sampling and feature selection of the training data, each decision tree only sees part of the data and features. By synthesizing the prediction results of different decision trees, the bias that may occur in a single decision tree can be reduced, and the possibility of overfitting can be lowered, enabling the model to better generalize to new news text data and improving the overall classification accuracy. The dimension of the feature vector extracted by TF-IDF is relatively high, and the random forest can effectively handle high-dimensional features and avoid the fitting problem.

[0114] Specifically, first collect a large number of data sets with historical labels. Each record in these data sets contains the weights of each word in the news text as features, as well as the corresponding topic labels. The labels include categories such as politics, economy, sports, society, technology, and entertainment to ensure the accuracy and integrity of the data, providing a reliable basis for subsequent model training and evaluation.

[0115] Randomly divide the prepared data set into a training set and a test set according to a ratio of 7:3. Randomly sample with replacement from the training set, and each time draw a certain number of data samples. Repeat multiple times to construct multiple different training subsets to increase data diversity and reduce the risk of overfitting. For each randomly sampled training subset, randomly select a part of the feature variables to construct a CART decision tree model. Build the decision tree based on information gain and the Gini coefficient, and continuously split the nodes until the predetermined stopping condition is reached.

[0116] Combine multiple decision trees to form a random forest model. When predicting new news text data, let each decision tree give its own prediction results. Then take the average of these prediction results as the final prediction of the random forest model and evaluate the random forest model through the test set.

[0117] S3, construct a dictionary containing sentiment words and corresponding sentiment polarity to assist in determining the sentiment tendency of the text;

[0118] Further, in step S3, constructing a dictionary containing sentiment words and corresponding sentiment polarities includes the following steps:

[0119] S31, obtaining various news texts in the historical annotation data set and user comments in the relevant news comment area;

[0120] S32, filter out high-frequency words in the text and manually annotate them based on the characteristics of the news;

[0121] S33, integrating the existing general sentiment dictionary with the words extracted and annotated from the news data, and removing duplicates;

[0122] S34. The constructed dictionary is cross-validated using news texts of different types and topics and updated regularly.

[0123] It should be noted that the received news text data includes key persons, organizations, locations, time and classification of the news text.

[0124] Specifically, obtain historical annotated data sets through the API interface, parse the obtained JSON or other format data, and extract the strings of news text and comment content. Clean the data, including: remove special characters, convert to unified code, word segmentation, etc. Set frequency thresholds, filter high-frequency words by counting the frequency of each word, and annotate high-frequency words according to news characteristics, including annotating sentiment tendencies and topic categories. Integrate existing general sentiment dictionaries and store them in a structured data format. Merge the words extracted and annotated from news data with general sentiment dictionaries, and use data processing technology to quickly detect and remove duplicate words. Select representative news texts of different types and topics as verification sets, use the built dictionaries to compare known correct annotations or manual evaluation results, optimize and adjust the dictionaries based on the verification results, and repeat the verification and update of the dictionaries according to the preset time period.

[0125] S4. Calculate the similarity scores between different news texts based on preset standards to provide data support for subsequent personalized recommendation deduplication processing and public opinion analysis.

[0126] Furthermore, in step S4, the similarity scores between different news texts are calculated according to a preset standard, including the following steps:

[0127] S41. Build a vocabulary covering all words in the unified news dataset;

[0128] S42, converting the news text into a single vector and calculating the TF-IDF feature vector;

[0129] S43, obtaining similarity scores between different texts by combining the preset standard with the similarity, wherein the similarity calculation formula is:

[0130]

[0131] Among them, a i 、b i are the i-th element in the feature vectors of text A and B respectively, are the means of the eigenvectors, n represents the dimension of the eigenvectors A and B, i.e. the number of elements, is the standard deviation of the eigenvector A, is the standard deviation of the eigenvector B.

[0132] It should be noted that accurately calculating the similarity between news texts can effectively remove duplicate or highly similar news content, provide users with diverse and novel recommendations, improve the quality of subsequent personalized recommendations, and help to more accurately identify and group similar public opinion viewpoints, analyze the mainstream trends of public opinion and the distribution of different viewpoints.

[0133] The personalized push module is used to extract features based on the collected user behavior data set to build a user portrait, monitor the user's new behavior data in real time and update the user portrait, provide a highly personalized news recommendation list based on the deep semantic analysis results, establish a user feedback channel and adjust the recommendation strategy.

[0134] Furthermore, extracting features from the collected user behavior data set to construct a user portrait specifically includes the following steps:

[0135] Collect relevant user behavior data, including browsing history, reading time, interactive activity, interest keyword weights, and topic attention weights;

[0136] Clean and preprocess the collected data to remove duplicate and invalid data records and handle missing values;

[0137] Extract relevant features from the preprocessed data, including article browsing frequency, average reading time, interactive activity, and interest keyword weights;

[0138] The calculation formula for article browsing frequency is as follows:

[0139]

[0140] Among them, F represents the frequency of article browsing, N i Indicates the number of times a user browses articles in a specific category, N c Indicates the total number of views by the user;

[0141] The average reading time is calculated as follows:

[0142]

[0143] Among them, T represents the average reading time, M represents the number of times a user reads articles of a specific category, and T j Indicates the time it takes a user to read an article of a specific category for the jth word;

[0144] The calculation formula for interactive activity is as follows:

[0145]

[0146] Among them, A represents the interactive activity, N t 、N r 、N s Respectively represent the number of likes, comments and shares, N b Indicates the total number of articles viewed by users;

[0147] The calculation formula for interest keyword weight is as follows:

[0148]

[0149] Among them, W represents the weight of the interest keyword, L is the number of keywords, and F k represents the number of times the kth keyword appears, C k Represents the importance coefficient of the kth keyword;

[0150] The extracted features are integrated to construct a user portrait. The user's image is represented by a vector. Each feature is used as a dimension of the vector. The user portrait is updated regularly based on the accumulation of user behavior data.

[0151] Specifically, the user's browsing frequency reflects the user's initial interest in different topics, and the average time the user spends reading each article is calculated. For articles of each topic, the average reading time is calculated separately. A longer average reading time means that the user has a deeper attention and interest in the topic. High interactive reading means that the user is more actively involved in and concerned about the topic. Key interest keywords are extracted from the user's browsing history, comment content, etc. For example, if the word "artificial intelligence" appears frequently and is important in the user's behavior data, it will be given a higher weight.

[0152] The topic classification of news articles obtained through deep semantic analysis is matched with the topics with high frequency of browsing, long reading time or high interactive activity shown in the user portrait. For example, if the user shows a strong interest in science and technology topics, news on science and technology topics will be recommended first; considering the user's emotional inclination towards different topics in previous reading, when the user shows positive emotions towards science and technology news, more positive science and technology news will be recommended, and related topic news that may cause negative emotions will be avoided; when the user is interested in a specific key person (such as a certain technology entrepreneur) or content related to a specific time (such as the relevant time of a certain historical period), news involving related elements will be pushed first, and users will be provided with the latest developments and reports on the people or time they are concerned about in a timely manner. The semantic similarity between news texts is used to reduce news with highly similar content, and at the same time, based on the semantic similarity between the news to be recommended and the news that the user was interested in in the past, news with higher similarity will be placed in front.

[0153] By integrating user portrait features and deep semantic analysis results, a highly personalized news recommendation list is generated for users that accurately suits their interests and needs, thereby improving their reading experience and satisfaction with the system.

[0154] Furthermore, it also includes a public opinion monitoring module, which is used to monitor user feedback and public opinion dynamics after news release in real time, evaluate the dissemination effect and influence of news through deep semantic analysis results, set relevant thresholds to timely adjust the recommendation strategy and news release strategy, and specifically includes the following steps:

[0155] Access the public opinion system to automatically monitor multiple data sources in real time;

[0156] Based on the results of deep semantic analysis, the sentiment tendency of news texts and the sentiment classification of user comments are obtained to evaluate the user's sentiment tendency towards news;

[0157] According to historical annotated data sets and industry standards, set relevant thresholds for user feedback volume and sentiment ratio, and regularly evaluate and adjust the thresholds;

[0158] Adjust the personalized recommendation algorithm according to relevant thresholds, and adjust the news push time according to public opinion trends and user active time.

[0159] Specifically, connect to the public opinion monitoring system, determine the multiple data sources that need to be monitored, ensure that relevant data after the news is released can be obtained in real time, and collect user feedback data after the news is released in real time, such as user comments, likes, forwarding, sharing and other behavioral data and related text content.

[0160] The deep semantic analysis module is used to analyze news texts and user comments, extract keywords, semantic information and emotional elements from the text, and judge the emotional tendency of the news text itself (such as positive, negative, neutral). The same deep semantic analysis is performed on the content of user comments, and each comment is classified as positive, negative or neutral. The number and proportion of comments with different emotional classifications are counted to fully understand the user's emotional response to the news. The emotional tendency of the news text and the emotional classification results of user comments are combined to evaluate the overall emotional tendency of users towards the news. If the news text is positive but a large number of user comments are negative, then the overall emotional tendency may be negative. If the news text is negative but a large number of user comments are positive, then the overall emotional tendency may be positive.

[0161] Based on the historical annotated data set, analyze the user feedback volume (such as the number of comments, the number of likes, the number of reposts, etc.) and sentiment ratio (the proportion of positive, negative, and neutral comments) data of similar news in the past. Refer to the general standards and experience values ​​in the industry to set the initial user feedback volume threshold (such as the number of comments reaching 1,000, the number of likes reaching 500, etc.) and sentiment ratio threshold (such as the proportion of positive comments less than 30% is a negative communication effect). Statistics and analysis of the user feedback volume and sentiment ratio of news are conducted at regular intervals. Compare the actual data with the set threshold to evaluate the rationality and effectiveness of the threshold. If the actual data shows that the original threshold is too high or too low, resulting in an inaccurate assessment of the news communication effect and influence, the threshold is appropriately adjusted according to the evaluation results. For example, if it is found that a large number of news are misjudged as having poor communication effects under the original threshold, but in fact the user feedback is positive, then the relevant threshold is appropriately lowered; otherwise, the threshold is appropriately increased.

[0162] When the amount of user feedback and the proportion of emotional tendency of news exceed or fall below the preset threshold, the personalized push module is optimized appropriately. If the news has a good dissemination effect (such as a large amount of user feedback and a high proportion of positive emotional tendency), the recommendation weight and ranking of the news in the personalized recommendation list are increased, and the recommendation frequency is increased; otherwise, its recommendation weight is reduced or the recommendation is stopped. Analyze public opinion dynamics and user active time data to understand the user's attention and participation in news in different time periods. According to the user's active time period (such as 7-10 pm is the peak period of user activity), news with good dissemination effect will be pushed during the user's active time period to increase the exposure and dissemination effect of the news. For news with poor dissemination effect, the push frequency can be reduced to avoid waste of resources.

[0163] The above implementation modes are merely descriptions of the preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary engineering and technical personnel in the field shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A news information publishing platform based on big data intelligent mining, characterized in that: include: Multi-source data collection module, deep semantic analysis module and personalized push module; The multi-source data acquisition module, the deep semantic analysis module and the personalized push module are communicatively connected; The multi-source data acquisition module is used to obtain relevant news data and user behavior data from various data sources, and integrate heterogeneous data from different channels through data cleaning and fusion technology to generate a unified news data set and user behavior data set; The deep semantic analysis module is used to perform deep semantic analysis on the integrated unified news data set to generate deep semantic analysis results, including topic classification, sentiment tendency judgment, key person time extraction and semantic similarity calculation; The personalized push module is used to extract features based on the collected user behavior data set to build a user portrait, monitor the user's new behavior data in real time and update the user portrait, provide a highly personalized news recommendation list based on the deep semantic analysis results, establish a user feedback channel and adjust the recommendation strategy.

2. The news information publishing platform based on big data intelligent mining according to claim 1 is characterized in that: The multi-source data acquisition module includes: a data acquisition unit and a data processing unit, wherein: A data collection unit is used to obtain news related information from a data source that provides an API, and formulate relevant call rules and frequency limits according to the API document. The data source includes: mainstream news websites, social media platforms, professional forums, blogs, and video sharing platforms; the news related information includes title, text, release time, location, and author; The data processing unit is used to clean the collected news-related information, remove noise data, duplicate data and incomplete data, convert data from different data sources into a unified format, and formulate relevant data standards, including field naming, data type and value range; integrate similar data from multiple data sources based on familiarity.

3. The news information publishing platform based on big data intelligent mining according to claim 1 is characterized in that: The deep semantic analysis module is used to perform deep semantic analysis on the collected news text, and specifically includes the following steps: S1. Using news text as input, the preset BERT model is used to extract features and perform preliminary semantic understanding on the news text, and identify entity information in the news text, including key persons, organizations, places, and times; S2. According to the preset news topic classification model, the classification rules are constructed by the news text features to determine the topic category to which the news belongs; S3, construct a dictionary containing sentiment words and corresponding sentiment polarity to assist in determining the sentiment tendency of the text; S4. Calculate the similarity scores between different news texts based on preset standards to provide data support for subsequent personalized recommendation deduplication processing and public opinion analysis.

4. The news information publishing platform based on big data intelligent mining according to claim 3 is characterized in that: In step S1, the steps for building the preset BERT model are as follows: S11, annotate the key people, organizations, places and times in the large amount of news text data collected and divide the data set into training set, verification set and test set according to 7:2:1; S12. Preprocess the news text, including cleaning special characters, word segmentation, converting the news text into an input format acceptable to the BERT model, adding special tags and converting words in the news text into indexes in the model vocabulary; S13, passing the processed input news text to the BERT model for forward propagation to obtain the hidden state of the model output; S14. Add a fully connected layer on top of the output of the BERT model as a classification layer for entity recognition. S15. Train the model using the training set, define the loss function and optimizer, calculate the loss in each training batch and update the model parameters through back propagation, evaluate the performance of the trained model on the validation set, and use accuracy, recall, and F1 value to evaluate the effect of entity recognition.

5. The news information publishing platform based on big data intelligent mining according to claim 3 is characterized in that: In step S2, the steps for building the preset news topic classification model are as follows: S21, collect historical annotation data sets, the data sets contain news on different topics; S22. Use text segmentation technology to split news text into independent words or phrases, and remove stop words to reduce the number of features during model training; S23. The TF-IDF model is used to calculate the weight of each word and is input into a preset random forest model as a feature value, and the topic classification described in the corresponding news text is output.

6. The news information publishing platform based on big data intelligent mining according to claim 5 is characterized in that: In step S23, the preset random forest model includes the following building steps: S231, data preparation: prepare a historical annotated data set including features and labels, wherein the features are weights of each word in the historical annotated data set, and the labels include the following topics: politics, economy, sports, society, technology, and entertainment; S232, data set division: divide the data set into a training set and a test set, using 70% of the data samples in the data set as the training set and 30% of the samples as the test set; S233, Random Sampling: Randomly extract a certain number of data samples from the training set and construct multiple different training sets with replacement; S234, constructing a decision tree: for each randomly sampled training set, randomly selecting a number of feature variables to construct a CART decision tree model; S235, Ensemble Decision Tree: Integrate multiple decision trees into a random forest model and use the average prediction of multiple decision trees to make the final prediction; S236. Model evaluation. Use evaluation metrics to evaluate the classification performance of the random forest model on the test set.

7. The news information publishing platform based on big data intelligent mining according to claim 3 is characterized in that: In step S3, constructing a dictionary containing sentiment words and corresponding sentiment polarities comprises the following steps: S31, obtaining various news texts in the historical annotation data set and user comments in the relevant news comment area; S32, filter out high-frequency words in the text and manually annotate them based on the characteristics of the news; S33, integrating the existing general sentiment dictionary with the words extracted and annotated from the news data, and removing duplicates; S34. The constructed dictionary is cross-validated using news texts of different types and topics and updated regularly.

8. The news information publishing platform based on big data intelligent mining according to claim 3 is characterized in that: In step S4, the similarity scores between different news texts are calculated according to a preset standard, including the following steps: S41. Build a vocabulary covering all words in the unified news dataset; S42, converting the news text into a single vector and calculating the TF-IDF feature vector; S43, obtaining similarity scores between different texts by combining the preset standard with the similarity, wherein the similarity calculation formula is: Among them, a i 、b i are the i-th element in the feature vectors of text A and B respectively, are the means of the eigenvectors, n represents the dimension of the eigenvectors A and B, i.e. the number of elements, is the standard deviation of the eigenvector A, is the standard deviation of the eigenvector B.

9. The news information publishing platform based on big data intelligent mining according to claim 1 is characterized in that: The personalized push module extracts features based on the user's behavior data to build a user portrait, which specifically includes the following steps: Collect relevant user behavior data, including browsing history, reading time, interactive activity, interest keyword weights, and topic attention weights; Clean and preprocess the collected data to remove duplicate and invalid data records and handle missing values; Extract relevant features from the preprocessed data, including article browsing frequency, average reading time, interactive activity, and interest keyword weights; The calculation formula for article browsing frequency is as follows: Among them, F represents the frequency of article browsing, N i Indicates the number of times a user browses articles in a specific category, N c Indicates the total number of views by the user; The average reading time is calculated as follows: Among them, T represents the average reading time, M represents the number of times a user reads articles of a specific category, and T j Indicates the time it takes a user to read an article of a specific category for the jth word; The calculation formula for interactive activity is as follows: Among them, A represents the interactive activity, N t 、N r 、N s Respectively represent the number of likes, comments and shares, N b Indicates the total number of articles viewed by users; The calculation formula for interest keyword weight is as follows: Among them, W represents the weight of the interest keyword, L is the number of keywords, and F k represents the number of times the kth keyword appears, C k Indicates the importance coefficient of the kth keyword; The extracted features are integrated to construct a user portrait, which is represented by a vector. Each feature is a dimension of the vector. The user portrait is updated regularly based on the accumulation of user behavior data.

10. The news information publishing platform based on big data intelligent mining according to claim 1 is characterized in that: It also includes a public opinion monitoring module, which is used to monitor user feedback and public opinion dynamics after news release in real time, evaluate the dissemination effect and influence of news through deep semantic analysis results, set relevant thresholds to timely adjust recommendation strategies and news release strategies, and specifically includes the following steps: Access to the public opinion system to enable real-time monitoring of multiple data sources; Based on the results of deep semantic analysis, the sentiment tendency of news texts and the sentiment classification of user comments are obtained to evaluate the user's sentiment tendency towards news; According to historical annotated data sets and industry standards, set relevant thresholds for user feedback volume and sentiment ratio, and regularly evaluate and adjust the thresholds; Adjust the personalized recommendation algorithm according to relevant thresholds, and adjust the news push time according to public opinion trends and user active time.

Citation Information

Patent Citations

  • Personalized news recommendation device and method based on news content and theme feature

    CN102831234A

  • Method and device for judging classification to which article belongs based on word segmentation weight, and storage medium

    CN110674289A

  • Intelligent automobile article classification method and system based on media field

    CN114281992A

  • News information aggregation method based on data mining and deep learning

    CN116541486A

  • Resource recommendation system and method based on knowledge graph

    CN117573894A

Cited By

  • Artificial intelligence real-time news recommendation system based on model

    CN119474553A

  • Media information acquisition method and system, electronic equipment and storage medium

    CN120353990A

  • A media information acquisition method and system, an electronic device, and a storage medium

    CN120353990B

  • Public opinion monitoring system and method based on big data

    CN120561280A

  • Method and device for mapping attribute features of news into single primary colors in multi-mode RGB intelligent color matching

    CN120723911A