Publication trend prediction method and system based on big data

Abstract

The invention discloses a publication trend prediction method and system based on big data, and the system comprises a data collection module, a feature extraction and analysis module, a trend prediction model construction module, a trend prediction and evaluation module, and a result visualization and interaction module. The output end of the data acquisition module is electrically connected with the input end of a feature extraction and analysis module, and the output end of the feature extraction and analysis module is electrically connected with the input end of a trend prediction model construction module. According to the publication trend prediction method and system based on the big data, the big data technology is utilized, publication related data are widely collected from multiple data sources, the problem that data samples of a traditional method are limited is solved, the prediction result is more comprehensive, objective and accurate, and the real demand and trend of the market can be reflected.

Description

Technical Field

[0001] The present invention relates to the field of big data analysis and publishing technology, and specifically to a publication trend prediction method and system based on big data. Background Art

[0002] With the rapid development of information technology, the number and types of publications have exploded. How to accurately grasp future trends in the vast amount of publication information is of great significance to publishers, authors, libraries and the entire cultural industry.

[0003] Traditional publication trend forecasting methods often rely on expert experience and small-scale market research, which have problems such as strong subjectivity, limited data samples, low forecast accuracy and poor timeliness. The emergence of big data technology provides a new way to solve these problems. By collecting, sorting and analyzing massive amounts of publication-related data, it can more comprehensively and accurately predict the development trend of publications and provide decision-making support for all relevant parties. In this regard, we propose a publication trend forecasting method and system based on big data to solve the above problems. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method and system for predicting publication trends based on big data, which solves the problems raised in the background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a publication trend prediction system based on big data, including a data acquisition module, a feature extraction and analysis module, a trend prediction model construction module, a trend prediction and evaluation module, and a result visualization and interaction module. The output end of the data acquisition module is electrically connected to the input end of the feature extraction and analysis module, the output end of the feature extraction and analysis module is electrically connected to the input end of the trend prediction model construction module, the output end of the trend prediction model construction module is electrically connected to the input end of the trend prediction and evaluation module, and the output end result visualization of the trend prediction and evaluation module is electrically connected to the input end of the interaction module.

[0006] Preferably, the data acquisition module includes a web crawler unit and a data cleaning unit, the feature extraction and analysis module includes a text feature extraction unit and a data association analysis unit, the trend prediction model construction module includes a model selection unit and a model training unit, the trend prediction and evaluation module includes a trend prediction unit and a prediction evaluation unit, and the result visualization and interaction module includes a visualization display unit and an interactive operation unit.

[0007] Preferably, the web crawler unit is used to capture publication-related data from multiple data sources, the data cleaning unit is used to clean and preprocess the collected raw data to obtain a normalized data set, the text feature extraction unit is used to extract features of publication-related text data, and the data association analysis unit is used to perform association analysis on the data set to explore potential relationships between data.

[0008] Preferably, the model selection unit is used to select a suitable prediction model according to the prediction target and data characteristics, and the model training unit is used to train the selected model using the feature data to obtain a trained prediction model.

[0009] Preferably, the trend prediction unit is used to use the trained model to perform trend prediction on the data to be predicted and output the prediction result, and the prediction evaluation unit is used to evaluate the prediction result using the evaluation index and optimize the model according to the evaluation result.

[0010] Preferably, the visualization display unit is used to display the prediction results in a visualized form, and the interactive operation unit is used to provide a user interaction interface, allowing the user to operate the prediction results and adjust the system according to the user operation feedback.

[0011] Preferably, the multiple data sources include book sales platforms, library collection catalogs, academic databases, social media platforms, and official websites of publishing organizations; the text features extracted by the text feature extraction unit using natural language processing technology include keywords, subject terms, and sentiment tendencies; the prediction model selected in the trend prediction model construction module includes at least one of a time series model, a classification model, and a clustering model.

[0012] The present invention also discloses a method for predicting publication trends based on big data, which specifically comprises the following steps:

[0013] Step 1: Data collection step: Use web crawlers to capture publication-related data from multiple data sources, and perform cleaning and preprocessing;

[0014] Step 2: Feature extraction and analysis: Extract text features and conduct correlation analysis on the data, and store the results in the feature database;

[0015] Step 3: Trend prediction model construction step: Select a suitable prediction model and train it to obtain a trained prediction model;

[0016] Step 4: Trend prediction and evaluation step: Use the trained model to predict trends, and evaluate and optimize the prediction results;

[0017] Step 5: Result visualization and interaction step: Visualize the prediction results and provide a user interaction interface to make adjustments based on user operation feedback.

[0018] Beneficial Effects

[0019] The present invention provides a method and system for predicting publication trends based on big data. Compared with the prior art, it has the following beneficial effects:

[0020] (1) The present invention utilizes big data technology to extensively collect publication-related data from multiple data sources, overcoming the problem of limited data samples in traditional methods, making the prediction results more comprehensive, objective, and accurate, and able to reflect the real needs and trends of the market.

[0021] (2) Through in-depth feature extraction and correlation analysis of the data, more information and patterns hidden behind the data can be unearthed, providing rich feature dimensions for the prediction model and improving the prediction ability and accuracy of the model.

[0022] (3) By adopting a variety of advanced prediction models and through training and optimization, it is able to make accurate predictions for different prediction targets, meeting the diverse decision-making needs of the publishing industry.

[0023] (4) The design of the result visualization and interactive module makes the prediction results more intuitive and easy to understand, which is convenient for users to view, analyze and operate, improving user experience and decision-making efficiency. It also makes it easier for users to adjust and optimize the prediction process according to actual conditions, further improving the practicality and reliability of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a system block diagram of the present invention;

[0025] Figure 2 It is a schematic diagram of a data acquisition module of the present invention;

[0026] Figure 3 It is a schematic diagram of a feature extraction and analysis module of the present invention;

[0027] Figure 4 A schematic diagram of a trend prediction model construction module of the present invention;

[0028] Figure 5 It is a schematic diagram of the trend prediction and evaluation module of the present invention;

[0029] Figure 6 It is a schematic diagram of the result visualization and interaction module of the present invention.

[0030] In the figure: 01, data acquisition module; 02, feature extraction and analysis module; 03, trend prediction model construction module; 04, trend prediction and evaluation module; 05, result visualization and interaction module; 011, web crawler unit; 012, data cleaning unit; 021, text feature extraction unit; 022, data association analysis unit; 031, model selection unit; 032, model training unit; 041, trend prediction unit; 042, prediction evaluation unit; 051, visualization display unit; 052, interactive operation unit. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The present invention provides three technical solutions, including the following embodiments:

[0033] Example 1

[0034] See also Figure 1-Figure 6 A publication trend prediction system based on big data includes a data acquisition module 01, a feature extraction and analysis module 02, a trend prediction model construction module 03, a trend prediction and evaluation module 04, and a result visualization and interaction module 05. The output end of the data acquisition module 01 is electrically connected to the input end of the feature extraction and analysis module 02, the output end of the feature extraction and analysis module 02 is electrically connected to the input end of the trend prediction model construction module 03, the output end of the trend prediction model construction module 03 is electrically connected to the input end of the trend prediction and evaluation module 04, and the output end of the trend prediction and evaluation module 04 is electrically connected to the input end of the result visualization and interaction module 05.

[0035] The data acquisition module 01 includes a web crawler unit 011 and a data cleaning unit 012, the feature extraction and analysis module 02 includes a text feature extraction unit 021 and a data association analysis unit 022, the trend prediction model construction module 03 includes a model selection unit 031 and a model training unit 032, the trend prediction and evaluation module 04 includes a trend prediction unit 041 and a prediction evaluation unit 042, and the result visualization and interaction module 05 includes a visualization display unit 051 and an interactive operation unit 052.

[0036] First, in the data collection module 01, the web crawler unit 011 sets crawling rules for relevant data sources in the field of computer science, such as searching for book information in this field on major book sales platforms, screening the publication of papers in this field in academic databases, and capturing discussion topics and reviews about books in this field on social media platforms such as technical forums. After a period of crawling, a large amount of raw data is obtained. The data cleaning unit 012 cleans these data, removes duplicate book records, incomplete sales data, and invalid comment information, and unifies the data format, such as unifying the date format to "YYYY-MM-DD", and converting the text data into code.

[0037] In the feature extraction and analysis module 02, the text feature extraction unit 021 performs word segmentation on the book title, abstract and reader comments, removes stop words, and uses the TF-IDF algorithm to extract keywords and subject words, such as "artificial intelligence algorithm", "big data processing technology", etc. At the same time, the data association analysis unit 022 uses the association rule mining algorithm to find that in this field, books written by well-known experts often have higher sales and citation times, and books published within one year after the emergence of new technologies are more likely to receive attention. These feature and relationship data are stored in the feature database.

[0038] In the trend prediction model construction module 03, since we want to predict sales trends and hot topics, the model selection unit 031 selects the LSTM time series model for sales trend prediction and the naive Bayes classification model for hot topic classification prediction. The model training unit 032 uses the data in the feature database to train the two models, and divides the data into a training set and a test set in a ratio of 80:20. After multiple training and parameter adjustments, the mean square error of the LSTM model on the test set reached a low level, and the accuracy of the naive Bayes model also reached the predetermined standard, and two trained prediction models were obtained.

[0039] In the trend prediction and evaluation module 04, the latest computer science publication data in the current market is input into the trained model. The LSTM model predicts that the sales trend of books in this field in the next year will show a trend of first rising and then stabilizing, and there will be a sales peak within three months after the release of new technologies; the naive Bayes model predicts that "artificial intelligence and medical health applications" and "innovation of blockchain technology in the financial field" will become hot topics. The prediction evaluation unit 042 uses the mean square error and accuracy to evaluate the prediction results, and finds that the mean square error of the sales trend prediction is within an acceptable range, and the accuracy of the topic classification prediction has reached a high level, meeting the requirements of practical applications.

[0040] Finally, the result visualization and interaction module 05 displays the sales trend in the form of a line graph and presents the hot topics in the form of a word cloud graph. The user can choose to view the sales trend details of different time periods or learn more about the relevant book information under each hot topic through the interactive operation unit 052. At the same time, the user can also fine-tune the parameters of the prediction model based on his own experience and judgment, and re-run the prediction process to obtain a prediction result that better meets his expectations.

[0041] Example 2

[0042] Based on Example 1, see Figure 2-Figure 6 As shown, the web crawler unit 011 is used to capture publication-related data from multiple data sources, the data cleaning unit 012 is used to clean and preprocess the collected raw data to obtain a normalized data set, the text feature extraction unit 021 is used to extract features of publication-related text data, and the data association analysis unit 022 is used to perform association analysis on the data set to explore potential relationships between data.

[0043] The model selection unit 031 is used to select a suitable prediction model according to the prediction target and data characteristics, and the model training unit 032 is used to train the selected model using the feature data to obtain a trained prediction model.

[0044] The trend prediction unit 041 is used to use the trained model to perform trend prediction on the data to be predicted and output the prediction result. The prediction evaluation unit 042 is used to evaluate the prediction result using the evaluation index and optimize the model according to the evaluation result.

[0045] The visualization display unit 051 is used to display the prediction results in a visualized form, and the interactive operation unit 052 is used to provide a user interaction interface, allowing the user to operate the prediction results and adjust the system according to the user operation feedback.

[0046] Multiple data sources include book sales platforms, library collection catalogs, academic databases, social media platforms, and official websites of publishing organizations. The text feature extraction unit 021 uses natural language processing technology to extract text features including keywords, subject terms, and sentiment tendencies. The prediction model selected in the trend prediction model construction module 03 includes at least one of a time series model, a classification model, and a clustering model.

[0047] Example 3

[0048] Based on Example 2, see Figure 1 As shown, the present invention also discloses a publication trend prediction method based on big data, which specifically includes the following steps:

[0049] Step 1: Data collection step: Capture publication-related data from multiple data sources through a web crawler, and perform cleaning and preprocessing. Start the web crawler unit 011 of the data collection module 01, and capture publication-related data from multiple data sources according to preset rules and frequencies, and transmit the captured data to the data cleaning unit 012. The data cleaning unit 012 cleans and preprocesses the original data, removes invalid data, and converts the data format to obtain a standardized data set.

[0050] Step 2: Feature extraction and analysis: Perform text feature extraction and association analysis on the data, and store the results in a feature database. The text feature extraction unit 021 processes the text data to extract features such as keywords and subject terms. At the same time, the data association analysis unit 022 performs association analysis on the data set to mine the potential relationship between the data. The extracted and analyzed features and relationship data are stored in the feature database for subsequent model training.

[0051] Step 3: Trend prediction model construction step: Select a suitable prediction model and train it to obtain a trained prediction model. The model selection unit 031 selects a suitable prediction model according to the prediction target and data characteristics and initializes it. The model training unit 032 uses the data in the feature database to train the selected model, and adjusts the model parameters through multiple iterations until the model reaches the predetermined performance index on the validation set, thereby obtaining a trained prediction model.

[0052] Step 4: Trend prediction and evaluation step: Use the trained model to predict trends, and evaluate and optimize the prediction results. The trend prediction unit 041 inputs the publication data to be predicted into the trained prediction model to obtain the trend prediction results. The prediction evaluation unit 042 uses the evaluation index to evaluate the prediction results, records the evaluation results, and determines whether the model needs to be optimized based on the evaluation results. If the evaluation results are not ideal, return to the model building step, reselect the model or adjust the parameters for training; if the evaluation results meet the requirements, proceed to the next step;

[0053] Step 5: Visualization and interaction of results: Visualize the prediction results and provide a user interaction interface. Adjust according to user operation feedback. The visualization display unit 051 visualizes the prediction results, generates intuitive charts and reports, and displays them to the user on the user interface provided by the interactive operation unit 052. The user can view, filter, analyze and other operations on the prediction results through the interactive operation unit to obtain the required publication trend information. At the same time, the system adjusts and optimizes the visualization display and prediction model according to the user's operation feedback.

[0054] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0055] The above embodiments of the invention are described in detail, but the contents are only preferred embodiments of the invention and cannot be considered to limit the scope of the invention. All equivalent changes and improvements made within the scope of the invention should still fall within the scope of the invention.

Claims

1. A publication trend prediction system based on big data, characterized by: The invention comprises a data acquisition module (01), a feature extraction and analysis module (02), a trend prediction model construction module (03), a trend prediction and evaluation module (04), and a result visualization and interaction module (05), wherein the output end of the data acquisition module (01) is electrically connected to the input end of the feature extraction and analysis module (02), the output end of the feature extraction and analysis module (02) is electrically connected to the input end of the trend prediction model construction module (03), the output end of the trend prediction model construction module (03) is electrically connected to the input end of the trend prediction and evaluation module (04), and the output end result visualization and interaction module (05) is electrically connected to the input end of the trend prediction and evaluation module (04).

2. A publication trend prediction system based on big data according to claim 1, characterized in that: The data acquisition module (01) comprises a web crawler unit (011) and a data cleaning unit (012); the feature extraction and analysis module (02) comprises a text feature extraction unit (021) and a data association analysis unit (022); the trend prediction model construction module (03) comprises a model selection unit (031) and a model training unit (032); the trend prediction and evaluation module (04) comprises a trend prediction unit (041) and a prediction evaluation unit (042); and the result visualization and interaction module (05) comprises a visualization display unit (051) and an interactive operation unit (052).

3. The publication trend prediction system based on big data according to claim 1, characterized in that: The web crawler unit (011) is used to capture publication-related data from multiple data sources, the data cleaning unit (012) is used to clean and preprocess the collected raw data to obtain a normalized data set, the text feature extraction unit (021) is used to extract features of publication-related text data, and the data association analysis unit (022) is used to perform association analysis on the data set to mine potential relationships between the data.

4. A publication trend prediction system based on big data according to claim 2, characterized in that: The model selection unit (031) is used to select a suitable prediction model according to the prediction target and data characteristics, and the model training unit (032) is used to train the selected model using the feature data to obtain a trained prediction model.

5. The publication trend prediction system based on big data according to claim 2, characterized in that: The trend prediction unit (041) is used to use the trained model to perform trend prediction on the data to be predicted and output the prediction result. The prediction evaluation unit (042) is used to evaluate the prediction result using the evaluation index and optimize the model according to the evaluation result.

6. The publication trend prediction system based on big data according to claim 2, characterized in that: The visualization display unit (051) is used to display the prediction results in a visualized form, and the interactive operation unit (052) is used to provide a user interaction interface, allowing the user to operate the prediction results and adjust the system according to the user operation feedback.

7. The publication trend prediction system based on big data according to claim 2, characterized in that: The multiple data sources include book sales platforms, library collection catalogs, academic databases, social media platforms and official websites of publishing institutions. The text features extracted by the text feature extraction unit (021) using natural language processing technology include keywords, subject terms, and sentiment tendencies. The prediction model selected in the trend prediction model construction module (03) includes at least one of a time series model, a classification model, and a clustering model.

8. A method for predicting publication trends based on big data, characterized by: The specific steps include: Step 1: Data collection step: Use web crawlers to capture publication-related data from multiple data sources, and perform cleaning and preprocessing; Step 2: Feature extraction and analysis: Extract text features and conduct correlation analysis on the data, and store the results in the feature database; Step 3: Trend prediction model construction step: Select a suitable prediction model and train it to obtain a trained prediction model; Step 4: Trend prediction and evaluation step: Use the trained model to predict trends, and evaluate and optimize the prediction results; Step 5: Result visualization and interaction step: Visualize the prediction results and provide a user interaction interface to make adjustments based on user operation feedback.

Images