Real-time web crawler framework implementation method and system based on hot word detection

By using a real-time web crawler framework based on hot word detection, combined with deep learning models and adaptive scheduling algorithms, efficient, real-time and high-quality data crawling is achieved, solving the problems of low efficiency, slow strategy and low quality in existing technologies, and improving user experience and system performance.

CN119046521APending Publication Date: 2024-11-29STATE GRID ANHUI ELECTRIC POWER CO LTD ELECTRIC POWER SCI RES INST +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411007084.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Existing technologies for web crawlers suffer from low crawling efficiency, slow real-time adjustment of crawling strategies, and low quality of crawled data.

Method used

A real-time web crawler framework based on hot word detection is adopted, which combines deep learning models and adaptive scheduling algorithms. The user interface is designed using a pre-built front-end framework. The hot word detection module calculates the popularity score and dynamically adjusts the web crawler tasks. Distributed crawling technology is used for multi-node parallel crawling, and real-time updates and feedback of information are achieved through text processing and database storage.

Benefits of technology

It improves the crawling efficiency and data quality of web crawlers, ensures the real-time nature and relevance of information, optimizes the utilization of system resources, reduces the crawling of irrelevant data, and enhances the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046521A_ABST
    Figure CN119046521A_ABST
Patent Text Reader

Abstract

The invention provides a real-time web crawler framework implementation method and system based on hot word detection, and the method comprises the steps: hot word source selection: defining a hot word source, and constructing a hot word library; the hot word detection module is used for monitoring changes of hot word sources in real time and determining weights and priorities of hot words; web crawler scheduling: formulating a crawler scheduling algorithm, and starting or adjusting a crawling task; webpage capturing: accessing a target website by using a web crawler technology, and capturing HTML (Hypertext Markup Language) contents; and text analysis and processing: carrying out text analysis on the captured content, and carrying out text preprocessing. The technical problems of low crawling efficiency, slow real-time crawling strategy adjustment and low crawling data quality are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of information acquisition, and in particular to a method and system for realizing a real-time web crawler framework based on hot word detection. Background Art

[0002] With the rapid development of the Internet, the amount of information is growing exponentially, making it essential to effectively obtain and analyze network information. Web crawlers, as an automated tool for collecting information from the Internet, are essential for search engines, social media monitoring, news and public opinion analysis, and other fields. However, traditional web crawler strategies have some problems, such as indiscriminate data capture, resource waste, and lack of information timeliness. To solve these problems, a real-time web crawler strategy based on hot word detection has emerged.

[0003] A web crawler

[0004] A web crawler is an automated program used to crawl web page information on the Internet. Its background is closely related to the development of the Internet, the explosion of information, and the rise of search engines. With the continuous popularization and development of the Internet, the amount of information has increased rapidly. Users are faced with a huge amount of information, but how to quickly and accurately obtain the required information has become a difficult problem. The information on the Internet is growing explosively, and users need an efficient way to retrieve and obtain information of interest, and manual retrieval obviously can no longer meet the needs. People are in urgent need of a tool to organize and retrieve this information. The rise of search engines has solved this problem, and web crawlers are the core of the operation of search engines. In order to build the index of the search engine, a large amount of information on the Internet needs to be obtained. As an important part of the search engine, web crawlers provide real-time and comprehensive data support for search engines by automatically crawling and indexing web page content. In order to improve the efficiency and accuracy of information retrieval, an automated tool is needed to collect, organize, and index information on the Internet. This is one of the driving forces for the development of web crawlers. Web crawlers have a wide range of applications on the Internet, not only limited to search engines, but also involving multiple fields, including data mining, business intelligence, news aggregation, public opinion analysis, etc.

[0005] Second, hot word detection

[0006] With the continuous development of the Internet, the explosive growth of information has become a prominent feature of contemporary society. Users generate a large amount of text data in Internet spaces such as social media, news platforms, and forums, making it a challenge to quickly and accurately capture key information. Hot word detection has emerged as a technology for real-time analysis of text data and capturing key information, providing a solution to information overload. Hot word detection has shown a wide range of applications in many fields, providing key support for real-time information monitoring, market analysis, public opinion management, etc.

[0007] First, hot word detection plays an important role in social media monitoring. By analyzing the content posted by users on social media in real time, hot word detection can identify hot topics that are currently attracting attention and provide real-time user feedback and public opinion trends for enterprises, brands and government agencies.

[0008] Secondly, hot word detection has significant applications in news aggregation and recommendation. By analyzing keywords in news reports, hot word detection helps users obtain news related to current events and provides personalized news recommendation services, enabling users to obtain information more effectively.

[0009] In terms of advertising optimization, hot word detection can be used to analyze topics that users are interested in, providing advertisers with more accurate advertising strategies. By placing ads in content related to popular keywords, the exposure and click-through rate of ads can be increased.

[0010] Hot word detection also plays a key role in event monitoring and emergency response. By identifying keywords related to an event in real time, hot word detection can help relevant agencies to understand the development of the event in a timely manner, conduct emergency response and public opinion management, and maintain public safety and social stability.

[0011] In terms of market research and competitive intelligence, companies can use hot word detection to analyze user discussions on social media, understand market trends and competitor dynamics, and provide real-time information support for formulating marketing and product strategies.

[0012] Finally, hot word detection is of great significance to brand management and marketing. By real-time monitoring of hot keywords related to brands on social media, companies can better understand consumer feedback, respond to user needs in a timely manner, maintain brand reputation, and improve market competitiveness.

[0013] Overall, hot word detection technology plays a key role in information analysis and public opinion monitoring, providing various industries with an efficient and real-time means of information capture and analysis, helping to better cope with the challenges of today's information age.

[0014] Traditional web crawler technology has the following defects:

[0015] 1. Low crawling efficiency: Traditional web crawlers are usually limited by their design and algorithms, and may encounter performance bottlenecks when acquiring data, especially in the case of large-scale or complex web page structures. This may lead to slow crawling speeds and affect the efficiency of data collection, especially in scenarios that require a large amount of data;

[0016] 2. Slow real-time adjustment of crawling strategies: Dynamic changes in network environment and website structure may require crawlers to adjust their strategies to maintain effectiveness. Traditional crawlers may be slow to respond to these changes in real time, because adjusting crawling strategies requires time and resources, resulting in delays in adapting to new environments or changes in web page structures.

[0017] 3. Low quality of crawled data: Traditional crawlers often cannot ensure the quality of the data they obtain, which may be affected by the complexity of page structure, the diversity of information formats, and anti-crawler mechanisms. This may lead to problems with the accuracy, completeness, and consistency of the data, requiring further cleaning and verification steps to ensure the credibility and quality of the data.

[0018] Comparing our implementation method of the real-time web crawler framework based on hot word detection with these two public documents, our method has the following improvements:

[0019] 1. Integration of hot word detection and task scheduling:

[0020] Our method integrates hot word detection with web crawler task scheduling, dynamically adjusting web crawler tasks based on hot word priorities, user needs, and system resources, while methods in the public literature mainly focus on text vectorization and model building, without involving dynamic adjustment of task scheduling. This integrated method can more flexibly adjust web crawler tasks according to real-time conditions, improving the real-time performance and efficiency of the system.

[0021] 2. Hot word detection using a pre-built deep learning model:

[0022] Our method uses a pre-set deep learning model for hot word detection, which can more accurately extract hot words in the text. The methods in the public literature are mainly based on topic modeling and attention mechanism for text representation and feature extraction, and are not specifically optimized for hot word detection. By using a deep learning model, we can better capture the semantic information of the text and improve the accuracy and efficiency of hot word detection.

[0023] 3. Adaptive scheduling algorithm for dynamically adjusting web crawler tasks:

[0024] Our method uses an adaptive scheduling algorithm to dynamically adjust the frequency and scale of web crawler tasks based on hot word priorities, user needs, and system resources. The methods in the public literature are mainly based on new event topic clustering analysis and model updating based on multi-feature fusion. Our adaptive scheduling algorithm can more intelligently allocate web crawler tasks, optimize system resource utilization, and improve the efficiency and stability of web crawlers.

[0025] These improvements can solve some technical defects of the existing technology, such as:

[0026] 1. Inaccurate hot word detection:

[0027] Traditional hot word detection methods usually rely on keyword frequency statistics or simple matching rules, which are easily affected by noise and misjudgment. When these methods process large amounts of data or complex text, it is difficult to accurately identify truly important hot words, thus affecting the accuracy of analysis.

[0028] Our method uses a pre-built deep learning model for hot word detection. We are able to extract hot words in the text more accurately, thus avoiding the problem of inaccurate hot word detection in traditional methods.

[0029] 2. Web crawler task scheduling is not flexible:

[0030] Traditional web crawlers usually adopt fixed scheduling strategies and lack the ability to respond to real-time changes. This approach may cause crawler tasks to be too frequent or insufficient when faced with changes in network load or inconsistent data update frequency, affecting data capture efficiency and resource utilization.

[0031] Our method dynamically adjusts the adaptive scheduling algorithm of web crawler tasks. We can flexibly adjust the frequency and scale of web crawler tasks according to real-time conditions, avoiding the problem of inflexible scheduling of web crawler tasks in traditional methods.

[0032] 3. Low utilization of system resources:

[0033] Crawling tasks with fixed frequency and scale are prone to waste or shortage of resources, and cannot dynamically adjust tasks according to the current system load and network conditions. This inflexible scheduling strategy may cause system resources to be idle or overloaded, reducing overall resource utilization and crawling efficiency.

[0034] Our approach optimizes the frequency and scale of web crawler tasks through an adaptive scheduling algorithm. We are able to utilize system resources more effectively, improving the utilization of system resources and the efficiency of web crawlers. Summary of the invention

[0035] The technical problem to be solved by the present invention is: how to solve the technical problems of low crawling efficiency, slow real-time adjustment of crawling strategy and low quality of crawled data in the prior art.

[0036] The present invention adopts the following technical solutions to solve the above technical problems: A method for implementing a real-time web crawler framework based on hot word detection includes:

[0037] S1. Use the preset front-end framework to design the user interface and collect user input information in the user interface;

[0038] S2. Use the preset hot word detection module to obtain and train the preset deep learning model, perform hot word detection on the user input information, and calculate the heat score of each word through the hot word detection module. The specific formula is as follows:

[0039] Score(w i )=Attention(w i ,C)×Frequency(w i )

[0040] Among them, Score(w i ) represents word w i The score of Attention(w i ,C) represents word w i The importance weight in context C is calculated using the attention mechanism. Frequency(w i ) represents word w i The frequency of occurrence.

[0041] Specifically, the relevant parameters are as follows:

[0042] Word vector dimension: set to 300 dimensions to ensure the accuracy of vocabulary representation.

[0043] Frequency threshold: Set an empirical value of 0.01 to remove words with too low frequency.

[0044] Heat score threshold: Set an empirical value of 0.05 and only retain words with a heat score higher than this value.

[0045] The specific improvement logic includes:

[0046] Introducing deep learning models:

[0047] Compared with the traditional method based on word frequency statistics, the deep learning model is used for semantic understanding and hot word detection, which improves the ability to capture complex language phenomena;

[0048] Using the attention mechanism:

[0049] The attention mechanism is used to evaluate the importance of each word in the context. Compared with using only word frequency statistics, it can more accurately identify key hot words, especially in long texts;

[0050] Dynamic Threshold Adjustment:

[0051] Implement a dynamic threshold adjustment mechanism to adaptively adjust the heat score threshold according to the characteristics of the input data to avoid screening too many or too few hot words;

[0052] The specific beneficial effects produced by the aforementioned technical features and the principles of producing the beneficial effects include:

[0053] Improve the accuracy and robustness of hotword detection:

[0054] Through extensive pre-training, deep learning models can capture the deep semantic relationships of words and more accurately identify hot words with practical significance than traditional methods;

[0055] Improve your understanding of context:

[0056] The attention mechanism can dynamically evaluate the importance of words in different contexts, making the hot word detection results more relevant and accurate;

[0057] Flexible adaptation to different data scenarios:

[0058] The dynamic threshold adjustment mechanism enables this method to adapt to data sets of different types and sizes, and can maintain good hot word detection results regardless of whether it is real-time streaming data or batch processing data;

[0059] Reduce the interference of noise words:

[0060] Through the combination of deep learning models and attention mechanisms, we can effectively filter out those noise words that appear frequently but are not important, thereby improving the quality and reliability of hot word detection.

[0061] S3. For the hot word detection module, use the scheduling and controller to dynamically adjust the web crawler tasks according to the hot words;

[0062] S4. Using the pre-set web crawler module, using the distributed crawler technology, to perform multi-node parallel crawling, enabling multiple crawler nodes, using the distributed crawler technology to perform parallel crawling; each node crawls the text information of the target website in parallel, and aggregates the data into a central storage device;

[0063] Specifically, crawl the target website and plan the web crawler task according to the system performance information and the target website to crawl the text information. The task allocation formula is as follows:

[0064]

[0065] Among them, W i represents the weight assigned to the i-th node. i Indicates the priority of the i-th task. i Represents the available capacity or performance index of the i-th node.

[0066] Specifically, the control formula for the crawling frequency obtained by multi-node parallel crawling is as follows:

[0067]

[0068] Among them, F iIndicates the crawling frequency of the i-th node. max Indicates the preset maximum crawling frequency. i Represents the current load of the i-th node. i Represents the response rate of the target website to the i-th node.

[0069] The relevant parameters are as follows:

[0070] Number of nodes (N): Dynamically adjusted according to system performance and the scale of the target website.

[0071] Crawl frequency (F): The crawl frequency of each node is dynamically adjusted according to the restrictions of the target website.

[0072] Task Priority (P): Set the task priority based on the priority of the hot words and the relevance of the target website.

[0073] Timeout setting (T): The timeout threshold for each crawling task to optimize crawling efficiency.

[0074] The specific improvement logic of the above technical features is as follows:

[0075] Dynamic mission planning:

[0076] Dynamically plan and adjust crawling tasks according to system performance and the characteristics of the target website to improve crawling efficiency and flexibility;

[0077] Intelligent Scheduling:

[0078] Use load balancing strategies and task priorities to achieve intelligent task scheduling and optimize resource utilization;

[0079] Distributed parallel crawling:

[0080] Adopting a distributed architecture, multiple nodes crawl in parallel to improve crawling speed and system scalability;

[0081] Real-time performance monitoring:

[0082] Monitor system performance and access status of target websites in real time, dynamically adjust crawling strategies, and ensure system stability and crawling continuity.

[0083] The beneficial effects and principles of the aforementioned technical features include:

[0084] Improve crawling efficiency:

[0085] Multi-node parallel crawling significantly improves the speed of data crawling and can obtain a large amount of text information in a shorter time.

[0086] Enhance system robustness:

[0087] The distributed architecture and intelligent scheduling mechanism improve the robustness and fault tolerance of the system. Even if a node fails, it will not affect the overall crawling task.

[0088] Optimize resource utilization:

[0089] Dynamic task planning and load balancing strategies optimize the use of system resources, avoiding waste of resources and accumulation of tasks;

[0090] Improve the timeliness and relevance of data:

[0091] Through real-time performance monitoring and intelligent scheduling, crawling tasks can be adjusted and updated in a timely manner to ensure that the acquired data is highly timely and relevant;

[0092] Reduce the pressure on the target website:

[0093] Through reasonable crawling frequency control and distributed crawling methods, the access pressure on a single target website can be reduced, and the risk of being blocked by the target website can be lowered.

[0094] S5, using a text processing engine and a preset parsing library tool to perform text processing operations on the text information to obtain text structured information;

[0095] S6. Use hot words to build a hot word library. According to the hot word library and the preset word vector model, design a hot word matching and filtering module to match and filter text structured information; according to the corresponding weights of the hot words, filter to obtain hot topic related information as the target content;

[0096] S7, selecting a preset database to establish an index structure for data storage and retrieval operations, and storing and retrieving the processed data;

[0097] S8, using a preset real-time update and feedback module and a preset network protocol to perform an information interconnection update operation and deliver the currently updated target content to the user;

[0098] S9. Use the preset exception handling and optimization module to automate the exception handling mechanism, obtain and handle crawling process exceptions based on user feedback results.

[0099] The present invention adopts a hot word detection deep learning model to accurately identify the hot words input by the user. By improving the training and application of the model, the user's intention can be understood more accurately. The present invention designs an adaptive scheduling algorithm for real-time adjustment of network crawler tasks. The algorithm flexibly adapts to the changes in the hot words input by the user, improving the real-time performance and user satisfaction of the system. The present invention uses distributed crawler technology to achieve multi-node parallel crawling, thereby improving the efficiency of information collection. This innovation in technical implementation makes the system more scalable and efficient. The present invention uses a hot word matching and filtering module, combined with a hot word library and a word vector model, to achieve more accurate information filtering.

[0100] In a more specific technical solution, in S1, the preset front-end framework includes: React.

[0101] In a more specific technical solution, in S2, the deep learning model includes: a BERT model, which obtains context information and diversity information to perform model training operations on the BERT model to extract hot words, and the input text is encoded by the BERT model to generate a word vector for each word.

[0102] Specifically, the attention weight calculation formula (based on BERT's self-attention mechanism) is as follows:

[0103]

[0104] Where Q is the query vector (Query), K is the key vector (Key), and V is the value vector (Value). k is the dimension of the key vector.

[0105] Specifically, the improved logic of extracting hot words using the BERT model is as follows:

[0106] Fusion of multiple information

[0107] Combining context information and word frequency information, the hot word score is calculated by weighted calculation to more comprehensively evaluate the importance of the word;

[0108] Real-time hot word updates

[0109] Using the powerful semantic understanding ability of the BERT model, the hot word list is updated in real time to ensure the timeliness and relevance of hot words;

[0110] Enhance understanding of user needs

[0111] Based on the hot word weights, a more sophisticated user demand model is built to improve the understanding of user intent and the accuracy of feedback.

[0112] The beneficial effects and principles of the aforementioned technical features include:

[0113] Improve the accuracy of hot word extraction

[0114] By combining contextual information and word frequency information, the BERT model can more accurately identify key hot words in the text and enhance the flexibility of the system.

[0115] Dynamically adjust the weight coefficient so that the system can adapt to different data types and application scenarios, with higher flexibility and adaptability;

[0116] Improve user experience

[0117] Real-time update of hot word lists and personalized feedback mechanisms can better meet user needs and improve user experience and satisfaction;

[0118] Optimize resource utilization

[0119] Accurate hot word extraction and demand analysis optimize the use of system resources and avoid unnecessary calculations and data processing.

[0120] Through the hot word detection of the present invention, the web crawler can better understand the crawled data, provide more accurate data analysis results, and optimize the effect of data analysis. For example, the user's interest trend can be understood through hot word detection to provide support for the recommendation system. For enterprises, hot word detection can help understand market dynamics and consumer demand and provide a basis for decision-making. For example, e-commerce websites can understand the most popular products through hot word detection and then adjust their sales strategies.

[0121] In a more specific technical solution, in S3, the priority of hot words, user needs, and system resources are obtained and processed to utilize an adaptive scheduling algorithm to allocate the frequency and scale of web crawler tasks.

[0122] Specifically, the adaptive scheduling algorithm includes:

[0123] S31. Task frequency allocation: Dynamically adjust the crawling frequency of the crawler task according to the priority of hot words and system resources. The formula is as follows:

[0124]

[0125] where F i Indicates the crawling frequency of the i-th node. max Indicates the preset maximum crawling frequency. i Indicates the priority of hot words. i Represents the current load of the i-th node. i Represents the response rate of the target website to the i-th node.

[0126] S32. Task scale adjustment: According to the system resources and the status of the target website, the scale formula of the crawler task is dynamically adjusted as follows:

[0127]

[0128] Where: S i S represents the scale of crawling tasks assigned to the i-th node. max Indicates the preset maximum task size. C i It represents the available capacity or performance index of the ith node. N represents the total number of crawler nodes.

[0129] The specific improvement logic of the above technical features is as follows:

[0130] Multi-factor comprehensive assessment:

[0131] Combine hot word scores, user needs and system resources, and calculate comprehensive priorities by weight to ensure the rationality and scientificity of task allocation;

[0132] Real-time dynamic adjustment:

[0133] According to the system performance and the status of the target website, the frequency and scale of crawler tasks can be adjusted dynamically in real time to improve the flexibility and adaptability of the system;

[0134] Adaptive Scheduling Algorithm:

[0135] Using adaptive scheduling algorithms, tasks are dynamically allocated based on priority and system resources to optimize resource utilization and task execution efficiency;

[0136] Intelligent monitoring and feedback:

[0137] Monitor system performance and network status in real time, and dynamically adjust crawler tasks through an intelligent feedback mechanism to ensure system stability and efficient operation.

[0138] The beneficial effects and principles of the above technical features are as follows:

[0139] Improve crawling efficiency:

[0140] Dynamically adjust crawling frequency and task scale to ensure that high-priority tasks are executed first and improve overall crawling efficiency;

[0141] Optimize resource utilization:

[0142] Dynamically allocate tasks according to system load and resource usage, optimize system resource utilization and avoid resource waste;

[0143] Enhance system robustness:

[0144] Real-time monitoring and dynamic adjustment mechanisms improve the robustness and fault tolerance of the system and ensure stable operation of tasks under different load conditions;

[0145] Improve data relevance and real-time performance:

[0146] Dynamically adjust crawling tasks based on user needs and hot word priorities to ensure that the acquired data is highly relevant and real-time.

[0147] By detecting hot words, the web crawler of the present invention can understand the topics that users are currently most concerned about, so as to give priority to crawling web pages related to these topics, thereby improving the efficiency and practicality of crawling and improving the efficiency of crawling.

[0148] The present invention can help web crawlers understand current hot spots in real time through hot word detection, adjust crawling strategies in time, capture the latest hot information, and update crawled content in real time.

[0149] The hot word detection operation of the present invention can help web crawlers screen out valuable information, avoid crawling a large amount of irrelevant or low-quality data, and improve the quality of crawled data.

[0150] In a more specific technical solution, in S4, the web crawler module includes but is not limited to: Scrapy crawler framework; managing the task queue through scheduling and controller, and regularly starting and stopping the crawling task operation; setting the crawling frequency according to the website robots.txt rules.

[0151] The present invention realizes real-time update and feedback mechanism through WebSocket technology, so that users can obtain the latest information instantly. This has a significant improvement in user experience and improves the real-time nature of information.

[0152] In a more specific technical solution, in S9, the algorithm and parameters are adjusted according to the user feedback results.

[0153] In a more specific technical solution, the real-time web crawler framework implementation system based on hot word detection includes:

[0154] The user interface module is used to design the user interface using the preset front-end framework and collect user input information in the user interface;

[0155] A hot word detection module is used to acquire and train a preset deep learning model, perform hot word detection on user input information to obtain hot words, and the hot word detection module is connected to the user interface module;

[0156] The scheduling and controller is used to dynamically adjust the web crawler tasks according to the hot words. The scheduling and controller is connected with the hot word detection module;

[0157] The web crawler module is used to crawl the target website with multi-node parallel crawling using distributed crawler technology. According to the system performance information and the target website, the web crawler task is planned to crawl text information. The web crawler module is connected with the scheduling and controller;

[0158] A text processing engine is used to perform text processing operations on text information using a preset parsing library tool to obtain text structured information. The text processing engine is connected to the web crawler module;

[0159] Hot word matching and filtering module, used to build a hot word library using hot words, and design a hot word matching and filtering module based on the hot word library and the preset word vector model to match and filter text structured information; according to the corresponding weight of the hot words, filter to obtain hot topic related information as the target content, and the hot word matching and filtering module is connected to the text processing engine;

[0160] A database memory is used to select a preset database to establish an index structure for data storage and retrieval operations, store and retrieve processed data, and the database memory is connected to the hot word matching and filtering module;

[0161] A real-time update and feedback module is used to utilize a preset network protocol to perform information interconnection update operations and transmit the currently updated target content to the user. The real-time update and feedback module is connected to a database storage device;

[0162] The exception handling and optimization module is used to utilize the automated exception handling mechanism to obtain and handle crawling process exceptions based on user feedback results. The exception handling and optimization module is connected to the real-time update and feedback module.

[0163] Compared with the prior art, the present invention has the following advantages:

[0164] The present invention adopts a hot word detection deep learning model to accurately identify the hot words input by the user. By improving the training and application of the model, the user's intention can be understood more accurately. The present invention designs an adaptive scheduling algorithm for real-time adjustment of network crawler tasks. The algorithm flexibly adapts to the changes in the hot words input by the user, improving the real-time performance and user satisfaction of the system. The present invention uses distributed crawler technology to achieve multi-node parallel crawling, thereby improving the efficiency of information collection. This innovation in technical implementation makes the system more scalable and efficient. The present invention uses a hot word matching and filtering module, combined with a hot word library and a word vector model, to achieve more accurate information filtering.

[0165] Through the hot word detection of the present invention, the web crawler can better understand the crawled data, provide more accurate data analysis results, and optimize the effect of data analysis. For example, the user's interest trend can be understood through hot word detection to provide support for the recommendation system. For enterprises, hot word detection can help understand market dynamics and consumer demand and provide a basis for decision-making. For example, e-commerce websites can understand the most popular products through hot word detection and then adjust their sales strategies.

[0166] By detecting hot words, the web crawler of the present invention can understand the topics that users are currently most concerned about, so as to give priority to crawling web pages related to these topics, thereby improving the efficiency and practicality of crawling and improving the efficiency of crawling.

[0167] The present invention can help web crawlers understand current hot spots in real time through hot word detection, adjust crawling strategies in time, capture the latest hot information, and update crawled content in real time.

[0168] The hot word detection operation of the present invention can help web crawlers screen out valuable information, avoid crawling a large amount of irrelevant or low-quality data, and improve the quality of crawled data.

[0169] The present invention realizes real-time update and feedback mechanism through WebSocket technology, so that users can obtain the latest information instantly. This has a significant improvement in user experience and improves the real-time nature of information.

[0170] The present invention solves the technical problems existing in the prior art of low crawling efficiency, slow real-time adjustment of crawling strategies and low quality of crawled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0171] Figure 1 This is a schematic diagram of basic modules of a real-time web crawler framework based on hot word detection according to Example 1 of the present invention;

[0172] Figure 2 This is a schematic diagram of the basic steps of the method for implementing a real-time web crawler framework based on hot word detection according to Example 2 of the present invention. DETAILED DESCRIPTION

[0173] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0174] Example 1

[0175] like Figure 1 As shown, the real-time web crawler framework implementation system based on hot word detection provided by the present invention includes the following basic modules:

[0176] User interface 1 provides an interactive interface for user input and real-time feedback. In this embodiment, the user inputs the hot words of interest and views the real-time crawled information.

[0177] In the user interface design of this embodiment, the user inputs the hot words of interest through the interface, and the interface transmits the user input to the hot word detection module 2 set in the back end. Real-time feedback is pushed to the user through technologies such as WebSocket, showing the dynamic changes of hot topics in the form of charts or prompts.

[0178] Hot word detection module 2 uses deep learning model BERT to perform real-time hot word detection, outputs hot word list and corresponding weights, and hot word detection module 2 is connected to user interface 1.

[0179] During the operation of the hot word detection module 2 of this embodiment, the deep learning model BERT is used for hot word detection. The model is trained to understand the context and recognize the hot words input by the user. Each detected hot word is extracted and assigned a corresponding weight or priority.

[0180] The scheduling and controller 3 implements an adaptive scheduling algorithm and adjusts the crawler task according to the information output by the hot word detection module 2. The scheduling and controller 3 controls the start, stop and adjustment of the network crawler. The scheduling and controller 3 is connected to the hot word detection module 2.

[0181] In this embodiment, the scheduling and controller 3 implements an adaptive scheduling algorithm and dynamically adjusts the web crawler tasks according to the output of the hot word detection module 2. In this embodiment, the priority of the hot words and system resources are considered to reasonably allocate the frequency and scale of the crawler tasks to adapt to the real-time changes of the hot words input by the user.

[0182] The web crawler module 4 uses the distributed crawler framework Scrapy-Cluster to achieve multi-node parallel crawling. Comply with website crawling rules to ensure legal crawling. The web crawler module 4 is connected to the scheduling and controller 3.

[0183] In this embodiment, the web crawler module 4 uses a framework such as Scrapy, combined with distributed crawler technology, to crawl the HTML content of the target website in parallel with multiple nodes. The scheduler is responsible for managing the task queue and regularly starting and stopping crawler tasks. The crawler module follows the crawling rules of the website to ensure efficient and legal acquisition of information.

[0184] The text processing engine 5 uses Spacy to parse the text and extract key information. It performs text preprocessing. In this embodiment, the text preprocessing operations include but are not limited to: word segmentation, part-of-speech tagging, and removal of stop words. The text processing engine 5 is connected to the web crawler module 4.

[0185] In this embodiment, the text processing engine 5 uses libraries such as Spacy to perform text parsing and preprocessing. The preprocessed text is converted into structured data to facilitate subsequent hot word matching and filtering.

[0186] The hot word matching and filtering module 6 matches and filters the text using the hot word library and the word vector model, and filters according to the weight or priority of the hot word. The hot word matching and filtering module 6 is connected to the text processing engine 5.

[0187] In this embodiment, a hot word matching and filtering module 6 is designed to match and filter texts in combination with a hot word library and a word vector model. The weight or priority of hot words is used to filter texts to ensure that information related to hot topics is retained and the quality of information is improved.

[0188] The database storage 7 uses Elasticsearch to create an index and stores information about hot topics in the database to provide efficient data retrieval and analysis. The database storage 7 is connected to the hot word matching and filtering module 6.

[0189] In this embodiment, a database, such as Elasticsearch, is selected to establish an index structure to support efficient data storage and retrieval. The processed information is stored in the database to ensure the reliability and retrievability of the data.

[0190] The real-time update and feedback module 8 sets up a real-time update mechanism to monitor the changes of hot topics and information, provides WebSocket real-time data push, and ensures that users can obtain the latest information. The real-time update and feedback module 8 is connected to the database storage 7.

[0191] In this embodiment, the real-time update and feedback module 8 feeds back the real-time crawled information and changes in hot topics to the user interface in real time through technologies such as WebSocket, so that users can instantly obtain the latest information through the interface, thereby enhancing the real-time performance of the system.

[0192] The exception handling and optimization module 9 implements the exception handling mechanism to handle network exceptions, crawling failures, etc. The crawler strategy is optimized regularly, and the algorithm and parameters are adjusted according to the feedback results. The exception handling and optimization module 9 is connected to the real-time update and feedback module 8.

[0193] In this embodiment, an exception handling and optimization module 9 is designed to handle abnormal situations that may occur during the crawling process. The stability and efficiency of the system are improved through an automated exception handling mechanism and regular optimization strategies.

[0194] Example 2

[0195] like Figure 2 As shown, the real-time web crawler framework implementation system based on hot word detection provided by the present invention includes the following basic modules:

[0196] Step S1, designing a user interface;

[0197] In this embodiment, the user interface is the entrance for the user to interact with the system, and the design needs to consider user experience, ease of use and real-time feedback.

[0198] In this embodiment, a modern front-end framework such as React is used to create an intuitive and interactive interface through component-based design. An input box is provided for users to enter the hot words they are interested in, and the dynamic changes of hot topics are displayed through charts or real-time prompts. Technologies such as WebSockets are used to ensure that users can obtain the latest information in a timely manner.

[0199] Step S2, performing hot word detection;

[0200] In this embodiment, a deep learning model, such as BERT, is selected for hot word detection. BERT has achieved remarkable results in NLP tasks and can more accurately identify hot words input by users. Through model training, hot words are extracted and corresponding weights or priorities are assigned to each hot word. By executing step S2, user needs are understood and feedback is given, so the training of the model needs to fully consider context and diversity.

[0201] Step S3, using the scheduling and controller 3 for the hot word detection module 2;

[0202] In this embodiment, the scheduling and controller 3 dynamically adjusts the web crawler tasks according to the output of the hot word detection module 2. An adaptive scheduling algorithm is implemented to reasonably allocate the frequency and scale of crawler tasks by considering the priority of hot words, user needs and system resources. This ensures that the system can flexibly respond to changes in hot words input by users, dynamically adjust crawler strategies, and improve the real-time and accuracy of information.

[0203] Step S4, crawling the target website using the web crawler module 4;

[0204] In this embodiment, crawler frameworks such as Scrapy are used in combination with distributed crawler technology to ensure multi-node parallel crawling. The task queue is managed by the scheduler to achieve regular start and stop of crawling tasks;

[0205] In this embodiment, the web crawler module 4 needs to set the crawling frequency according to the robots.txt rules of the website, follow the crawling etiquette, and avoid being blocked. The planning of the crawler task should fully consider the performance of the system and the characteristics of the target website.

[0206] Step S5, processing the crawled text information through the text processing engine 5;

[0207] In this embodiment, the text processing engine 5 uses libraries such as Spacy to perform text parsing, including word segmentation, part-of-speech tagging, and removal of stop words. By using advanced text processing technology, the system can more accurately extract useful information from the captured HTML content. In this embodiment, by executing step S5, the unstructured text data is converted into structured information to prepare for subsequent processing.

[0208] Step S6, designing hot word matching and filtering module 6;

[0209] In this embodiment, in the design process of the hot word matching and filtering module 6, the hot word library and the word vector model are combined to match and filter the text. The weight or priority of the hot word is used to filter out the information related to the hot topic. In this embodiment, by executing step S6, it is ensured that the target content is filtered out from a large amount of text information, and the quality and relevance of the information are improved.

[0210] Step S7, storing and retrieving the processed data;

[0211] In this embodiment, a database such as Elasticsearch is selected to establish an index structure to support efficient data storage and retrieval. The database is responsible for storing processed information to ensure the reliability and retrievability of the data. Reasonable index design can improve retrieval efficiency and provide better performance for the system.

[0212] Step S8, ensuring the real-time nature of user information acquisition through the real-time update and feedback module 8;

[0213] In this embodiment, the real-time update and feedback module uses technologies such as WebSocket to ensure that information is fed back to the user interface in real time. Through scheduled tasks or event-driven, the real-time crawled information and changes in hot topics are immediately delivered to the user. This step is related to the user experience, ensuring that users can obtain the latest data in the first place, and enhancing the real-time performance of the system.

[0214] Step S9: various problems that occur during the crawling process are handled through the exception handling and optimization module;

[0215] In this embodiment, an exception handling and optimization module 9 is designed to handle abnormal situations that may occur during the crawling process, including but not limited to: network timeout, website structure changes. An automated exception handling mechanism and regular optimization strategies are implemented to adjust algorithms and parameters based on feedback results to improve the stability and efficiency of the system.

[0216] Step S10: Complete testing and deployment.

[0217] After the above steps S1 to S9 are completed, comprehensive testing is carried out, including but not limited to: unit testing, integration testing and system testing, to ensure the normal operation of each module of the system. After the test passes, deployment is carried out to put the system into actual operation. Before deployment, it is necessary to ensure the security, reliability and performance of the system, follow the appropriate deployment process, and monitor the operating status of the system to ensure that the system can operate stably for a long time.

[0218] Deep learning hot word detection: Use deep learning models (such as BERT) to detect hot words and accurately capture user focus. Compared with traditional rule-based or keyword matching methods, this improves the accuracy of hot word detection and makes the system more intelligent.

[0219] Adaptive scheduling algorithm: The adaptive scheduling algorithm is implemented through the scheduling and controller module, which adjusts the web crawler tasks in real time according to the hot words input by the user, and flexibly adapts to the changes in user interests. This is fundamentally different from the traditional fixed scheduling strategy and improves the real-time performance of the system.

[0220] Distributed crawler technology: Using frameworks such as Scrapy combined with distributed crawler technology to achieve multi-node parallel crawling, improving the efficiency of information collection. Compared with the traditional single-node crawler system, this enhances the horizontal scalability and crawling degree of the system.

[0221] Hot word matching and filtering: Design a hot word matching and filtering module, use the hot word library and word vector model, combined with the weight and priority of hot words, to more accurately match and filter information. Compared with the traditional rule-based filtering method, this improves the accuracy and relevance of information filtering.

[0222] Real-time update and feedback: Using WebSocket and other technologies to achieve real-time update and feedback, users can instantly obtain the latest crawled information and changes in hot topics, enhancing the real-time nature of the system. This is more efficient than traditional manual refresh or scheduled refresh methods.

[0223] In summary, the present invention adopts a hot word detection deep learning model to accurately identify the hot words input by the user. By improving the training and application of the model, the user's intention can be understood more accurately. The present invention designs an adaptive scheduling algorithm for real-time adjustment of network crawler tasks. The algorithm flexibly adapts to changes in hot words input by users, thereby improving the real-time performance and user satisfaction of the system. The present invention uses distributed crawler technology to achieve multi-node parallel crawling, thereby improving the efficiency of information collection. This innovation in technical implementation makes the system more scalable and efficient. The present invention uses a hot word matching and filtering module, combined with a hot word library and a word vector model, to achieve more accurate information filtering.

[0224] Through the hot word detection of the present invention, the web crawler can better understand the crawled data, provide more accurate data analysis results, and optimize the effect of data analysis. For example, the user's interest trend can be understood through hot word detection to provide support for the recommendation system. For enterprises, hot word detection can help understand market dynamics and consumer demand and provide a basis for decision-making. For example, e-commerce websites can understand the most popular products through hot word detection and then adjust their sales strategies.

[0225] By detecting hot words, the web crawler of the present invention can understand the topics that users are currently most concerned about, so as to give priority to crawling web pages related to these topics, thereby improving the efficiency and practicality of crawling and improving the efficiency of crawling.

[0226] The present invention can help web crawlers understand current hot spots in real time through hot word detection, adjust crawling strategies in time, capture the latest hot information, and update crawled content in real time.

[0227] The hot word detection operation of the present invention can help web crawlers screen out valuable information, avoid crawling a large amount of irrelevant or low-quality data, and improve the quality of crawled data.

[0228] The present invention realizes real-time update and feedback mechanism through WebSocket technology, so that users can obtain the latest information instantly. This has a significant improvement in user experience and improves the real-time nature of information.

[0229] The present invention solves the technical problems existing in the prior art of low crawling efficiency, slow real-time adjustment of crawling strategies and low quality of crawled data.

[0230] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for implementing a real-time web crawler framework based on hot word detection, characterized in that: The method comprises: S1. Design a user interface using a preset front-end framework, and collect user input information on the user interface; S2. Using a preset hot word detection module, obtaining and training a preset deep learning model, performing hot word detection on the user input information to obtain hot words, and calculating the heat score of each word through the hot word detection module; S3, using the scheduling and controller for the hot word detection module to dynamically adjust the web crawler task according to the hot words; S4. Using a pre-set web crawler module and distributed crawler technology, perform multi-node parallel crawling to crawl a target website, plan the web crawler task according to system performance information and the target website, crawl text information, and aggregate the text information to a central storage device; S5, using a text processing engine and a preset parsing library tool to perform text processing operations on the text information to obtain text structured information; S6, using the hot words to build a hot word library, and designing a hot word matching and filtering module according to the hot word library and the preset word vector model to match and filter the text structured information; according to the corresponding weights of the hot words, filtering to obtain hot topic related information as the target content; S7, selecting a preset database to establish an index structure for data storage and retrieval operations, and storing and retrieving the processed data; S8, using a preset real-time update and feedback module and a preset network protocol to perform an information interconnection update operation and deliver the currently updated target content to the user; S9. Use the preset exception handling and optimization module to automate the exception handling mechanism, obtain and handle crawling process exceptions based on user feedback results.

2. The method for implementing a real-time web crawler framework based on hot word detection according to claim 1 is characterized in that: In S1, the preset front-end framework includes: React.

3. The method for implementing a real-time web crawler framework based on hot word detection according to claim 1 is characterized in that: In S2, the deep learning model includes: a BERT model, which obtains context information and diversity information to perform model training operations on the BERT model to extract hot words and assign hot word corresponding weights to each of the hot words; and performs user demand understanding and feedback operations based on the hot word corresponding weights.

4. The method for implementing a real-time web crawler framework based on hot word detection according to claim 1 is characterized in that: In S2, the heat score is obtained by processing using the following logic: Score(w i )=Attention(w i ,C)×Frequency(w i ) In the formula, Score(w i ) represents word w i The score of Attention(w i ,C) represents word w i The importance weight in context C is calculated using the attention mechanism, Frequency(w i ) represents word w i The frequency of occurrence; The following logic is used to calculate the attention weight based on BERT's self-attention mechanism: In the formula, Q is the query vector Query, K is the key vector Key, V is the value vector Value, d k is the dimension of the key vector.

5. The method for implementing a real-time web crawler framework based on hot word detection according to claim 1 is characterized in that: In S3, the priority, user demand, and system resources of the hot words are obtained and processed to allocate the frequency and scale of the web crawler tasks using an adaptive scheduling algorithm.

6. The method for implementing a real-time web crawler framework based on hot word detection according to claim 5 is characterized in that: The adaptive scheduling algorithm includes: S31, performing task frequency allocation: dynamically adjusting the crawling frequency of the web crawler task according to the priority of the hot words and system resources using the following logic: In the formula, F i represents the crawling frequency of the i-th node, F max Indicates the preset maximum crawling frequency, P i Indicates the priority of hot words, L i Represents the current load of the i-th node, R i Represents the response rate of the target website to the i-th node. S32, adjusting the scale of the web crawler task: dynamically adjusting the scale of the web crawler task according to the system resources and the status of the target website using the following logic: In the formula, S i represents the scale of crawling tasks assigned to the i-th node, S max Indicates the preset maximum task size, C i represents the available capacity or performance index of the i-th node, and N represents the total number of crawler nodes.

7. The method for implementing a real-time web crawler framework based on hot word detection according to claim 1, characterized in that: In S4, the web crawler module includes but is not limited to: Scrapy crawler framework; managing the task queue through the scheduling and controller, and regularly starting and stopping the crawling task operation; setting the crawling frequency according to the website robots.txt rules.

8. The method for implementing a real-time web crawler framework based on hot word detection according to claim 1, characterized in that: The S4 includes: S41, planning the web crawler task, wherein the task allocation operation is performed using the following logic: Where W i represents the weight assigned to the i-th node. i Indicates the priority of the i-th task. i represents the available capacity or performance index of the i-th node; S42. Use the following logic to obtain the crawling frequency through multi-node parallel crawling: In the formula, F i represents the crawling frequency of the i-th node, F max Indicates the preset maximum crawling frequency, L i represents the current load of the i-th node, R i Represents the response rate of the target website to the i-th node.

9. The method for implementing a real-time web crawler framework based on hot word detection according to claim 1, characterized in that: In S9, the algorithm and parameters are adjusted according to the user feedback result.

10. A real-time web crawler framework implementation system based on hot word detection, characterized in that: The system comprises: A user interface module is used to design a user interface using a preset front-end framework and collect user input information on the user interface; A hot word detection module, used to acquire and train a preset deep learning model, perform hot word detection on the user input information to obtain hot words, and the hot word detection module is connected to the user interface module; A scheduling and controller, used to dynamically adjust the web crawler task according to the hot words, the scheduling and controller is connected to the hot word detection module; A web crawler module, used to perform multi-node parallel crawling with distributed crawler technology, crawl the target website, plan the web crawler task according to the system performance information and the target website, so as to crawl text information, and the web crawler module is connected to the scheduling and controller; A text processing engine, used to perform text processing operations on the text information using a preset parsing library tool to obtain text structured information, the text processing engine being connected to the web crawler module; A hot word matching and filtering module is used to construct a hot word library using the hot words, and to design a hot word matching and filtering module based on the hot word library and the preset word vector model to match and filter the text structured information; according to the hot word corresponding weight, filter to obtain hot topic related information as the target content, and the hot word matching and filtering module is connected to the text processing engine; A database memory is used to select a preset database to establish an index structure for data storage and retrieval operations, and to store and retrieve processed data. The database memory is connected to the hot word matching and filtering module; A real-time update and feedback module, used to utilize a preset network protocol to perform an information interconnection update operation and transmit the currently updated target content to the user, wherein the real-time update and feedback module is connected to the database storage; The exception handling and optimization module is used to utilize the automated exception handling mechanism to obtain and handle crawling process exceptions based on user feedback results. The exception handling and optimization module is connected to the real-time update and feedback module.

Citation Information

Cited By

  • Cooking robot menu management method and system based on big data and medium

    CN119692501A