Information acquisition scheduling system based on dynamic weight optimization

Abstract

The invention relates to the technical field of network security intelligence, in particular to an information acquisition scheduling system based on dynamic weight optimization, which comprises a data monitoring and acquisition module, a scheduling queue generation module, a content analysis and feedback module, a crawler execution module and a content analysis and feedback module. Collecting user interaction data, content change rate and the like of a target intelligence source through a data monitoring and collecting module; the dynamic weight calculation module calculates an activeness score, a historical hit rate and a low-frequency protection gain according to the data and fuses the data to obtain a comprehensive final weight; a scheduling queue generation module builds a main queue and a low-frequency guarantee pool according to weight sorting, and distributes differential grabbing parameters; the crawler execution module performs distributed parallel grabbing; the content analysis and feedback module judges validity and feeds back adjustment parameters to form a closed loop; according to the method, the capture coverage rate and timeliness of the low-frequency high-value intelligence source can be improved, the utilization rate of system resources is optimized, and the requirements of network security intelligence for information comprehensiveness, timeliness and accuracy are met.

Description

Technical Field

[0001] This invention belongs to the field of network security intelligence technology, specifically relating to an information collection and scheduling system based on dynamic weight optimization. Background Technology

[0002] In the field of cybersecurity intelligence, web crawlers, as a key information gathering tool, are widely used to acquire important information such as threat intelligence, public opinion trends, and vulnerability disclosures in real time. Current mainstream information crawling strategies mainly include prioritizing large websites, periodic crawling based on historical update frequency, and traditional methods such as depth-first or breadth-first traversal. While these strategies can ensure a certain level of data timeliness when dealing with high-frequency update sources such as news sites and official blogs, they are significantly inadequate when handling low-frequency update sources such as replies in security forum posts, forum discussion groups, and technical community comments. It is important to note that while these low-frequency update sources typically exhibit characteristics such as sparse information dissemination, deep interaction structures, and strong semantic implicitness, they are crucial sources for acquiring intelligence on Advanced Persistent Threat (APT) activities, vulnerability exploitation dynamics, and underground black market activities. However, traditional web crawlers generally lack the ability to keenly perceive and schedule low-frequency, high-value information, making it difficult to achieve timely capture and comprehensive coverage of hidden threat clues with limited resources. Ultimately, they fail to meet the high standards of comprehensiveness, timeliness, and accuracy required for cybersecurity intelligence. Existing technologies have the following main shortcomings when handling low-frequency update sources: (1) Insufficient coverage of low-frequency intelligence update sources: Existing crawler scheduling strategies mostly adopt the mechanism of prioritizing large sites or based on historical access records. The core optimization direction is concentrated on high-frequency update sources such as news sites and portal websites. However, for low-frequency intelligence sources such as security forums and vulnerability disclosure platforms that contain high-value threat intelligence, existing methods lack targeted scheduling mechanisms, making it difficult to identify and capture key threat information that suddenly appears in a timely manner (such as vulnerability exploit code, attack event discussions, etc.), which leads to the omission of important intelligence and seriously affects the comprehensiveness of security threat perception.

[0003] (2) Lack of dynamic adjustment mechanism in the face of changes in intelligence sources: Most existing technologies rely on static crawling strategies, such as crawling at fixed time intervals and link sorting based on PageRank algorithm, and lack the ability to dynamically optimize according to the real-time activity level and content value of intelligence sources; especially when sudden security events (such as vulnerability exposure and escalation of threat activities) cause a sharp change in the activity level of intelligence sources, the system cannot adaptively adjust the crawling priority and depth, which can easily lead to unreasonable resource allocation and make it difficult to ensure the timeliness and accuracy of key intelligence acquisition at the same time.

[0004] (3) Insufficient response to dynamic signals such as user interaction: The analysis logic of existing topic crawlers mostly relies on the relevance of the text content or link structure of the intelligence source, and fails to effectively incorporate dynamic value signals such as user interaction behavior (such as the number of comments, the number of shares, and user participation) into the evaluation system. In practical applications, some long-term static intelligence sources may become hot topics again due to new interactive behaviors such as new replies, but traditional crawlers cannot perceive such interactive signals, and there is usually a crawling delay of several hours or even longer, ultimately missing the best intelligence capture window.

[0005] (4) Information omission and insufficient acquisition of deep information: Low-frequency intelligence sources generally have the characteristics of deep content nesting, scattered information and complex presentation structure, such as multi-layered reply discussions, code areas that need to be manually expanded, and cross-link related content; traditional crawler crawling methods are prone to two types of problems: on the one hand, when the main content of the page does not change significantly, it is easy to misjudge that it is unrelated to the target topic, thus missing the potential value information in the page; on the other hand, the ability to penetrate deep links is insufficient, and it is impossible to effectively extract nested or aggregated content; at the same time, the existing technology lacks a deep crawling strategy that coordinates with dynamic scheduling, which further reduces the overall crawling efficiency.

[0006] In view of this, the present invention is hereby proposed. Summary of the Invention

[0007] To address the aforementioned technical problems in existing technologies, this invention provides an information collection and scheduling system based on dynamic weight optimization, which solves the problems of insufficient coverage of low-frequency intelligence sources, static and rigid scheduling strategies, high latency in capturing hot information, and low system resource utilization in traditional crawlers.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows: An information collection and scheduling system based on dynamic weight optimization includes: Data monitoring and collection module: used to collect multi-dimensional data from target intelligence sources and output user interaction data, content change rate, keyword matching degree and historical update frequency to the dynamic weight calculation module; Dynamic weight calculation module: It is used to receive multi-dimensional data output by the data monitoring and collection module, calculate the activity score, historical hit rate and low-frequency intelligence source protection gain of the target intelligence source in sequence, and then calculate the comprehensive final weight based on the activity score, historical hit rate and low-frequency intelligence source protection gain, and output the comprehensive final weight to the scheduling queue generation module. The scheduling queue generation module is used to sort all intelligence sources to be crawled based on the comprehensive final weight, build a main queue and a low-frequency guarantee pool, configure differentiated crawling parameters according to the interval of the comprehensive final weight, generate a crawling task queue and output it to the crawler execution module. The crawler execution module is used to receive the crawling task queue, perform distributed parallel crawling of the intelligence source content, and output the crawled content to the content analysis and feedback module. Content analysis and feedback module: used to determine the validity of the crawled content and feed the determination result back to the dynamic weight calculation module to adjust the activity score and historical hit rate of the target intelligence source.

[0009] Furthermore, the multi-dimensional data collected by the data monitoring and acquisition module specifically includes: User interaction data: Obtain the number of comments, likes, shares, and visits within a specified time window by parsing the webpage structure or calling the API; Content change rate: Calculated by comparing the DOM tree signatures of the current page and the previous version of the page; Keyword matching degree: calculated based on a pre-set list of topic keywords and keyword weights; Historical update frequency: Calculated based on the content change rate of the target intelligence source, used to distinguish between high-frequency update sources and low-frequency update sources.

[0010] Furthermore, the formula for calculating the activity score by the dynamic weight calculation module is as follows:

[0011] in, To score activity level, This represents the sum of normalized user interaction data. For content change rate, The keyword matching score is used to determine the keyword matching score. for The weighting coefficients, This is the time decay coefficient.

[0012] Furthermore, the dynamic weight calculation module calculates the historical hit rate in the following way: Within the preset number of capture cycles, the historical hit rate It is equal to the ratio of the number of relevant pages crawled from that intelligence source to the total number of pages crawled from that intelligence source. The specific formula is: .

[0013] Furthermore, the dynamic weight calculation module calculates the low-frequency intelligence source protection gain. The formula is:

[0014] in, The historical update frequency of the target intelligence source. For historical update frequency high threshold, The historical update frequency threshold is low.

[0015] Furthermore, the dynamic weight calculation module calculates the final comprehensive weight. The formula is:

[0016] in, To score activity level, For historical accuracy, To protect the gain of low-frequency intelligence sources, These are the weighting coefficients.

[0017] Furthermore, the differentiated fetching parameters configured in the scheduling queue generation module are as follows: When the final weight is considered At that time, the crawling frequency is high, the crawling depth is 3 layers, and the priority is the highest. When the final weight is considered At that time, the crawling frequency was medium, the crawling depth was 2 layers, and the priority was high; When the final weight is considered At that time, the crawling frequency was low, the crawling depth was 1 layer, and the priority was medium. when At that time, the crawling frequency was extremely low, the crawling depth was 1 layer, and the priority was low.

[0018] Furthermore, when constructing the main queue and the low-frequency guarantee pool, the scheduling queue generation module divides the total crawling resources into main queue resources and low-frequency guarantee pool resources, and the main queue is determined according to the final comprehensive weight. Generate tasks in descending order and allocate main queue resources; Low-frequency protection pool protects gain based on low-frequency intelligence source Sampling is used to ensure that at least a preset minimum number of low-frequency update sources are captured in each capture cycle.

[0019] Furthermore, the crawler execution module adopts distributed technology, which executes crawling tasks in parallel through multiple computing nodes.

[0020] Furthermore, the content analysis and feedback module determines validity in the following way: The text features of the crawled content are compared with those of the historical versions of the intelligence source. The text features include hash value, structural differences and similarity. If a change in content is detected, it is determined to be a valid update. If the content is completely identical, it is determined to be an invalid crawl. When the content analysis and feedback module feeds the judgment result back to the dynamic weight calculation module, if it is a valid update, it will adjust both the activity score and the historical hit rate; if it is an invalid crawl, it will only adjust the activity score.

[0021] Compared with existing technologies, the information collection and scheduling system based on dynamic weight optimization provided by this invention includes: a data monitoring and collection module, which collects user interaction data, content change rate, keyword matching degree, and historical update frequency of target intelligence sources; a dynamic weight calculation module, which calculates activity score, historical hit rate, and low-frequency intelligence source protection gain based on the above data, and then integrates them to obtain a comprehensive final weight; a scheduling queue generation module, which constructs a main queue and a low-frequency protection pool according to the comprehensive final weight and configures differentiated crawling parameters; a crawler execution module, which uses distributed technology to execute crawling tasks in parallel; and a content analysis and feedback module, which determines the validity of the crawled content and feeds it back to the dynamic weight calculation module, adjusting relevant parameters to form a closed-loop optimization. This invention can achieve accurate quantitative evaluation of intelligence source value, improve the coverage and timeliness of crawling low-frequency high-value intelligence sources, and optimize system resource utilization, meeting the needs of network security intelligence for comprehensiveness, timeliness, and accuracy. Attached Figure Description

[0022] Figure 1 This is an architecture diagram of an information collection and scheduling system provided in an embodiment of the present invention; Figure 2 The flowchart illustrates the usage of the information collection and scheduling system provided in this embodiment of the invention. Detailed Implementation

[0023] The technical solution of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0024] It should be noted that, unless otherwise specifically stated, the relative arrangement and numerical expressions of the components and steps described in these embodiments should not be construed as limiting the scope of the invention.

[0025] The following description of exemplary embodiments is merely illustrative and is not intended to limit the invention or its application or use in any way. Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail herein, but where applicable, such techniques, methods, and apparatus should be considered part of this specification.

[0026] Example 1 See Figure 1 , Figure 1 This is an architecture diagram of an information collection and scheduling system based on dynamic weight optimization proposed in this invention, which may specifically include: M1, Data Monitoring and Acquisition Module: Used for multi-dimensional data acquisition from the target intelligence source, outputting user interaction data, content change rate, keyword matching degree, and historical update frequency to the dynamic weight calculation module; primarily responsible for multi-dimensional monitoring and information acquisition from the target intelligence source, providing real-time, raw data input for the subsequent dynamic weight calculation module. It captures the following raw data from the target intelligence source in real-time or periodically: User interaction data: By parsing the webpage structure or calling APIs, data such as the number of comments, likes, shares, and visits within a specified time window can be obtained; Content change rate: The degree of content change is calculated by comparing the DOM tree signatures of the current page with those of the previous version of the page; Keyword matching score: Calculate the keyword matching score of the page content using a pre-set list of topic keywords and their weights; Historical update frequency: Based on the content change rate of the intelligence source, the historical update frequency of the intelligence source is calculated to distinguish whether the intelligence source is a high-frequency update source or a low-frequency update source.

[0027] M2, Dynamic Weight Calculation Module: Used to receive multi-dimensional data output by the data monitoring and acquisition module, sequentially calculate the activity score, historical hit rate and low-frequency intelligence source protection gain of the target intelligence source, and then calculate the comprehensive final weight based on the activity score, historical hit rate and low-frequency intelligence source protection gain, and output the comprehensive final weight to the scheduling queue generation module. This module receives raw data from the data monitoring and acquisition module. Through the algorithm model designed in this patent, it calculates a dynamically changing comprehensive weight value for the target intelligence source to be captured. This value directly determines the capture priority and capture depth of the intelligence source.

[0028] M21. Calculate the target intelligence source activity score using the following formula:

[0029] in, To score activity level, This represents the sum of normalized user interaction data. For content change rate, , The keyword matching score is used to determine the keyword matching score. for The weighting coefficients (adjustable, for example, can be set to...) ), The time decay coefficient, The time since the last capture, The decay coefficient controls the decrease in activity over time (smaller values ​​are used for high-frequency intelligence sources to extend memory, and larger values ​​are used for low-frequency intelligence sources to reduce the impact of obsolescence).

[0030] M22. Calculate historical hit rate: Within a preset number of capture cycles, the historical hit rate... It is equal to the ratio of the number of relevant pages crawled from that intelligence source to the total number of pages crawled from that intelligence source. The specific formula is:

[0031] M23, Low-frequency source protection gain calculation: Calculate the historical update frequency of each intelligence source. Quantified into continuous protection values Intelligence sources with lower update frequency The higher the value, the greater the protective gain in calculating the overall weight. The specific formula is as follows:

[0032] in, The historical update frequency of the target intelligence source. For historical update frequency high threshold, The historical update frequency threshold is low.

[0033] M24, the dynamic weight calculation module calculates the final comprehensive weight. The formula is:

[0034] in, To score activity level, For historical accuracy, To protect the gain of low-frequency intelligence sources, These are the weighting coefficients. .

[0035] M3, Scheduling Queue Generation Module: Used to sort all intelligence sources to be crawled based on the comprehensive final weight, construct the main queue and low-frequency guarantee pool, configure differentiated crawling parameters according to the interval of the comprehensive final weight, generate a crawling task queue and output it to the crawler execution module; The module divides the comprehensive weight value into four intervals, sets different crawling frequencies, crawling depths and priorities according to different intervals, and generates the final crawling task queue to achieve optimal resource utilization.

[0036] Specifically, the scheduling queue generation module is based on the comprehensive final weight output by the dynamic weight calculation module. All intelligence sources to be captured are sorted, taking into account both high-frequency and low-frequency intelligence sources, and a main queue is constructed. ), screening low-frequency protection pool ( And generate the final crawling task list according to the resource allocation ratio, and at the same time according to Different crawling frequencies, crawling depths, and priorities are set for different intervals. Specifically, this includes: M31, Priority URL Queue Construction: Based on the final weight All intelligence sources to be captured are sorted and formed into a priority queue. Meanwhile, to ensure continuous coverage of low-frequency sources, a strategy of allocating minimum resources is adopted: Allocate total crawling resources to the main queue With low-frequency protection pool , according to Generate tasks in descending order and allocate primary shares accordingly; according to Sampling ensures that at least one sample is captured in each cycle. A low-frequency source is needed to avoid long-term omissions.

[0037] M32. Configure crawling parameters: Dynamically configure crawling parameters based on the range of weight values. When the final weight is considered At that time, the crawling frequency is high (e.g., every 15 minutes), the crawling depth is 3 layers, and the priority is the highest. When the final weight is considered At that time, the crawling frequency is medium (e.g., per hour), the crawling depth is 2 layers, and the priority is high; When the final weight is considered At that time, the crawling frequency is low (e.g., every 6 hours), the crawling depth is 1 layer, and the priority is medium. when At that time, the crawling frequency is extremely low (e.g., daily), the crawling depth is 1 level (only the current page), and the priority is low.

[0038] M4, Crawler Execution Module: Used to receive the crawling task queue, perform distributed parallel crawling of intelligence source content, and output the crawled content to the content analysis and feedback module; This module generates fetching tasks according to the task instructions in the scheduling queue and performs the actual intelligence source fetching operation. It employs distributed technology to achieve parallel fetching of intelligence source content, executing fetching tasks in parallel across multiple computing nodes. This fully utilizes the computing resources of each node to improve the content fetching rate. M5, Content Analysis and Feedback Module: Used to determine the validity of the crawled content and feed the determination result back to the Dynamic Weight Calculation Module to adjust the activity score and historical hit rate of the target intelligence source.

[0039] This module determines the validity of page content crawled by the crawler execution module and feeds the result back to the scheduling queue generation module. This is used to adjust the weight values ​​of the target intelligence sources, forming a closed-loop adaptive optimization. Specifically, it includes: M51. Content Relevance Determination: By comparing the text features (hash value, structural differences, or similarity) of the crawled content with those of historical versions, if a change in page content is detected, it is considered a valid update; if the content is completely identical, it is considered an invalid crawl.

[0040] M52. Feedback and Adjustment: Feedback is given to the scheduling module based on the content update status to adjust the activity score and historical hit rate of the intelligence source, and to adjust the priority of the next round of crawling.

[0041] M53. System Workflow: The system uses the intelligence source seed database as its entry point, sequentially proceeding through the data monitoring and collection module, dynamic weight calculation module, scheduling queue generation module, crawler execution module, and content analysis and feedback module. The content analysis module judges the crawling results: if relevant, it stores and updates the historical hit rate and metadata; if irrelevant, it adjusts the crawling priority according to the feedback strategy and proceeds to the next round of tasks. The entire process forms a closed loop, continuously iterating and optimizing the crawling strategy and parameters.

[0042] See Figure 2 , Figure 2 The flowchart of the information collection and scheduling system based on dynamic weight optimization provided by this invention includes the following steps: S1. Data Collection and Monitoring: The data monitoring and collection module polls or listens to target intelligence sources from the intelligence source seed library, collecting multi-dimensional indicators such as user interaction data, content change rate, keyword matching degree, and historical update rate, providing basic data for weight calculation.

[0043] S2. Dynamic Weight Calculation: The dynamic weight calculation module receives the collected raw data, calculates the activity level of the target intelligence source, historical hit rate, and low-frequency intelligence source protection gain in sequence, and generates the final quantitative weight value to measure the current capture priority.

[0044] S3. Generating the Scheduling Queue: The scheduling queue generation module sorts all intelligence sources to be captured based on their comprehensive weights. Combining high-frequency and low-frequency target intelligence sources, it constructs a main queue and filters the low-frequency support pool, generating the final capture task list according to resource allocation. Simultaneously, it sets different capture frequencies, depths, and priorities based on differences in weight ranges.

[0045] S4. Crawling Task Execution: The crawler execution module receives crawling tasks from the scheduling queue and performs distributed parallel crawling according to priority to ensure execution efficiency and coverage.

[0046] S5. Content Analysis and Feedback: The content analysis module determines the validity of the crawled content. If relevant, it stores and updates the historical hit rate, and feeds the updated result back to the weight calculation module. If irrelevant, it only adjusts the weight of the metric without updating the historical hit rate. The feedback result will affect the next round of weight calculation, forming an adaptive closed-loop optimization.

[0047] In summary, the present invention has the following advantages: 1. By integrating multi-dimensional signals such as real-time user interaction, content change rate, and keyword matching degree, and combining historical hit rate feedback and low-frequency intelligence source protection mechanism, a precise and quantifiable comprehensive weight of intelligence source value is generated. The crawling priority can be arranged according to the comprehensive weight, which optimizes the problems of traditional crawlers such as delayed response to low-frequency high-value information and unreasonable resource allocation. 2. By introducing a low-frequency protection gain mechanism and an independent protection pool, it is possible to effectively identify and capture pages that have a low update frequency but contain high-value information; 3. By establishing an intelligent comprehensive weighted capture parameter mapping mechanism, precise and differentiated capture and scheduling of different valuable intelligence sources has been achieved.

[0048] The above specific embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An information collection and scheduling system based on dynamic weight optimization, characterized in that, include: Data monitoring and collection module: used to collect multi-dimensional data from target intelligence sources and output user interaction data, content change rate, keyword matching degree and historical update frequency to the dynamic weight calculation module; Dynamic weight calculation module: It is used to receive multi-dimensional data output by the data monitoring and collection module, calculate the activity score, historical hit rate and low-frequency intelligence source protection gain of the target intelligence source in sequence, and then calculate the comprehensive final weight based on the activity score, historical hit rate and low-frequency intelligence source protection gain, and output the comprehensive final weight to the scheduling queue generation module. The scheduling queue generation module is used to sort all intelligence sources to be crawled based on the comprehensive final weight, build a main queue and a low-frequency guarantee pool, configure differentiated crawling parameters according to the interval of the comprehensive final weight, generate a crawling task queue and output it to the crawler execution module. The crawler execution module is used to receive the crawling task queue, perform distributed parallel crawling of intelligence source content, and output the crawled content to the content analysis and feedback module. Content analysis and feedback module: used to determine the validity of the crawled content and feed the determination result back to the dynamic weight calculation module to adjust the activity score and historical hit rate of the target intelligence source.

2. The information collection and scheduling system based on dynamic weight optimization according to claim 1, characterized in that, The multi-dimensional data collected by the data monitoring and acquisition module specifically includes: User interaction data: Obtain the number of comments, likes, shares, and visits within a specified time window by parsing the webpage structure or calling the API; Content change rate: Calculated by comparing the DOM tree signatures of the current page and the previous version of the page; Keyword matching degree: calculated based on a pre-set list of topic keywords and keyword weights; Historical update frequency: Calculated based on the content change rate of the target intelligence source, used to distinguish between high-frequency update sources and low-frequency update sources.

3. The information collection and scheduling system based on dynamic weight optimization according to claim 1, characterized in that, The formula for calculating the activity score by the dynamic weight calculation module is as follows: in, To score activity level, This represents the sum of normalized user interaction data. For content change rate, The keyword matching score is used to determine the keyword matching score. for The weighting coefficients, This is the time decay coefficient.

4. The information collection and scheduling system based on dynamic weight optimization according to claim 1, characterized in that, The dynamic weight calculation module calculates the historical hit rate in the following way: Within the preset number of capture cycles, the historical hit rate It is equal to the ratio of the number of relevant pages crawled from that intelligence source to the total number of pages crawled from that intelligence source. The specific formula is: .

5. The information collection and scheduling system based on dynamic weight optimization according to claim 1, characterized in that, The dynamic weight calculation module calculates the low-frequency intelligence source protection gain. The formula is: in, The historical update frequency of the target intelligence source. For historical update frequency high threshold, The historical update frequency threshold is low.

6. The information collection and scheduling system based on dynamic weight optimization according to claim 1, characterized in that, The dynamic weight calculation module calculates the final comprehensive weight. The formula is: in, To score activity level, For historical accuracy, To protect the gain of low-frequency intelligence sources, These are the weighting coefficients.

7. The information collection and scheduling system based on dynamic weight optimization according to claim 6, characterized in that, The differentiated fetching parameters configured in the scheduling queue generation module are as follows: When the final weight is considered At that time, the crawling frequency is high, the crawling depth is 3 layers, and the priority is the highest. When the final weight is considered At that time, the crawling frequency was medium, the crawling depth was 2 layers, and the priority was high; When the final weight is considered At that time, the crawling frequency was low, the crawling depth was 1 layer, and the priority was medium. when At that time, the crawling frequency was extremely low, the crawling depth was 1 layer, and the priority was low.

8. The information collection and scheduling system based on dynamic weight optimization according to claim 6, characterized in that, When the scheduling queue generation module constructs the main queue and the low-frequency guarantee pool, it divides the total crawling resources into main queue resources and low-frequency guarantee pool resources. The main queue is determined according to the final comprehensive weight. Generate tasks in descending order and allocate main queue resources; Low-frequency protection pool protects gain based on low-frequency intelligence source Sampling is used to ensure that at least a preset minimum number of low-frequency update sources are captured in each capture cycle.

9. The information collection and scheduling system based on dynamic weight optimization according to claim 1, characterized in that, The crawler execution module adopts distributed technology, which executes crawling tasks in parallel through multiple computing nodes.

10. The information collection and scheduling system based on dynamic weight optimization according to claim 1, characterized in that, The content analysis and feedback module determines validity in the following way: The text features of the crawled content are compared with those of the historical versions of the intelligence source. The text features include hash value, structural differences and similarity. If a change in content is detected, it is determined to be a valid update. If the content is completely identical, it is determined to be an invalid crawl. When the content analysis and feedback module feeds the judgment result back to the dynamic weight calculation module, if it is a valid update, it will adjust both the activity score and the historical hit rate; if it is an invalid crawl, it will only adjust the activity score.

Images