Webpage information processing method and device based on intelligent agent, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses an agent-based webpage information processing method, device and equipment and a medium. And distributing the data to a capturing agent to obtain webpage original data. According to the method, model context information is generated by extracting webpage source information and release time information, page structure representation is constructed, key information nodes are positioned, structured data are generated, then multi-scene decision data are generated in combination with the structured data and the model context information in the analysis process, and a decision result is output through a decision reasoning module. By constructing a uniform data processing flow, webpage capture, context generation, structured analysis, analysis processing and decision reasoning are organically linked, efficient acquisition and intelligent processing of webpage information are realized, and timeliness and accuracy of decision are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an agent-based web page information processing method, device, equipment and storage medium. Background Art

[0002] In the fintech sector, companies often need to extract key content from massive web pages and quickly generate decision-making information when faced with high-frequency, sensitive information such as policy changes and macroeconomic events. However, existing information processing technologies lack the ability to identify and utilize the source and update timeliness of web pages in the allocation mechanism of web crawling tasks, resulting in uneven resource allocation, delayed crawling responses, and impacting the efficiency of obtaining high-value information. In addition, the dynamic loading characteristics of web page content are becoming increasingly complex, making it difficult for current systems to stably obtain complete web page data. In particular, significant omissions occur when processing script-rendered content and asynchronous response data, which in turn affects data integrity and the basis for subsequent analysis.

[0003] In the healthcare sector, medical policies, drug updates, and early warning information are often published via dynamic web pages. Existing systems lack contextual awareness of raw web page data, failing to automatically extract source and publication time information with business-critical value from the content. Consequently, subsequent analysis cannot accurately assess the reliability and timeliness of this information. Furthermore, when parsing web page structures to extract key information, current technology struggles to apply matching parsing templates to web pages from different sources. The lack of a unified parsing strategy invocation mechanism results in unstable extraction results and poor structured data quality, impacting the availability of downstream business systems.

[0004] In the general field of data processing, although existing systems can support the extraction of structured information, they still lack the ability to analyze and process the extraction results to generate multi-scenario decision-making data. In particular, in the absence of a unified contextual information driver, it is difficult to form a stable and reliable basis for decision-making. In the current process, the analysis stage of structured data is often decoupled from the contextual information, failing to integrate key metadata such as source and timeliness, limiting the ability to make accurate decisions in multiple business scenarios. In summary, existing technologies have insufficient capabilities in multiple key nodes such as task scheduling, web page raw data acquisition, contextual information extraction, structured data generation, and multi-scenario decision analysis, making it difficult to build a complete closed loop from web page data to decision results. Summary of the Invention

[0005] The main purpose of the present invention is to provide a web page information processing method, device, equipment and storage medium based on an intelligent agent, aiming to solve the technical problems that the existing technology cannot schedule crawling tasks based on the source and timeliness characteristics of the web page, and lacks a unified model context information construction mechanism, resulting in the disconnection between structured analysis and decision processing flow, and the inability to achieve integrated intelligent decision support.

[0006] To achieve the above objectives, the present invention provides an agent-based web page information processing method, comprising:

[0007] Configure scheduling agents, grasping agents, parsing agents, analysis agents and decision-making reasoning modules;

[0008] A web page address set is constructed by the scheduling agent, and a crawling task allocation table is generated based on the web page update timeliness and source characteristics of each web page in the web page address set, and the crawling tasks in the crawling task allocation table are sent to the crawling agent;

[0009] Executing the crawling task through the crawling agent and acquiring the original data of the web page in response to the web page data loading completion event;

[0010] Extracting webpage source information and publishing time information from the webpage original data to generate model context information;

[0011] Constructing a page structure representation of the web page original data through the parsing agent, locating key information nodes in the web page original data based on the page structure representation, and extracting key information from the web page original data to generate structured data;

[0012] The analysis agent performs analysis processing based on the structured data to generate multi-scenario decision data;

[0013] The decision reasoning module generates a decision result based on the multi-scenario decision data and the model context information.

[0014] Furthermore, to achieve the above-mentioned purpose, the present invention provides an agent-based web page information processing device, comprising:

[0015] Configuration module, used to configure the scheduling agent, grasping agent, parsing agent, analysis agent and decision reasoning module;

[0016] A scheduling module, configured to construct a web page address set through the scheduling agent, generate a crawling task allocation table based on the web page update timeliness and source characteristics of each web page in the web page address set, and send the crawling tasks in the crawling task allocation table to the crawling agent;

[0017] A crawling module, configured to execute the crawling task through the crawling agent and obtain the original data of the web page in response to a web page data loading completion event;

[0018] A context generation module, configured to extract webpage source information and publishing time information from the webpage original data to generate model context information;

[0019] a parsing module configured to construct a page structure representation of the original web page data through the parsing agent, locate key information nodes in the original web page data based on the page structure representation, and extract key information from the original web page data to generate structured data;

[0020] An analysis module, configured to perform analysis processing based on the structured data through the analysis agent to generate multi-scenario decision data;

[0021] An inference module is used to generate a decision result based on the multi-scenario decision data and the model context information through the decision inference module.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and an agent-based web page information processing program stored in the memory and runnable on the processor. When the agent-based web page information processing program is executed by the processor, the steps of the agent-based web page information processing method as described above are implemented.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a web page information processing program based on an intelligent agent is stored. When the web page information processing program based on an intelligent agent is executed by a processor, the steps of the web page information processing method based on an intelligent agent as described above are implemented.

[0024] Beneficial effects: The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as financial technology and medical health. It discloses a web page information processing method based on an intelligent agent, including: configuring a scheduling intelligent agent, a crawling intelligent agent, a parsing intelligent agent, an analysis intelligent agent and a decision-making reasoning module; constructing a web page address set and generating a crawling task allocation table, and sending the crawling task to the crawling intelligent agent; executing the crawling task and obtaining the original data of the web page; extracting the source information and release time information of the web page to generate model context information; constructing a page structure representation of the original data of the web page, locating key information nodes and generating structured data; performing analysis and processing based on the structured data to generate multi-scenario decision data; and generating decision results based on the multi-scenario decision data and model context information. The present invention assigns tasks to a set of web page addresses through a scheduling agent, and generates a crawling task allocation table based on the web page update timeliness and source characteristics to achieve fine scheduling of tasks; the original data of the web page is obtained through the crawling agent, and model context information containing source information and release time information is generated to provide a consistent data basis for subsequent parsing, analysis and reasoning; the model context information is used to participate in decision-making reasoning during the structured data generation and analysis processing, thereby realizing a continuous processing flow from web page acquisition to result generation, and improving the relevance of data processing and the accuracy of decision output. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0026] Figure 1 A schematic diagram of an application environment of an agent-based webpage information processing method according to an embodiment of the present invention;

[0027] Figure 2 This is a flow chart of an embodiment of an agent-based web page information processing method of the present invention;

[0028] Figure 3 Schematic diagram of functional modules of a preferred embodiment of an agent-based web page information processing device of the present invention;

[0029] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0030] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] The web page information processing method based on intelligent agent provided by the embodiment of the present invention can be applied in the following fields: Figure 1In an application environment, the user end communicates with the server end through the network. The server end can configure the scheduling agent, crawling agent, parsing agent, analysis agent, and decision reasoning module through the user end; construct a set of web page addresses and generate a crawling task allocation table, and send the crawling task to the crawling agent; execute the crawling task and obtain the original web page data; extract the web page source information and release time information to generate model context information; construct a page structure representation of the original web page data, locate key information nodes and generate structured data; perform analysis and processing based on the structured data to generate multi-scenario decision data; and generate decision results based on the multi-scenario decision data and model context information. The present invention assigns tasks to a set of web page addresses through a scheduling agent, and generates a crawling task allocation table based on the web page update timeliness and source characteristics to achieve fine scheduling of tasks; obtains the original data of the web page through the crawling agent, and generates model context information containing source information and release time information, providing a consistent data basis for subsequent parsing, analysis and reasoning; uses the model context information to participate in decision-making reasoning during the structured data generation and analysis processing, thereby achieving a continuous processing flow from web page acquisition to result generation, improving the relevance of data processing and the accuracy of decision output. Among them, the user end can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0033] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of an agent-based webpage information processing method provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the agent-based web page information processing method proposed by the present invention includes the following steps:

[0035] S10, configure the scheduling agent, grasping agent, parsing agent, analysis agent and decision reasoning module;

[0036] In this embodiment, a multi-agent system (MAS) architecture is used to achieve distributed decoupling and task autonomy of the data processing process. Each agent in the MAS system has autonomy, local view and decentralization characteristics, thereby ensuring that it can execute independently and collaborate efficiently throughout the entire process. Autonomy is reflected in the agent's independent decision-making and task response capabilities, supporting the independent completion of local tasks in the absence of a global view; the local view design effectively reduces the processing load of a single module and improves the horizontal scalability of the system; the decentralized deployment mechanism avoids single-point bottlenecks, improves the overall robustness of the system, and improves the stability and flexibility under high-concurrency data capture tasks.

[0037] During the configuration process, each type of agent or module has clearly defined tasks and is capable of independent deployment and coordinated invocation. The scheduling agent configuration process includes agent initialization, resource binding, task reception interface establishment, and task priority scheduling model loading. Its technical foundation is based on task scheduling algorithms, multi-agent coordination and allocation mechanisms, and event-driven heterogeneous web page scheduling strategies. It can be combined with a contextual resource manager to load rule engines, apply scheduling templates, and dynamically update resource weight matrices, thus supporting the flexible deployment of crawling tasks under conditions of multiple data sources and frequent updates.

[0038] The crawling agent's configuration primarily focuses on loading and controlling dynamic web page rendering within the browser simulation environment. Based on browser automation frameworks such as WebDriver, the agent must build logic for detecting page structure loading, listening for DOM node state changes, and extracting asynchronous response content. The page data capture process must not only cover static HTML code but also support the integrated extraction of JavaScript execution results and asynchronous request responses, generating the complete raw web page data.

[0039] The configuration of the parsing agent emphasizes understanding complex page structures and template-driven key information extraction. Its operation relies on a standardized rule set for structural tags, a source tag mapping table, and a context-sensitive parsing rule base. To enhance parsing adaptability, the parsing agent should be able to dynamically load parsing templates based on the web page source, and define target field paths, field format standards, and error tolerance parameters within the templates.

[0040] The core of the analytical agent configuration lies in the adaptation and analysis of multi-source structured data. It can load modules such as structured field standard tables, time series recognition components, and statistical indicator generation rules. It has the ability to automatically identify input structured data, map fields, and match analysis strategies. Furthermore, to support trend and cyclical identification across business scenarios, the analytical agent should also have a built-in multi-scenario configuration mapping interface and rule adaptation engine.

[0041] Configuration of the decision-making reasoning module includes establishing the model context input channel, enabling the rule-driven reasoning engine, and setting up the decision scenario selection mechanism. Its task is to activate decision rules and derive conclusions based on structured data and multi-scenario analysis results, taking into account contextual timeliness tags and source tags. The module must support configurable inference rule loading, dynamically select inference paths based on the scenario, and output quantitative or qualitative decision conclusions.

[0042] The above modules can be independently deployed through a distributed microservice architecture. The service capabilities of each module are marked at the system level through the service registration center. When executing path planning, the task flow controller dynamically schedules and calls the intelligent agent based on the service capabilities and load status, realizing a task-oriented data processing closed loop and seamless collaboration between modules.

[0043] In actual implementation, various agents can be encapsulated as independent service components using a containerized deployment approach. For example, the scheduling agent can be encapsulated as a service unit capable of managing crawling task flows. By configuring a crawling strategy file and a web page priority scheduling model, it can establish task queues based on web page source, update time, and historical response history. The crawling agent can be deployed by initializing multiple asynchronous crawling threads using browser emulation engines such as Selenium or Playwright. Upon completion, it performs DOM state snapshot extraction and page behavior monitoring. The parsing agent's initialization includes loading a template matcher, embedding a rule execution engine, and defining document structure mapping logic. Upon receiving a parsing request, it matches parsing templates based on web page context tags and performs structured information extraction node by node. The analysis agent supports the concurrent execution of multiple analysis paths and can adapt to different data processing requirements by binding field combinations, analysis model types, and analysis depth through parameter files. The decision-making and reasoning module deploys a rule-priority-driven decision graph structure, taking model context and analysis results as input nodes to drive the sequential execution of decision rule groups until the final inference conclusion is output.

[0044] Example: In the healthcare sector, a public healthcare regulatory platform needs to monitor drug approval information and epidemic notifications published on government websites in multiple regions in real time. A scheduling agent is configured to manage the cadence and priority of web crawling from multiple sources. The crawling agent captures dynamically updated policy pages through browser simulation. The parsing agent extracts key drug names and approval dates based on structured table templates matched to source sites. The analysis agent generates regulatory response alerts based on historical drug approval times and release frequencies. Finally, a decision-making and reasoning module outputs a list of risk levels to be monitored and feeds this information back to the medical data command system.

[0045] In the financial sector, a securities analysis platform deploys this module system to crawl tariff policy changes on the official websites and news sites of multiple financial institutions. The scheduling agent optimizes crawling frequency based on site timeliness and stability. The crawling agent captures economic indicator data embedded in dynamic web pages. The parsing agent extracts the target country, product category, and tax rate change fields from the page. The analysis agent combines historical securities volatility data to perform periodic change assessments. Finally, the decision-making and reasoning module generates hedging strategy recommendations or position adjustment instructions. This approach effectively improves the automation and responsiveness of strategy generation.

[0046] This embodiment, through the configuration of a scheduling agent, a crawling agent, a parsing agent, an analysis agent, and a decision-making and reasoning module, separates the responsibilities of each functional node in the web page information processing process and modularizes them. Within the same processing framework, a continuous processing chain from task allocation, data collection, content parsing, structured conversion, to decision output is established. This solves the problems of information fragmentation, processing delays, and decision lags caused by the separation of processing units in traditional processing flows. The configured agent architecture offers flexible expansion and on-demand deployment capabilities, automatically adapting execution paths to different source web page types and data structures of varying processing complexity, improving processing efficiency and the applicability and accuracy of the final decision results.

[0047] S20, constructing a web page address set through the scheduling agent, generating a crawling task allocation table based on the web page update timeliness and source characteristics of each web page in the web page address set, and sending the crawling tasks in the crawling task allocation table to the crawling agent;

[0048] In this embodiment, the scheduling agent serves as the starting module of the data processing flow, and is responsible for screening web addresses with business analysis value from the target web resource set, and constructing a web address set for subsequent task scheduling. The process of constructing a web address set not only includes the extraction of the address itself, but also involves the identification and pre-classification of address attributes. Web page update timeliness indicates the frequency or period of web page content updates. This feature can be estimated through page meta tags, server response header fields such as Last-Modified, ETag, or publishing time extracted from historical crawling records. Source features are used to describe the credibility of the publisher of a web page, the industry category to which it belongs, or the stability of its content. This feature can be annotated and standardized through domain name suffixes (such as .gov, .edu), source whitelists, URL path features, web page structure similarity models, and other methods.

[0049] The scheduling agent establishes a corresponding weight model based on the update timeliness and source characteristics of each web page in the web address collection. Combined with the current task cycle or policy window, it comprehensively generates crawling task priority indicators. The task allocation table is a scheduling data structure generated based on the weighted ranking. It describes the execution order of each web page task, the corresponding crawling parameters, and the task scheduling tags. The crawling task allocation table can be implemented using a queue, hash structure, or a prioritized task stack, and can be dynamically updated.

[0050] Sending grasping tasks to the grasping agent requires a stable data interface or message channel. Common implementations include event-triggered mechanisms, queue delivery mechanisms, or RESTful interface push mechanisms. Task pushes must include a complete web address, scheduling parameters, a unique task identifier, and a timestamp to ensure the grasping agent can efficiently and accurately respond to scheduling requests in a parallel processing environment.

[0051] In practical implementation, a scheduling agent can be configured to bind a web address source file or API source, which contains a large number of potentially available web addresses and their attribute tags. After parsing the source and extracting a set of candidate web addresses, the scheduling agent constructs a meta-attribute dictionary for each web page, including historical update time, last access response time, server type, and URL path characteristics. Update timeliness can be calculated by analyzing historical crawling result intervals or by modeling the update cycle index provided by third-party public data sources. Source characteristics can be combined with classification models to output a source credibility score.

[0052] The scheduling agent calculates web crawling priority scores based on these meta-attributes. For example, it uses the update timeliness index as a basis to weight the source credibility score to generate a comprehensive score, and then sorts the scores to form a crawling task allocation table. Each task record may include the crawl target URL, expected access time window, maximum number of retries, and the identification of the strategy group to which it belongs.

[0053] Once crawling tasks are assigned, the scheduling agent pushes them to the crawling agent via an internal task channel module. Task data can be transmitted using Kafka message channels, Redis queues, or HTTP-based push services. The crawling agent listens to this channel and pulls tasks according to the schedule, enabling parallel or batch crawling of different web pages.

[0054] Example: In a healthcare scenario, a dispatching agent extracts webpages for information on drug approval progress, public notices, and vaccine announcements from multiple sources, including the National Medical Products Administration and the Centers for Disease Control and Prevention. Tasks are prioritized based on page update times and the trustworthiness of the source. The system prioritizes frequently updated, officially published websites for the crawling agent, ensuring timely access to drug and vaccine data for the healthcare regulatory system.

[0055] In the financial field, the scheduling agent extracts tariff policy releases and financial regulatory announcement web pages from the Ministry of Finance, major financial media websites, and tariff database platforms. Based on the policy keywords carried in the URL path, historical change frequency, and domain name authority score, a crawling task sorting mechanism is constructed, allowing the securities market analysis system to prioritize obtaining important policy web page information that may trigger market reactions, thereby providing a more timely input source for subsequent price trend analysis and trading strategy generation.

[0056] This embodiment introduces two parameters, web page update timeliness and source characteristics, into the scheduling agent for task priority analysis. This allows the crawling task allocation process to precisely focus on web content with frequently changing information and reliable sources, effectively improving the overall efficiency of web crawling resources and the real-time nature of information coverage. The task allocation table construction mechanism enables structured management of task scheduling while supporting the orderly delivery of tasks. This addresses issues such as uneven target distribution and low-value web pages occupying crawling resources during the web crawling process, ensuring that subsequent data processing modules receive higher-quality web data input.

[0057] S30, executing the crawling task through the crawling agent, and obtaining the original data of the web page in response to the web page data loading completion event;

[0058] In this embodiment, the crawling agent is responsible for executing web page loading and data extraction operations based on the task content sent by the scheduling agent. A crawling task is the smallest execution unit consisting of a web page address extracted from the task allocation table, a scheduling priority, and related parameters. The crawling task execution process first involves loading the target web page. This loading process is implemented through a browser emulation driver, which provides a complete web page rendering environment, supporting the execution of scripts embedded in the web page, loading external resources, and handling asynchronous requests.

[0059] The webpage data loading completion event is the core trigger for determining whether the webpage content is extractable. This is typically implemented based on a webpage document status monitoring mechanism, such as monitoring the DOM tree construction status, whether a specific element has been rendered, or whether a specific network request has returned. The crawling agent must set the corresponding event judgment logic, which can be combined with explicit loading indicators (such as JavaScript variable status, loading class cancellation) or implicit behaviors (such as asynchronous request completion) to make a composite judgment.

[0060] After a webpage loads, the crawling agent extracts the webpage's raw data. This data includes not only the surface HTML code but also the dynamic content generated by JavaScript execution and responses obtained through asynchronous communication (e.g., AJAX and Fetch). This data is centrally extracted through a browser emulation driver interface, ultimately forming a raw data packet containing rendered HTML code, dynamically generated data, and network response content, which serves as input for subsequent parsing.

[0061] In one specific embodiment, the crawling agent is deployed as a containerized task execution node that supports browser simulation, and internally uses frameworks such as Selenium, Puppeteer or Playwright to simulate browser operations. After the system is initialized, the crawling agent loads the corresponding web page according to the assigned task. If the task contains a loading identification rule, the monitoring target of the web page loading completion event will be dynamically set, such as the appearance of a specified element, the DOM ready state or the XHR request completion signal. After the web page is loaded, the crawling agent extracts the final DOM structure of the page, that is, the rendered HTML content, and uses the script execution method in the browser context to extract dynamic content (such as capturing the content generated by JavaScript variables through document.evaluate). In addition, the crawling agent monitors the network request channel, records and extracts all asynchronous response content triggered by the front-end page, such as interface return data in JSON or XML format. All acquired data content will be unified and integrated to form the original data object of the web page, and attached with metadata such as the crawling timestamp, task identifier and source address for subsequent processing by the parsing agent. The crawling agent supports task execution logs and exception screenshot recording mechanisms to ensure that backtrace information can be provided when web page loading exceptions or script errors occur.

[0062] Example: In a healthcare data monitoring system, a crawler agent accesses the vaccine approval announcement page on the National Medical Products Administration's official website. The drug name and indication information on this page are dynamically written to the DOM using JavaScript, while the announcement batch number is asynchronously loaded using AJAX. After loading the page, the crawler agent listens for DOM change events and XHR completion signals. After rendering the target content, it uniformly extracts the dynamically generated complete page structure and network response data, generating high-quality raw web data for parsing.

[0063] In the financial information monitoring system, a crawling agent accesses a tariff change page on a financial news website. The core policy content on this page is loaded via a nested iframe, and related data charts are asynchronously requested via the Fetch API. The crawling agent uses browser emulation to recursively load the nested frames and extract all tax rate adjustment data from the network response. This ensures that the underlying data received by the securities risk model is comprehensive, authentic, and timely, providing data support for the subsequent generation of adjustment strategies.

[0064] This embodiment effectively overcomes the limitation of traditional crawler tools in being unable to obtain JavaScript dynamically generated content and asynchronous request data by simulating the real user browser environment, ensuring the ability to obtain complete data of dynamic web page structures. The monitoring mechanism for web page data loading completion events ensures the accuracy of data extraction timing and avoids the problem of missing information caused by extraction before the page is fully rendered. The unified structure of the web page raw data output method provides a stable and consistent data foundation for subsequent structured analysis, significantly improving the analysis accuracy and processing efficiency.

[0065] S40, extracting webpage source information and publishing time information from the webpage original data to generate model context information;

[0066] In this embodiment, webpage raw data is a composite data set generated by a crawling agent after loading and extracting the webpage. This data set includes the rendered HTML code, script execution results, and asynchronous response data. Within this data structure, webpage source information and publication time information, as high-level metadata elements, possess strong semantic localization capabilities and data traceability, serving as the semantic foundation for subsequent parsing and decision analysis.

[0067] Web page source information indicates the publisher, platform attributes, and trustworthiness of web data. This information can be comprehensively extracted by parsing URL addresses, domain suffixes (such as .gov, .org), content in HTML meta tags (such as publisher and generator), certificate information, the source of embedded frames on the page, and structural similarity rules. This process requires not only obtaining surface links but also identifying whether the web page embeds third-party content and whether there are jumps, redirects, or proxy mechanisms to establish a stable and accurate source mapping tag.

[0068] Publication time information typically refers to the time when webpage content was first generated or last updated. It can be inferred from a combination of multiple signals. Common sources include the publishdate and last-modified fields in HTML meta tags, time expressions in the body of the content, the Last-Modified field in the HTTP response header, and dynamically rendered timestamps in JavaScript scripts. Accurately extracting publication time is crucial for analyzing data timeliness and labeling decision-making data with time tags.

[0069] After extracting webpage source information and publication time information, they are combined to construct a unified model context. This model context is encapsulated as a key-value mapping or nested object, including source tags, timeliness tags, extraction paths, parsing methods, and confidence levels. This provides contextual input for subsequent parsing agent template matching and analysis agent modeling.

[0070] In one specific implementation, after the web page data is loaded, the crawling agent pushes the original data to the upstream and downstream processing chains. The module that extracts the web page source information and release time information will first perform structural decomposition on the original HTML structure, extracting the meta block, link reference, script declaration block and main content area in the header area. The extraction of source information first analyzes the domain name through the URL parsing component to determine its ownership and suffix classification (such as .gov is identified as a government agency) and calls the configured source weight dictionary for labeling. The content structure analysis engine is further used to identify web page footers, watermarks, LOGO icon alt tags, navigation bar reference content, etc., to supplement the source auxiliary identification mark. Finally, a structured source label object containing the main domain name, source category, and confidence level is formed. The release time information extraction logic is completed based on the linkage of multiple modules such as text regularization, time expression identifier, JavaScript parser, etc. The system scans HTML for meta tags, such as name="publishdate" and property="article:published_time." It also locates sentences in the body of the article that begin with keywords like "Published Time:" and "Updated Time:," extracting the date and time information that follows. In asynchronous scripts, JavaScript rendering timestamps are extracted by listening for DOM change events. All candidate time information is fed into a time normalization module for format verification, deduplication, and sorting, selecting the primary publication time field. Once the source and publication time information is extracted, a structured model context information object is constructed using the context model generator. This object is then bound to the current webpage's original data using a one-to-one identifier, forming a context input pair for subsequent parsing.

[0071] Example: In the medical and health data acquisition system, the system crawled the original data of the vaccination guide webpage of the National Center for Disease Control and Prevention. The system identified the main domain name of the webpage as www.chinacdc.cn and marked it as "authoritative medical source" through the government agency domain name dictionary. The HTML header contains <meta name＝"publishdate"content＝"2023-12-01"> The system identifies the value in the field as release time information and, after normalization, generates a label with the value "Release Date: 2023-12-01." The model context information for this webpage is labeled "Source: Authoritative Medical Institution" and "Availability: 2023-12-01."

[0072] In the financial business regulatory monitoring system, a crawling agent retrieved a policy update announcement from the U.S. Treasury Department's website. It identified the source as "Government Financial Regulatory Department" using the domain name home.treasury.gov and the text "Updated: March 3, 2024" at the top of the text. After parsing, it converted the information into the time-sensitive tag "2024-03-03." This model contextual information is used in subsequent analysis modules to determine whether the policy change is recent and to participate in securities volatility modeling and analysis.

[0073] This embodiment significantly enhances the ability of subsequent modules to understand the semantic context and data timeliness of web pages by automatically extracting source information and release time information from the original web page data and generating structured model context information. Source information provides a basis for determining trust levels, enabling the system to select appropriate parsing rules and strategies based on the source type; release time information imbues the data with temporal attributes, facilitating time series modeling and data update judgment by the analysis module. Overall, this improves the contextual adaptability and accuracy of data processing, avoiding data bias and model misjudgments caused by unknown web page sources or missing release times.

[0074] S50, constructing a page structure representation of the web page original data by the parsing agent, locating key information nodes in the web page original data based on the page structure representation, and extracting key information from the web page original data to generate structured data;

[0075] In this embodiment, after receiving the raw webpage data, the parsing agent needs to structurally model the rendered HTML code to construct a page structure representation. This page structure representation is an abstract modeling result of the webpage hierarchy, tag nesting, and element attributes. It is usually expressed as a document object model structure, recording the tag type, hierarchical position, style attributes, and content mapping path of each page node. The core purpose of constructing this structural representation is to restore the page's display logic and provide a basis for parsing path selection.

[0076] Model context information provides constraints on the parsing strategy during this process. In particular, web page source information is used to match target parsing templates within the parsing rule library. Each template corresponds to a specific web page source or structural style, and includes target data field definitions and structural positioning rules such as XPath expressions, tag combination patterns, and text block offset distances. Template matching can be performed directly based on source tags, or by selecting the optimal template from multiple candidate templates through structural similarity comparison.

[0077] On the basis of constructing the page structure representation, the parsing agent performs structure traversal and matching positioning based on the target data field list given in the template, and extracts key information nodes. Key information nodes refer to HTML elements or their derived mapping areas on the page that contain target data field values, such as the ones containing policy titles. <h1>Node, including release time <meta> Labels, cells containing numerical information, etc.

[0078] After successfully locating a key information node, the parsing agent extracts the node's content, including both the node's plain text data and structural attribute fields, such as class, href, and data-* attributes. The extracted text and attribute fields are then collectively submitted to a consistency verification process, which uses a rules engine to verify the data type, unit format, and content legitimacy. For example, date fields must conform to a standardized year-month-day format, and amount fields must exclude content containing descriptive text.

[0079] Ultimately, the qualified text content and attribute fields are encapsulated into a structured data object. This object structure refers to the pre-defined format defined in the target parsing template and may take the form of key-value mapping, nested objects, array collections, etc., and is output together with metadata such as the original web page identifier and field extraction path for subsequent analysis modules to call.

[0080] In one specific embodiment, the system's deployed parsing agent uses a rule-driven DOM parsing engine, combining XPath and regular expression location mechanisms to analyze page structure. At the start of a parsing task, the agent first uses an HTML parser to parse the HTML code in the webpage's raw data into a tree structure, building an in-memory DOM model. This model records each node's type, text content, attribute information, and its hierarchical path within the tree. Based on the source tag in the model context, such as "financial website" or "official government website," the parsing agent accesses a set of matching parsing templates. Each template contains a set of field extraction rules, and the system determines the final template to use based on priority matching and page structure hash digest comparison. For example, if the source is "cn.gov" and the DOM structure's hash proximity to a preset template A exceeds a threshold, the system selects template A as the task template. The system then reads a list of target field configurations from the template, such as "title field: / / h1" and "publication date field: / / meta[@name='publishdate']," and performs path traversal based on the current DOM model. During the traversal process, if the template location path does not exist, the system triggers the fallback path mechanism, such as trying a keyword + relative path combination for remedial extraction. After the node is successfully located, the system extracts the node's text content and attributes dictionary, and performs field format verification logic. The title field must be a string and cannot be empty or contain special symbols; the date field must be successfully parsed using the ISO 8601 format; and the numerical fields must be checked for numerical validity and have unified units (e.g., "100 million yuan" must be uniformly converted to "1 billion"). The data that passes the verification is standardized and encapsulated into a structured JSON object, and label information such as the field extraction path, rule hit information, and context ID are attached, and output as a structured data stream. The entire parsing process will record the field hit rate, jump path, and abnormal field logs for subsequent rule updates and parsing template optimization.

[0081] Example description: In the medical and health business field, after receiving the vaccination notice webpage published on the website of the National Health Commission, the parsing agent first constructs a page structure representation based on the rendered HTML code in the original data of the webpage. This structure representation is a DOM tree structure that records the hierarchical relationship, tag type and accompanying attribute fields of all nodes in the page. In this DOM tree, the agent matches the path rules specified in the template, such as "the main title of the text is located at the top of the page.< / h1> <h1>Node", "Vaccination age group is located in <div class＝"content">The agent then locates various key information nodes in sequence by traversing the structure and judging the relative path. Key information fields such as "vaccine recipient," "number of vaccination doses," and "vaccine brand" are extracted. Finally, the agent encapsulates the extracted structured fields into a standardized JSON structure for subsequent health record data comparison and vaccine coverage modeling.

[0082] In the FinTech business domain, a parsing agent constructs a page structure representation for a "Tariff Adjustment Policy News" page captured from a connected financial information portal. In this scenario, the constructed page structure is also a DOM tree, whose node structure includes the main content area, title area, time tags, body paragraphs, and table-type data nodes. Based on a parsing template that maps this DOM structure to source tags, the parsing agent uses predefined path rules to quickly identify the news headline, release date, body paragraphs involving tax rate percentages, and the product category and amount fields within nested tables. In this scenario, the page structure representation not only provides a hierarchical location for content areas but also helps the agent skip invalid content areas such as navigation bars and advertising sections. The resulting structured fields include "Tariff Policy Title," "Affected Goods," "Adjustment Amount," and "Release Time," providing high-confidence semantic input for subsequent policy impact assessment and securities market trend prediction modules.

[0083] This embodiment builds a page structure representation to enable the system to have a structural hierarchical understanding of web page content, avoiding the traditional parsing method's reliance on hard-coded surface HTML structures and improving parsing flexibility and adaptability. By matching parsing templates for field extraction, parsing accuracy is enhanced and the cost of configuring parsing rules is significantly reduced. The output format of structured data is unified and rigorously structured, providing a high-quality, semantically strong input foundation for subsequent analysis modules, effectively improving the reliability and efficiency of multi-scenario decision modeling.

[0084] S60, performing analysis processing by the analysis agent based on the structured data to generate multi-scenario decision data;

[0085] In this embodiment, after receiving structured data, the analytical agent begins a deep analysis process based on data dependency features and scenario parameters. Structured data consists of text and field sets extracted and formatted by the analytical agent, typically including the target webpage's core entity elements, numerical fields, classification information, and time tags. The core purpose of this analytical processing is to uncover potential patterns reflected in the current data through statistical modeling, time series trend extraction, and inter-field correlation analysis. This analysis is then combined with pre-set multi-business scenario configurations for semantic mapping, thereby generating multi-scenario decision data for use by the inference module.

[0086] During the analysis process, the system first identifies key fields in the structured data, such as price, quantity, and event time. It then combines this with the historical structured data stored in the database to perform field alignment and time window selection. This step can be performed using specific field name mapping rules or a data dictionary to standardize fields and ensure structural consistency when comparing data.

[0087] The analytical agent then performs statistical correlation analysis on the structured and historical structured data. This process includes calculating field means, variances, standard deviations, ranges, and covariances, and constructing a correlation matrix between fields to assess linear and nonlinear relationships. A set of statistical indicators serves as an intermediate variable, determining dimensions such as the degree of data anomaly, the magnitude of change, and the direction of trends.

[0088] The analytical agent further identifies cyclical patterns and trends in the data based on statistical indicators and the time series structure of historical data. This includes detecting rising and falling inflection points, delayed responses, cyclical patterns, and outliers. Cyclic patterns can reflect the periodic trends of changes in the data domain, while trends can be used to predict future changes.

[0089] The generation of multi-scenario decision data is based on scenario parameter adaptation logic. The system pre-defines multiple business scenario configurations, each containing corresponding policy variables, constraint boundaries, and field mappings, such as the "Securities Trend Assessment," "Medical Warning Trend," and "Commodity Price Anomaly" scenario configurations. The analytical agent maps identified trends and cyclical results to the indicator system for each business scenario, generating a structured set of analysis results for that scenario. This set, serving as multi-scenario decision data, is typically structured as a field-indicator-scenario triplet, enabling cross-scenario linkage and reasoning input capabilities.

[0090] In one specific implementation, the system's operational analysis agent utilizes a statistical analysis module built on Pandas and StatsModels, along with a proprietary scenario adaptation engine. Upon receiving structured data, the agent first maps all fields to a unified analysis field domain based on standardized field name mapping rules. It then calls a historical data reading interface to load historical structured data with consistent field structures and matching timeframes. After alignment, the agent performs batch statistical metric calculations, generating standardized numerical metric tables. The system maintains a library of scenario configurations categorized by industry. Each scenario configuration includes a field selection table, threshold settings, weighting parameters, and business interpretation rules. In the "Tariff Change Forecast" scenario, if the trend in the "Tariff Adjustment Range" field in the current data exceeds a set threshold and the negative correlation coefficient with the "Total Import and Export Commodity Volume" field exceeds a given value, the system marks the scenario as "Alert" and generates a set of scoring indicators indicating the degree of policy impact. All logical rules in the scenario mapping process are extensible and support version iteration. Once generated, the multi-scenario decision data is packaged in a structured format with source data pointers, analysis rule versions, and context labels, and then output to the decision reasoning module or front-end display system. The system supports dynamic addition of scenario configurations and re-execution of the mapping process based on existing data sets to ensure that decision models can be called to meet different business needs.

[0091] Example: In the healthcare sector, structured data includes data on sudden infectious disease outbreaks in a specific region, such as the number of new cases, the time of reporting of suspected cases, and patient medical records. The analytical agent, combining historical case data from the same season over the past five years, identifies that the current case curve is growing at a rate significantly higher than two standard deviations from the mean, and also finds that the cyclical trend is highly consistent with a specific historical epidemic phase. In the "Regional Medical Resource Allocation" scenario, the system generates decision data based on this data, including recommended indicators such as "Epidemic Surge Warning," "Recommendation for Allocation of Ventilators," and "Recommendation for Opening Fangcang Cabin Hospitals," along with fluctuation curves and threshold information.

[0092] In the fintech business sector, structured data records changes in import and export tariffs on a specific category of goods, along with monthly trading volume data for the past three months. Analysis by an intelligent agent revealed a significant negative correlation between tariff adjustments and commodity trading volume, with the trading volume fluctuation cycle highly similar to that of past trade periods. In the "Cross-border Risk Asset Management" scenario, the system mapped these analysis results into multi-scenario decision-making indicators, such as "Highly Sensitive Category Flag" and "Recommendation to Lower the Risk Rating of Related ETFs," along with contextual support, including the source tags "Government Announcement" and "High Update Time."

[0093] This embodiment uses a fusion analysis of structured and historical data to not only identify trends and cyclical characteristics of target fields, but also automatically maps these characteristics into interpretable analytical indicators within each scenario, based on different scenario configurations. This approach improves the structural consistency and cross-scenario applicability of decision data generation, avoiding the misjudgments and lags often associated with manually configured scenario rules in traditional systems. It also provides a more reliable and context-coupled decision input data source for subsequent modules.

[0094] S70, generating a decision result based on the multi-scenario decision data and the model context information through the decision reasoning module.

[0095] In this embodiment, the core task of the decision-making inference module is to derive a final, executable decision outcome within a given rule structure, based on the multi-scenario decision data from the analytical agent and the model context information generated earlier in the capture phase. To achieve this, the module's technical architecture employs a logically layered approach to organize multiple submodules, ensuring that input information can be interpreted, integrated, and mapped across different semantic dimensions.

[0096] First, the decision-making inference module receives multi-scenario decision data, including trend results, rule triggers, and corresponding quantitative indicators across multiple business scenarios. Each data item is labeled with a scenario, field identifier, and analysis basis, such as "Industry price average decline exceeds warning threshold" or "Regulatory event exposure frequency continues to rise." Before entering the inference logic, this structured data is parsed and mapped to predefined rule templates to form a set of policy activations to be evaluated.

[0097] Model context information includes the source tag, publication time tag, and other contextual attributes of web data. These fields do not directly contribute to business metric calculations, but rather serve as weighting factors in the inference process to assess data credibility and validity. Specifically, the source tag is typically compared with the "trust level" field in the rule template. For example, the source tag of a government website might be mapped to a high-confidence node, while that of a news aggregation site might be labeled neutral or low-weighted. During inference, the same metric value generated by different sources will have different weights, affecting the path distribution of the final inference result.

[0098] The decision-making inference module processes the input structure using a decision graph model or conditional rule tree. Each node in the rule tree represents an indicator or decision condition, such as "Indicator X > Upper Limit Y and Source Weight ≥ 0.8." Leaf nodes represent decision actions, such as "Mark as High-Risk Event" or "Push Automatic Response Policy." Rule paths are evaluated by calculating the probability of each path being satisfied and selecting the highest priority path from all satisfied paths as the primary inference result for the current input.

[0099] To achieve semantic-level integration, the system introduces a contextual enhancement factor mechanism, converting the timeliness labels in the model's contextual information into the input of a decay function. For example, if the time difference between the analyzed data and the release time exceeds a certain threshold, the system exponentially decays the weight of that data in reasoning to prevent old data from misleading decision outputs. The implementation method can be referred to in the following formula:

[0100] Weight correction value = original weight × exp(-λ × time difference)

[0101] Raw weight refers to the basic confidence weight assigned to a piece of input data within the model's context, based on factors such as the trustworthiness of the webpage's source and the integrity of the data structure. Generally, data from authoritative sources (such as government websites and professional statistical agencies) have higher raw weights, while anonymous forums and automatically aggregated pages have lower raw weights.

[0102] The time difference represents the interval between the current system time and the data's release time. It can be expressed in hours, days, or other uniform time units. A larger time difference indicates that the data may be "older," and its validity and reference value decrease.

[0103] λ (lambda) is an adjustable "decay coefficient" that controls the effect of time differences on the original weights. It is usually a positive real number. A larger λ value results in faster weight decay; a smaller λ value indicates a higher tolerance for outdated data.

[0104] exp(-λ × time difference) is a typical exponential decay function. This function decreases monotonically. The larger the time difference, the smaller the output of the function, indicating that the "effective weight" of the data naturally decays over time.

[0105] Through the above mechanism, the entire reasoning process not only relies on business rules, but also makes full use of the contextual semantics of time and source for dynamic judgment.

[0106] After generating a decision, the module also records all rule path hit information, context sources used, triggered scenario tags, and other information for subsequent system audits, version comparisons, and user feedback tracking. This information is output in a unified JSON structure, supporting structured writes to the logging system and subsequent module calls in the decision visualization system.

[0107] In one specific embodiment, the decision-making reasoning module is built based on an inference engine framework and consists of three parts: a rule matching submodule, a weight fusion submodule, and an output encapsulation submodule. The inference engine performs logical mapping based on a JSON rule definition table. Each rule contains input fields, matching conditions, threshold ranges, and response action fields. In actual operation, the system extracts fields such as "scenario identifier," "key indicator fields," and "trend labels" from the multi-scenario decision data received by the analysis agent and loads the scenario rule set. The matching process uses a conditional tree-based path selection strategy, mapping multiple indicators to corresponding weighted paths in the rule nodes. A weighted calculation is performed based on the confidence coefficient of the "source label" in the model context information to produce the final scoring result. The decision-making reasoning module also incorporates a time-correlation correction mechanism. When the release time in the model context information is later than the set validity threshold, the system automatically reduces the weight of the data in the decision or marks it as low-confidence to prevent misleading inferences caused by outdated data. After the module is run, the decision results generated include structural fields such as "recommended action", "risk level", "trust score", and "trigger indicator set". They are output to the visualization module or other system components in a JSON structure, and the decision rule path and hit rule ID are recorded for subsequent backtracking and version comparison.

[0108] This embodiment integrates structured scenario decision data and contextual metadata through a decision reasoning module, and effectively improves the response accuracy and stability of decision results by introducing source credibility and timeliness indicators.

[0109] The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as financial technology and medical health. It discloses an agent-based web page information processing method, including: configuring a scheduling agent, a crawling agent, a parsing agent, an analysis agent and a decision-making reasoning module; constructing a web page address set and generating a crawling task allocation table, and sending the crawling task to the crawling agent; executing the crawling task and obtaining the original data of the web page; extracting the source information and release time information of the web page to generate model context information; constructing a page structure representation of the original data of the web page, locating key information nodes and generating structured data; performing analysis and processing based on the structured data to generate multi-scenario decision data; and generating a decision result based on the multi-scenario decision data and the model context information. The present invention assigns tasks to a set of web page addresses through a scheduling agent, and generates a crawling task allocation table based on the web page update timeliness and source characteristics to achieve fine scheduling of tasks; the original data of the web page is obtained through the crawling agent, and model context information containing source information and release time information is generated to provide a consistent data basis for subsequent parsing, analysis and reasoning; the model context information is used to participate in decision-making reasoning during the structured data generation and analysis processing, thereby realizing a continuous processing flow from web page acquisition to result generation, and improving the relevance of data processing and the accuracy of decision output.

[0110] In one embodiment, the above step S20 includes:

[0111] S201, filtering web page addresses that meet current business requirements from a preset target web page address set to construct a web page address set;

[0112] S202, analyzing the update frequency and source authority tag of each web page address in the web page address set, and generating a web page update timeliness parameter and a source characteristic parameter respectively;

[0113] S203, analyzing the crawling priority weight of each web page address according to the web page update timeliness parameter and the source characteristic parameter;

[0114] S204, prioritizing the web page address set based on the crawling priority weights, and generating a crawling task allocation table;

[0115] S205, according to the priority order of the crawling tasks in the crawling task allocation table and the current server load status, the crawling tasks are allocated to the crawling agents according to a preset allocation strategy.

[0116] In this embodiment, the scheduling agent is configured to dynamically identify and schedule web crawling tasks with high information density and decision-making value within a frequently changing web environment. The scheduling agent first accesses a pre-defined set of target web addresses, typically drawn from a manually maintained industry knowledge base, a list of known high-value sources, or a pool of addresses generated based on historical task records. A screening mechanism then integrates current business needs, such as filtering on keywords like tariff policies and medical advisories, to construct a set of web addresses tailored to the current task objectives.

[0117] The scheduling agent performs update frequency and source feature analysis on each web page in the constructed set of web page addresses. Update frequency can be determined by comparing historical content pre-crawled by the crawler with metadata, or calculated based on RSS timestamps and the Last-Modified field in the server response header. Source authority tags are generated based on the site domain name (e.g., .gov, .edu) or content attribute tags and quantified into source feature parameters through a credibility scoring mechanism.

[0118] The webpage update timeliness parameters and source feature parameters are then used to calculate crawl priority weights. This weighting can be based on a linear weighting model, fuzzy logic reasoning, or a machine learning-based scoring model. For example, pages from websites with frequent updates and strong source authority should be given higher weights than pages from websites with unstable content or questionable sources.

[0119] According to the crawling priority weight, the system sorts all the addresses in the web address set to form a crawling task allocation table, which not only contains the sorting information, but also marks the web address of each task, task label, estimated resource consumption of the task and other information.

[0120] Considering both system stability and crawling performance, the scheduling agent determines specific task allocation strategies based on the current server load. For example, it can employ a "lowest load first" or "task content classification + resource partitioning" mechanism to assign different types of tasks to nodes with corresponding loads. Tasks are ultimately delivered to the crawling agent via an asynchronous interface or message queue, which then executes the tasks accordingly.

[0121] In one implementation, the scheduling agent performs regular screening on the content in the target web page address set based on keyword configuration rules, removes content links that are not related to the business scenario, and only retains web page addresses that meet industry labels (such as "tariffs", "clinical notifications", and "transaction supervision"). For the update frequency parameter, the update score of the web page can be constructed by periodically capturing the version snapshots of the same web page in the past seven days, calculating the number of changes; for the source feature parameter, the domain name authority level of the website where it is located (such as .gov is 1.0, .org is 0.8, .com is 0.6, etc.) plus the credibility level of the site in the knowledge graph can be referenced. The calculation of task priority weights can be implemented through a weighting function, for example:

[0122] Priority weight = 0.6 × source confidence + 0.4 × update time frequency score

[0123] After the sorting is completed, the system can obtain the current CPU usage and task buffer depth of each node based on the server status query interface, and select the "lowest load + similar tasks merged first" method based on the scheduling strategy to assign high-frequency changing tasks to nodes with sufficient computing power, and try to merge tasks with similar sources to reduce duplicate crawling.

[0124] This embodiment constructs crawling task priorities by taking web page update frequency and source authority as comprehensive evaluation dimensions. It can dynamically identify web page resources with strong timeliness and high information density, and intelligently allocate crawling tasks based on the server operation status, thereby improving the overall resource utilization of the system and the efficiency of key data acquisition, avoiding redundant crawling and resource bottlenecks, and achieving priority coverage of high-value web page data, providing a more timely and highly confident data source for subsequent information analysis and reasoning decisions.

[0125] In one embodiment, the above step S30 includes:

[0126] S301, using the crawling agent to load the target web page corresponding to the crawling task based on browser simulation driving;

[0127] S302, monitoring the webpage data loading status driven by the browser simulation and capturing a webpage loading completion event;

[0128] S303, in response to the webpage loading completion event, extracting the rendered hypertext markup language code, script execution result and asynchronous response data of the target webpage from the browser simulation driver;

[0129] S304: Generate original webpage data including the rendered hypertext markup language code, script execution results, and asynchronous response data.

[0130] In this embodiment, the configuration of the crawling agent is intended to achieve complete acquisition of all valid content in the target web page when faced with the complex dynamic loading logic of modern web pages. During the execution of the crawling task, the crawling agent drives the browser simulation module to initiate a loading request. This module simulates the behavior of a real user browser, including executing page scripts, triggering asynchronous requests, and building the page DOM structure. The browser simulation driver is usually built based on a headless browser such as the Chromium kernel, and has high-fidelity page rendering capabilities.

[0131] During the loading process, the crawler agent sets a loading status listener to monitor whether key elements or resources on the page have finished rendering. This monitoring action can be based on whether a specific DOM node is accessible, whether the network request has fully completed, or whether a JavaScript event has triggered. The loading completion event is a key trigger for the page rendering to enter a stable state. The crawler agent must accurately determine this timing to avoid prematurely extracting page data before the content is fully generated.

[0132] After the event is triggered, the crawling agent obtains information of three core dimensions from the browser simulation driver: rendered Hypertext Markup Language code (i.e., HTML DOM after JavaScript rendering), script execution results (including nested calculation expression output, variable area filling content, etc.) and asynchronous response data (such as JSON or XML data loaded asynchronously through XHR, fetch, etc.).

[0133] These three types of data together constitute the raw data of a webpage, serving as the foundation for subsequent structured parsing and model analysis. HTML code is used to construct the static content structure, script execution results reflect page changes after interactive logic, and asynchronous response data provides real-time update information. These three types of data are combined to generate standardized raw data units for the webpage.

[0134] In its implementation, the crawling agent invokes tools such as Headless Chrome to load the target URL via the WebDriver protocol, setting a maximum rendering wait time and target element identification criteria. For example, loading can be considered complete when an element with the class name price-container appears on the page. The listener uses document.readyState and the page's network request completion signal to determine the page's status. After loading is complete, it calls the interface provided by the simulation driver to extract the complete DOM tree structure, using document.documentElement.outerHTML to capture the rendered HTML. It then uses browser console interfaces such as window.__RENDERED_DATA__ to retrieve the script output variable. It also monitors fetch / XHR request responses, intercepting the response content using the Chrome DevTools Protocol API and extracting the returned structured data (such as JSON objects), including table data, nested lists, and chart source data. Ultimately, these three types of data are packaged and encapsulated into the original webpage data structure, enabling subsequent modules to identify fields, reconstruct content, and analyze the page's origin. This data structure is stored in JSON format, with each field clearly distinguishing its source (DOM, JS, or XHR) and recording the crawl timestamp and target webpage address.

[0135] This embodiment fully simulates the user's browser behavior, so that the crawling agent can not only access static page content, but also extract script-generated data and asynchronously load content in a dynamic rendering environment, significantly improving the integrity of content acquisition for web pages with complex structures and frequent interactions. It is especially suitable for content scenarios where key information relies on JavaScript generation or needs to be triggered by user behavior, providing an accurate and rich source of original data for subsequent data structure parsing and analysis models.

[0136] In one embodiment, after the above step S40, the method further includes:

[0137] S4001, classifying the webpage original data and the model context information according to the source domain name and the crawling timestamp, and storing them in a distributed file storage module;

[0138] S4002, calling a parallel processing model to perform noise data cleaning processing on the original web page data;

[0139] S4003, associating the cleaned web page original data with the model context information and storing them in the distributed file storage module;

[0140] S4004, in response to the data reading request of the parsing agent, read the target data from the distributed file storage module and transfer it to the parsing agent.

[0141] In this embodiment, after the webpage raw data and model context information are combined, they must enter a stable storage and cleaning process to provide neatly structured and traceable data input for the subsequent parsing module. The webpage raw data is a comprehensive data unit extracted from the crawling module, including page rendering results, script execution output, and asynchronous request responses. The model context information, including the webpage source information and its corresponding crawling timestamp, has strong content credibility and time reference value. Together, the two form the traceability foundation for subsequent structured processing.

[0142] Web page raw data and model context information are categorized by source domain name and crawl timestamp to achieve logical organization and hierarchical management of stored data. Source domain names can be extracted through URL parsing or HTTP header information, for example, "www.cdc.gov" is categorized as a source type of "National Center for Disease Control." Crawl timestamps, derived from system records of completed crawls and accurate to the second, are used for time series data analysis and data cleansing strategy matching. After categorization, the storage directory structure can adopt a two-layer path combination, such as / source=gov_cdc / date=20250422 / , to facilitate efficient retrieval and scheduling.

[0143] The connected distributed file storage module is typically based on a scale-out file system architecture for large-scale web data, such as HDFS, MinIO, or object storage services, achieving decentralized management through key-value path mapping. After the raw web data is written, content-level noise cleaning is required to further improve parsing efficiency and data quality.

[0144] The parallel processing model handles high-throughput data preprocessing, and its architecture supports hybrid stream and batch processing frameworks such as MapReduce or Spark Streaming. Cleaning tasks include identifying and removing advertising tags, decorative HTML elements unrelated to business fields, and known invalid DOM nodes (such as "footer" and "side-banner"). This is done using regular expressions and HTML parsing libraries (such as BeautifulSoup or jsoup) for cleaning and filtering.

[0145] After cleaning, the raw webpage data is reassociated with the previously generated model context and written back into the distributed file storage module using the associated structure. This data reconstruction ensures that the parsing module not only retrieves the clean content when it subsequently reads the data, but also obtains the corresponding time and source context for intelligent template matching and rule filtering.

[0146] During the parsing request phase, the parsing agent specifies the data source and time parameters through the data reading interface, triggering the loading of the target data from the storage module. The data reading mechanism supports features such as lazy loading, path filtering, and field mapping, ensuring that the parsing module reads data files that match the current task, completing the end-to-end closed-loop data flow.

[0147] In a specific implementation, the original data of the web page can be generated by the crawling agent and then encapsulated into a JSON object according to structured fields. The fields include HTML structure, script output, asynchronous data, etc. The model context information also uses structured fields to describe the source domain name (such as "finance.yahoo.com") and the crawling timestamp (such as "2025-04-22T15:33:10Z"). Call the distributed storage interface (such as HDFS API) and use the path / source=finance.yahoo.com / date=20250422 / as the key to write the original data and context information to the corresponding directory. Subsequently, the batch HTML content cleaning task is submitted through MapReduce, and the HTML parser is used to remove non-business-related parts of the node structure. At the same time, blocks marked as advertisements or dynamically inserted content are identified to ensure that the retained data only contains the main text content and core data segments. The cleaned web page original data is repackaged and combined with its original context information into an extended data unit, and rewritten back to the original path or stored in a new path such as / source=finance.yahoo.com / date=20250422 / cleaned / . This provides the parsing agent with a more timely and semantically consistent data source. During the parsing task initiation phase, the parsing agent sends information containing the source domain name and time period (such as the source is "finance.yahoo.com" and the time is "2025-04-22") to the file storage module through a parameterized API. The storage module returns all matching files under the path and completes the reading, submitting the data containing the HTML body, field list and corresponding time context to the parsing engine.

[0148] This embodiment establishes a categorized storage structure for webpage raw data and model context information, combined with a parallel cleansing model and context rebinding mechanism. This system effectively manages and controls raw data in an orderly manner, ensuring the availability, accuracy, and timeliness of data during the parsing phase without sacrificing data breadth. This processing chain reduces the data filtering burden on the parsing agent during runtime, improves overall processing speed, and ensures data credibility, contributing to the construction of a reusable structured data foundation for multiple scenarios.

[0149] In one embodiment, the above step S50 includes:

[0150] S501, matching a target parsing template in a predefined parsing strategy library according to the web page source characteristics in the model context information by the parsing agent;

[0151] S502, constructing a document object model structure of the original web page data based on the rendered hypertext markup language code in the original web page data by the parsing agent;

[0152] S503: Based on the target data field list in the target parsing template, traverse the document object model structure to locate key information nodes corresponding to each target data field in the target data field list;

[0153] S504, extracting text content and attribute fields from the key information node, and verifying format consistency of the text content and attribute fields;

[0154] S505: Encapsulate the verified text content and attribute fields into structured data according to a preset format of the target parsing template.

[0155] In this embodiment, constructing a page structure representation is the core operation in the parsing phase. Its essence is to perform hierarchical parsing of the rendered Hypertext Markup Language code in the original web page data to generate a structured representation for data field positioning. This structure generally adopts the Document Object Model, that is, by parsing the HTML structure to generate a node tree with parent-child node and sibling node relationships, which can express the semantics and positional relationship of each part of the page. On this basis, the parsing agent uses the web page source features recorded in the model context information as the index basis for the parsing strategy, and matches the target parsing template from the predefined parsing strategy library.

[0156] Web page source features typically include website domain names, page path structures, or specific source tags. For example, "www.cdc.gov" represents a public health information source, while "finance.yahoo.com" represents a financial news source. In the parsing strategy library, source features are used as search keys to configure targeted structure templates and field mapping rules. The target parsing template includes a list of target data fields, such as "Product Name," "Tariff Rate," and "Effective Date," and specifies their common representation and location paths within the page structure.

[0157] Building the Document Object Model (DOM) structure relies on an HTML parsing engine to extract tags, categorize attributes, and create hierarchical models from rendered code. Each HTML element in this structure is mapped to a node in a tree, containing its tag name, attribute dictionary, text content, and a collection of child nodes. Common structural identifiers within nodes include ID, class, and nesting depth, which serve as crucial indicators for locating information.

[0158] Based on the target data fields recorded in the target parsing template, the parsing agent conducts a deep traversal of the document object model structure. Whenever a node is found whose structural path matches the target field description, it is considered a key information node. Matching rules can use XPath expressions, CSS Selectors, or machine learning-based feature models to determine whether it is the target node.

[0159] After identifying key information nodes, the parsing agent extracts text content, such as "Tariff rate is 25%," and attribute fields, such as "Product type = Electromechanical," recorded in the data-label attribute. The extracted results are then validated using regular expressions, content templates, or field validation rules to ensure their format meets pre-defined requirements, such as ensuring that the tax rate field is in percentage format and that the time field is a standardized timestamp.

[0160] The content with consistent format is then integrated into structured data units, usually encapsulated in JSON or key-value pair format, and uniformly named according to the field names set in the target parsing template, thereby generating a data entity with a standard structure and controllable fields for use in the subsequent analysis stage.

[0161] One implementation method is to analyze the domain name of the web page recorded in the context information of the model loaded by the agent, which is "finance.yahoo.com", and match the corresponding parsing template in the parsing strategy library. The "Securities Code" field in the template is usually located at <span class＝"ticker">Within the node, the "Current Price" field is located in <div class＝"price-container">The parsing agent then calls the HTML parser to perform DOM modeling on the rendered HTML. The generated structure records the hierarchical structure and attributes of all page tags. By traversing the DOM tree and matching field description rules, the parsing agent identifies key information nodes containing the target field among multiple nodes. After identifying the node corresponding to the "price" field, the parsing agent extracts its innerText value "152.37" and verifies whether the content is in floating-point format. If the verification passes, the value is encapsulated into a structured data entity with the field name "current_price".

[0162] This embodiment greatly improves the efficiency and accuracy of extracting valid data from multi-source heterogeneous web pages by converting the original web page content into a standardized structured data format. By using the document object model to express the page structure hierarchy, the parsing agent can accurately locate the position of the required fields in the page, avoiding the performance loss caused by full-text traversal. At the same time, the parsing strategy matching mechanism implemented by combining source characteristics and template rules makes it highly adaptable and stable when facing changes in different website structures. The format verification mechanism effectively improves the reliability and consistency of the parsed data, providing high-quality data input for subsequent analysis and decision-making.

[0163] In one embodiment, the above step S60 includes:

[0164] S601, reading historical structured data associated with the structured data from a database;

[0165] S602, performing statistical correlation analysis on the structured data and historical structured data to generate a statistical indicator set;

[0166] S603, identifying trend changes and periodic patterns of target data fields based on the statistical indicator set and the time series characteristics of the historical structured data;

[0167] S604, adapting the trend changes and periodic laws to different decision-making scenarios based on preset multi-business scenario configuration parameters to generate scenario-based analysis results;

[0168] S605: Associating the scenario analysis results with the timeliness and source tags in the model context information to generate multi-scenario decision data.

[0169] In this embodiment, the analysis agent must first determine the source and format of the structured data during its analysis process. This data is typically generated by the parsing agent during pre-processing and has standard fields, a unified format, and contextual annotations, making it readily available for mathematical analysis. Fields in structured data can include price, tax rate, timestamp, category name, and more.

[0170] The first step in the analysis process is to retrieve historical structured data from the database that is related to the current structured data. Relatedness refers to historical samples that exhibit consistency in field content or sequential continuity over time. For example, if the structured data includes fields such as "Product Name" and "Tariff Rate," the historical data must contain at least the same or equivalent fields, such as "Product Type" and "Adjustment Ratio."

[0171] Statistical correlation analysis aims to measure the numerical relationships and trends between current structured data and historical structured data. This process can employ statistical metrics such as mean, variance, covariance, and Pearson correlation coefficient to form a set of statistical indicators. The mean reflects the baseline level of historical data, the variance describes the range of fluctuation, and the correlation coefficient assesses the strength of the association between the current field and the target dependent variable. For example, the negative correlation between tax rate changes and export volume can be quantified using the correlation coefficient.

[0172] Based on a set of statistical indicators, the analytical agent further explores the time series characteristics of the target field in the structured data. Time series characteristics refer to the trend, seasonality, or cyclical patterns exhibited by data points over time. This step is typically accomplished using methods such as sliding window statistics, period mean comparison, or frequency domain analysis. Specific methods include but are not limited to Bollinger Bands analysis, smoothed exponential regression, or spectrum-based cycle detection algorithms. Through this processing, the system can identify data behavioral patterns such as "monotonically rising prices" or "quarterly tariff adjustments."

[0173] Multi-scenario configuration parameters are a set of interpretation frameworks and rule-mapping logic pre-configured by the analytical agent based on different application scenarios. They are used to interpret these trend and cyclical characteristics into meaningful decision-making indicators. These parameters take the form of business tags, priorities, threshold models, and response mechanisms. Based on these pre-configured scenario parameters, the analytical agent maps trend changes or cyclical patterns into scenario-specific analysis results. For example, "abnormal price increases" are mapped as "high-risk investment warnings," and "cyclical tariff fluctuations" are mapped as "phased import adjustment recommendations."

[0174] At this stage, the timeliness and source tags within the model context are used to assign temporal weights and source confidence levels to the analysis conclusions. Timeliness tags assess the temporal proximity between current structured data and historical data, while source tags measure the data's credibility. For example, conclusions from government websites are prioritized over those from commercial websites. The final multi-scenario decision data combines the analysis results with the contextual tags, forming structured decision support inputs that can be processed by subsequent reasoning modules.

[0175] In a real-world application, structured data contains fields for "medical device category," "import tariff rate," and "change time." The analysis agent retrieves historical tax rate change records for the same device category over the past three years from the database, selecting a subset of historical data covering the same broad device category. This historical structured data is combined with current data to calculate the average tax rate and fluctuation range, resulting in statistical indicators such as "average tax rate 8.2%" and "variance 2.1." Further analysis of the time series reveals a peak in tax rate adjustments in the fourth quarter of each year. Given that the current time is the end of the third quarter, the analysis agent determines that the current time is a critical point before a potential change. Based on preconfigured parameter tables for multiple business scenarios, such as finance, customs, and healthcare, the analysis agent maps this cyclical pattern into scenario-based analysis results, such as "recommending early completion of import procedures" or "assessing the financial impact." Furthermore, because the data source is the Ministry of Finance official website, the source label is assessed as "high confidence," and the timeliness label indicates "updated this month," giving increased weight to the analysis results. The final multi-scenario decision data generated includes "Tax rate change cycle: seasonal", "Response strategy: early customs clearance", "Confidence level: high", and "Recommendation output: trigger the cross-border payment pre-examination module".

[0176] This embodiment leverages the combined analysis of structured and historical data to empower the system with trend mining and cycle identification capabilities, eliminating reliance on static rules. The analytical agent not only assesses data variance but also integrates various business scenario logic to achieve horizontal adaptation and policy mapping. By integrating analysis results with contextual tags, the decision-making system can dynamically assess data timeliness and source reliability, improving the accuracy and applicability of overall response strategies.

[0177] In one embodiment, after the above step S70, the method further includes:

[0178] S801, configuring a visualization agent, and receiving the structured data and decision results through the visualization agent;

[0179] S802, in response to a screening condition input by a user terminal, generating a visual chart based on the structured data and the decision result;

[0180] S803, rendering the visual chart into an interactive display interface that supports zooming and filtering operations;

[0181] S804: Output the interactive display interface to the user terminal.

[0182] In this embodiment, configuring a visualization agent is a prerequisite for enabling terminal data interaction. This requires initializing a component set consisting of a graphics rendering module, an interaction control module, and a data binding module. This component set is deployed in a front-end-callable runtime environment, with the ability to adapt to multiple terminal rendering engines. This configuration action is typically triggered automatically after back-end decision reasoning is complete, or dynamically loaded through a user interface.

[0183] When the visualization agent receives structured data and decision results, it parses the extracted label fields, statistical indicators, and data dimension information from the structured data and combines them with the classification or prediction output generated by the decision results to form a unified data display benchmark. During this process, a field mapping strategy is used to ensure that the display fields are consistent with the decision logic fields to avoid information drift.

[0184] To respond to filter conditions entered by the user, we need to parse input content such as time intervals, category filters, and value ranges and dynamically bind them to the chart drawing logic. This process is usually triggered by an event listening mechanism, such as when the user drags the timeline or toggles a drop-down option.

[0185] Visual charts must be generated based on the filtered data and graphics rendering configuration described above. Chart rendering engines (such as Plotly and ECharts) must be used to construct chart structures that adapt to the terminal resolution and interaction habits. These charts include, but are not limited to, bar charts, line charts, scatter plots, and heat maps. Chart interactions must support mechanisms such as zooming, filtering, and click-to-click interaction, ensuring immediate feedback through event monitoring and dynamic data refresh.

[0186] The final interactive display interface is rendered into a structure recognizable by the front-end through a unified rendering module and output to the user's terminal. "Output" here includes HTTP response, WebSocket push, or embedded component mounting, which is dynamically selected based on the user's terminal operating environment.

[0187] In actual deployment, the visualization agent is built on front-end frameworks (such as Vue and React) and embedded in the main application page in a componentized manner. After the system completes decision reasoning, the server calls the chart generation API, encodes the structured data and decision results into JSON format, and pushes it to the front-end component via a REST API. The front-end component parses the structured data, identifying the chart field types, classification dimensions, and timeline dimensions. For securities analysis, for example, the system can map the "Industry" field as the legend, plot the "Price" field as the Y-axis value, and distribute the "Date" field on the X-axis. Decision results, such as "High Volatility Alert," can be displayed using color or annotations. Users select "Time Range: Last 7 Days" and "Industry: Finance" in the visualization interface, triggering the front-end data filtering logic. The agent re-filters the data based on these conditions and triggers the chart redraw function, presenting the decision information chart from the user's desired perspective. The display interface also adapts to different device sizes through a responsive layout, such as using an accordion control and vertical scrolling layout on mobile devices and a full legend and horizontal display area on desktop devices.

[0188] Example: When a securities analysis firm monitors the impact of US tariff policies on the securities market in real time, a scheduling agent first selects webpage addresses from tariff information portals and financial news platforms and assigns tasks to multiple crawling agents based on the website's update frequency and trustworthiness. The crawling agent uses browser emulation technology to sequentially load the webpages, extracting information such as the tariff adjustment table rendered by JavaScript and asynchronously returned market reaction data, generating the webpage's raw data, including HTML source code and structured response content. Subsequently, the parsing agent selects a matching template from a template library based on the website type, constructs a document object model (DOM), extracts fields such as tariff rates, policy release date, and affected commodities, and encapsulates them as structured data. The analysis agent further combines industry securities price fluctuations and historical tariff policies over the past three years to identify price-sensitive industries through time series modeling and indicator calculation. This data is then applied to various business scenarios, such as investment decision-making and risk warning. The decision-making inference module considers the data's source (e.g., government official website) and timeliness to output different strategy recommendations. Finally, financial analysts use a visual interface to dynamically view price curves and recommended strategies for different industries and adjust their investment portfolios.

[0189] When a medical insurance system tracks the impact of regional medical insurance policy updates on the drug supply chain, the scheduling agent first constructs a collection of web page addresses from the official websites of provincial and municipal medical insurance bureaus and medical policy announcement websites. It analyzes the source stability and release frequency, dynamically generates a task schedule, and assigns it to the crawling agent. The crawling agent automatically accesses these pages, extracts page elements such as the latest changes to the medical insurance reimbursement catalog, drug price adjustments, and policy release dates, and generates the raw web page data. The parsing agent classifies and matches medical insurance templates based on source classification, parsing and structuring content such as reimbursement drug codes and quota policy fields from HTML. The analyzing agent compares historical medical insurance adjustment data from across the country, identifying patterns in reimbursement time intervals and price fluctuations for specific drugs in different provinces, and maps these patterns to decision-making scenarios such as hospital procurement forecasts and medical insurance fund allocation. The decision reasoning module outputs a list of affected drugs and a recommended procurement cadence, which is displayed graphically to the terminal system, enabling transparent tracking of medical insurance policies and dynamic response from medical service providers.

[0190] This embodiment deploys a visualization agent and graphically binds structured data to decision results, enabling users to directly perceive data trends, classification characteristics, and decision recommendations on the terminal, significantly improving decision transparency and efficiency. This not only lowers the barrier to data understanding but also enables data-driven proactive interaction, facilitating cross-scenario data interpretation and comparison.

[0191] In one embodiment, an agent-based web page information processing device is provided, which corresponds one-to-one to the agent-based web page information processing method in the above embodiment. Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the agent-based web page information processing device of the present invention. These include a configuration module 10, a scheduling module 20, a crawling module 30, a context generation module 40, a parsing module 50, an analysis module 60, and an inference module 70. Each functional module is described in detail below:

[0192] Configuration module 10, used to configure the scheduling agent, the grasping agent, the parsing agent, the analysis agent and the decision-making and reasoning module;

[0193] The scheduling module 20 is configured to construct a web page address set through the scheduling agent, generate a crawling task allocation table based on the web page update timeliness and source characteristics of each web page in the web page address set, and send the crawling tasks in the crawling task allocation table to the crawling agent;

[0194] The crawling module 30 is configured to execute the crawling task through the crawling agent and obtain the original data of the web page in response to the web page data loading completion event;

[0195] A context generation module 40 is configured to extract webpage source information and publishing time information from the webpage original data to generate model context information;

[0196] A parsing module 50 is configured to construct a page structure representation of the original web page data through the parsing agent, locate key information nodes in the original web page data based on the page structure representation, and extract key information from the original web page data to generate structured data;

[0197] An analysis module 60 is configured to perform analysis based on the structured data through the analysis agent to generate multi-scenario decision data;

[0198] The reasoning module 70 is used to generate a decision result based on the multi-scenario decision data and the model context information through the decision reasoning module.

[0199] In one embodiment, the scheduling module 20 is specifically configured to:

[0200] Filtering web page addresses that meet current business needs from a preset target web page address set to construct a web page address set;

[0201] Analyzing the update frequency and source authority tag of each web page address in the web page address set to generate a web page update timeliness parameter and a source characteristic parameter respectively;

[0202] Analyze the crawling priority weight of each web page address based on the web page update timeliness parameter and source characteristic parameter;

[0203] Prioritizing the web page address set based on the crawling priority weights to generate a crawling task allocation table;

[0204] According to the priority order of the crawling tasks in the crawling task allocation table and the current server load status, the crawling tasks are allocated to the crawling agents according to a preset allocation strategy.

[0205] In one embodiment, the capture module 30 is specifically configured to:

[0206] By means of the crawling agent, a target web page corresponding to the crawling task is loaded based on a browser simulation drive;

[0207] Monitor the web page data loading status driven by the browser simulation and capture the web page loading completion event;

[0208] In response to the webpage loading completion event, extracting the rendered hypertext markup language code, script execution results, and asynchronous response data of the target webpage from the browser simulation driver;

[0209] Generating web page original data including the rendered hypertext markup language code, script execution result and asynchronous response data.

[0210] In one embodiment, the context generation module 40 is specifically configured to:

[0211] Classifying the web page original data and the model context information according to the source domain name and the crawling timestamp, and storing them in a distributed file storage module;

[0212] Calling a parallel processing model to perform noise data cleaning processing on the original web page data;

[0213] The cleaned web page original data is associated with the model context information and stored in the distributed file storage module;

[0214] In response to the data reading request of the parsing agent, the target data is read from the distributed file storage module and transmitted to the parsing agent.

[0215] In one embodiment, the parsing module 50 is specifically configured to:

[0216] Matching a target parsing template in a predefined parsing strategy library by the parsing agent according to the web page source characteristics in the model context information;

[0217] Constructing a document object model structure of the web page original data based on the rendered hypertext markup language code in the web page original data by the parsing agent;

[0218] Based on the target data field list in the target parsing template, traverse the document object model structure to locate the key information node corresponding to each target data field in the target data field list;

[0219] Extracting text content and attribute fields from the key information node, and verifying format consistency of the text content and attribute fields;

[0220] The verified text content and attribute fields are encapsulated into structured data according to the preset format of the target parsing template.

[0221] In one embodiment, the analysis module 60 is specifically configured to:

[0222] Reading historical structured data associated with the structured data from a database;

[0223] Performing statistical correlation analysis on the structured data and historical structured data to generate a set of statistical indicators;

[0224] Based on the statistical indicator set and the time series characteristics of the historical structured data, identifying the trend changes and periodic laws of the target data field;

[0225] According to the preset multi-business scenario configuration parameters, the trend changes and periodic laws are adapted to different decision-making scenarios to generate scenario-based analysis results;

[0226] The scenario analysis results are associated with the timeliness and source tags in the model context information to generate multi-scenario decision data.

[0227] In one embodiment, the reasoning module 70 is specifically configured to:

[0228] Configuring a visualization agent, and receiving the structured data and decision results through the visualization agent;

[0229] In response to a screening condition input by a user terminal, generating a visual chart based on the structured data and the decision result;

[0230] Rendering the visualization chart into an interactive display interface that supports zooming and filtering operations;

[0231] Output the interactive display interface to the user terminal.

[0232] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the server side of a web page information processing method based on an intelligent agent.

[0233] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a web page information processing method based on an intelligent agent.

[0234] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0235] Configure scheduling agents, grasping agents, parsing agents, analysis agents and decision-making reasoning modules;

[0236] A web page address set is constructed by the scheduling agent, and a crawling task allocation table is generated based on the web page update timeliness and source characteristics of each web page in the web page address set, and the crawling tasks in the crawling task allocation table are sent to the crawling agent;

[0237] Executing the crawling task through the crawling agent and acquiring the original data of the web page in response to the web page data loading completion event;

[0238] Extracting webpage source information and publishing time information from the webpage original data to generate model context information;

[0239] Constructing a page structure representation of the web page original data through the parsing agent, locating key information nodes in the web page original data based on the page structure representation, and extracting key information from the web page original data to generate structured data;

[0240] The analysis agent performs analysis processing based on the structured data to generate multi-scenario decision data;

[0241] The decision reasoning module generates a decision result based on the multi-scenario decision data and the model context information.

[0242] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0243] Configure scheduling agents, grasping agents, parsing agents, analysis agents and decision-making reasoning modules;

[0244] A web page address set is constructed by the scheduling agent, and a crawling task allocation table is generated based on the web page update timeliness and source characteristics of each web page in the web page address set, and the crawling tasks in the crawling task allocation table are sent to the crawling agent;

[0245] Executing the crawling task through the crawling agent and acquiring the original data of the web page in response to the web page data loading completion event;

[0246] Extracting webpage source information and publishing time information from the webpage original data to generate model context information;

[0247] Constructing a page structure representation of the web page original data through the parsing agent, locating key information nodes in the web page original data based on the page structure representation, and extracting key information from the web page original data to generate structured data;

[0248] The analysis agent performs analysis processing based on the structured data to generate multi-scenario decision data;

[0249] The decision reasoning module generates a decision result based on the multi-scenario decision data and the model context information.

[0250] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0251] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0252] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0253] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention. < / h1>

Claims

1. A web page information processing method based on an agent, characterized in that: The following steps are involved: Configure scheduling agents, grasping agents, parsing agents, analysis agents and decision-making reasoning modules; A web page address set is constructed by the scheduling agent, and a crawling task allocation table is generated based on the web page update timeliness and source characteristics of each web page in the web page address set, and the crawling tasks in the crawling task allocation table are sent to the crawling agent; Executing the crawling task through the crawling agent and acquiring the original data of the web page in response to the web page data loading completion event; Extracting webpage source information and publishing time information from the webpage original data to generate model context information; Constructing a page structure representation of the web page original data through the parsing agent, locating key information nodes in the web page original data based on the page structure representation, and extracting key information from the web page original data to generate structured data; The analysis agent performs analysis processing based on the structured data to generate multi-scenario decision data; The decision reasoning module generates a decision result based on the multi-scenario decision data and the model context information.

2. The agent-based web page information processing method according to claim 1, wherein: The method comprises: constructing a web page address set by the scheduling agent, generating a crawling task allocation table based on the web page update timeliness and source characteristics of each web page in the web page address set, and sending the crawling tasks in the crawling task allocation table to the crawling agent, including: Filtering web page addresses that meet current business needs from a preset target web page address set to construct a web page address set; Analyzing the update frequency and source authority tag of each web page address in the web page address set to generate a web page update timeliness parameter and a source characteristic parameter respectively; Analyze the crawling priority weight of each web page address based on the web page update timeliness parameter and source characteristic parameter; Prioritizing the web page address set based on the crawling priority weights to generate a crawling task allocation table; According to the priority order of the crawling tasks in the crawling task allocation table and the current server load status, the crawling tasks are allocated to the crawling agents according to a preset allocation strategy.

3. The agent-based web page information processing method according to claim 1, wherein: The crawling agent executes the crawling task and obtains the original data of the web page in response to the web page data loading completion event, including: By means of the crawling agent, a target web page corresponding to the crawling task is loaded based on a browser simulation drive; Monitor the web page data loading status driven by the browser simulation and capture the web page loading completion event; In response to the webpage loading completion event, extracting the rendered hypertext markup language code, script execution results, and asynchronous response data of the target webpage from the browser simulation driver; Generating web page original data including the rendered hypertext markup language code, script execution result and asynchronous response data.

4. The agent-based web page information processing method according to claim 1, wherein: After extracting webpage source information and publishing time information from the webpage original data to generate model context information, the method further includes: Classifying the web page original data and the model context information according to the source domain name and the crawling timestamp, and storing them in a distributed file storage module; Calling a parallel processing model to perform noise data cleaning processing on the original web page data; The cleaned web page original data is associated with the model context information and stored in the distributed file storage module; In response to the data reading request of the parsing agent, the target data is read from the distributed file storage module and transmitted to the parsing agent.

5. The agent-based web page information processing method according to claim 1, wherein: Constructing a page structure representation of the webpage original data by the parsing agent, locating key information nodes in the webpage original data based on the page structure representation, and extracting key information from the webpage original data to generate structured data, including: Matching a target parsing template in a predefined parsing strategy library by the parsing agent according to the web page source characteristics in the model context information; Constructing a document object model structure of the web page original data based on the rendered hypertext markup language code in the web page original data by the parsing agent; Based on the target data field list in the target parsing template, traverse the document object model structure to locate the key information node corresponding to each target data field in the target data field list; Extracting text content and attribute fields from the key information node, and verifying format consistency of the text content and attribute fields; The verified text content and attribute fields are encapsulated into structured data according to the preset format of the target parsing template.

6. The agent-based web page information processing method according to claim 1, wherein: The analysis agent performs analysis processing based on the structured data to generate multi-scenario decision data, including: Reading historical structured data associated with the structured data from a database; Performing statistical correlation analysis on the structured data and historical structured data to generate a set of statistical indicators; Based on the statistical indicator set and the time series characteristics of the historical structured data, identifying the trend changes and periodic laws of the target data field; According to the preset multi-business scenario configuration parameters, the trend changes and periodic laws are adapted to different decision-making scenarios to generate scenario-based analysis results; The scenario analysis results are associated with the timeliness and source tags in the model context information to generate multi-scenario decision data.

7. The agent-based web page information processing method according to claim 1, wherein: After the decision reasoning module generates a decision result based on the multi-scenario decision data and the model context information, the method further includes: Configuring a visualization agent, and receiving the structured data and decision results through the visualization agent; In response to a screening condition input by a user terminal, generating a visual chart based on the structured data and the decision result; Rendering the visualization chart into an interactive display interface that supports zooming and filtering operations; Output the interactive display interface to the user terminal.

8. A web page information processing device based on an agent, characterized in that: The agent-based web page information processing device includes: Configuration module, used to configure the scheduling agent, grasping agent, parsing agent, analysis agent and decision reasoning module; A scheduling module, configured to construct a web page address set through the scheduling agent, generate a crawling task allocation table based on the web page update timeliness and source characteristics of each web page in the web page address set, and send the crawling tasks in the crawling task allocation table to the crawling agent; A crawling module, configured to execute the crawling task through the crawling agent and obtain the original data of the web page in response to a web page data loading completion event; A context generation module, configured to extract webpage source information and publishing time information from the webpage original data to generate model context information; a parsing module configured to construct a page structure representation of the original web page data through the parsing agent, locate key information nodes in the original web page data based on the page structure representation, and extract key information from the original web page data to generate structured data; An analysis module, configured to perform analysis processing based on the structured data through the analysis agent to generate multi-scenario decision data; An inference module is used to generate a decision result based on the multi-scenario decision data and the model context information through the decision inference module.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and an agent-based web page information processing program stored in the memory and capable of running on the processor. When the agent-based web page information processing program is executed by the processor, the steps of the agent-based web page information processing method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores an agent-based web page information processing program, which, when executed by a processor, implements the steps of the agent-based web page information processing method according to any one of claims 1 to 7.

Images