Method and device for collecting detail page of bidding source, electronic equipment, medium and product

Abstract

This invention provides a method, apparatus, electronic device, medium, and product for collecting details pages of bidding sources, relating to the field of information collection technology. The method includes: upon obtaining a target bidding source task from a message queue based on a browser instance, determining list page information based on the target bidding source task; generating an initial path expression based on the list page information; and, if the initial path expression is verified to be valid, determining the target details page based on the initial path expression. The technical solution of this invention, by adaptively generating the initial path expression, ultimately achieves the extraction of the target details page, reducing reliance on manually configured path expressions and maintaining a high success rate in collecting the target details page even when the webpage structure changes.

Description

Technical Field

[0001] This invention relates to the field of information collection technology, and in particular to a method, apparatus, electronic device, medium, and product for collecting details from bidding sources. Background Technology

[0002] Currently, most mainstream web scraping technologies adopt a single-process or simple multi-threaded architecture, which often suffers from efficiency bottlenecks and uneven resource scheduling when processing large-scale target sites.

[0003] Currently, in existing technologies, the data extraction process usually relies on manually pre-configuring path expressions (XPath expressions) to locate target data. The maintenance cost of manually configuring XPath is high, and it also lacks the ability to adapt to dynamic changes in the webpage structure.

[0004] Therefore, there is an urgent need for a method to collect details pages from bidding sources. This method should adaptively generate initial path expressions to ultimately extract the target details page, reduce reliance on manual XPath configuration, and maintain a high success rate in details page collection even when the webpage structure changes. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, medium, and product for collecting details pages of bidding sources. It addresses the shortcomings of existing technologies where data extraction typically relies on manually pre-configured path expressions (XPath expressions) to locate target data. Manual XPath configuration is costly to maintain and lacks adaptability to dynamic changes in webpage structure. This invention achieves the extraction of target details pages by adaptively generating initial path expressions, reducing reliance on manual XPath configuration and maintaining a high success rate in collecting target details pages even when the webpage structure changes.

[0006] This invention provides a method for collecting details from bidding sources, comprising the following steps.

[0007] If the target tender source task to be collected is obtained from the message queue based on the browser instance, the list page information is determined based on the target tender source task to be collected; Generate an initial path expression based on the list page information; If the initial path expression is verified to be valid, the target detail page is determined based on the initial path expression.

[0008] According to the present invention, a method for collecting details pages of bidding sources generates an initial path expression based on list page information, comprising: cleaning the list page information to obtain candidate list page information; and performing structural statistics on the candidate list page information to generate an initial path expression.

[0009] According to the present invention, a method for collecting details pages of bidding sources further includes: if the initial path expression is found to be invalid, re-parseing the list page information to generate candidate path expressions and verifying whether the candidate path expressions are valid; if the candidate path expressions are verified to be valid, collecting the target details page of the target bidding source task according to the candidate path expressions; if the candidate path expressions are verified to be invalid, continuing to execute the step of re-parseing the list page information.

[0010] According to the present invention, a method for collecting details of a tender source includes, before obtaining the target tender source task to be collected from the message queue based on the browser instance, the method further includes: determining the task priority of the tender source task to be collected; and writing the tender source task to be collected into the message queue according to the task priority.

[0011] The method for collecting details of a bidding source according to the present invention further includes: obtaining the queue occupancy rate and resource utilization rate of the message queue; and adjusting the number of browser instances based on the queue occupancy rate and resource utilization rate.

[0012] According to the present invention, a method for collecting details pages of bidding sources, determining a target details page based on a candidate path expression, includes: obtaining the time field of each list item in the candidate path expression; scoring the candidate nodes in each list item and determining the scoring result if the time field is greater than or equal to a preset time threshold; generating a target path expression based on the scoring result; and determining the target details page based on the target path expression.

[0013] This invention also provides a device for collecting details pages of bidding sources, comprising the following modules: The information collection module is used to determine the list page information based on the target bidding source task when the target bidding source task is obtained from the message queue according to the browser instance. The expression generation module is used to generate initial path expressions based on list page information. The details page acquisition module is used to determine the target details page based on the initial path expression, provided that the initial path expression is valid.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for collecting details pages of any of the above-described tender sources.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for collecting details pages of any of the above-described tender sources.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a method for collecting details pages of any of the above-described tender sources.

[0017] This invention provides a method, apparatus, electronic device, medium, and product for collecting details pages of bidding sources. When a target bidding source task is obtained from a message queue based on a browser instance, list page information is determined according to the target bidding source task; an initial path expression is generated based on the list page information; and, if the initial path expression is verified to be valid, the target details page is determined based on the initial path expression. This invention addresses the shortcomings of existing technologies where data extraction typically relies on manually pre-configuring path expressions (XPath expressions) to locate target data. Manual XPath configuration is costly to maintain and lacks adaptability to dynamic changes in webpage structure. This invention achieves the extraction of the target details page by adaptively generating the initial path expression, reducing reliance on manual XPath configuration and maintaining a high success rate in collecting the target details page even when the webpage structure changes. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the method for collecting details from the tender source provided by this invention.

[0020] Figure 2 This is the second flowchart illustrating the method for collecting details from the tender source provided by this invention.

[0021] Figure 3 This is a schematic diagram of the structure of the tender source detail page collection device provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] The following is combined with Figure 1 The present invention describes a method for collecting details pages of bidding sources. This method is applicable to situations where bidding sources are sampled. The subject executing this method can be an electronic device or a bidding source details page collection device installed in the electronic device. The bidding source details page collection device can be implemented through software, hardware, or a combination of both. Figure 1 This is one of the flowcharts illustrating the method for collecting details from the tender source provided by this invention, such as... Figure 1 As shown, the method includes the following steps 101, 102 and 103.

[0025] Step 101: If the target tender source task to be collected is obtained from the message queue based on the browser instance, determine the list page information based on the target tender source task to be collected.

[0026] In this step, the tender source task to be collected refers to the tender source task whose webpage structure has changed, obtained from the message queue through the browser instance. The message queue is mainly used to store the task information and node status of the tender source task to be collected, and this embodiment does not limit this.

[0027] A browser instance refers to an instance that is pre-created according to the browser instance pool / browser context reuse. The browser instance is used to obtain list page information, which may include, for example, multiple nodes, child nodes, the statistical tag type of nodes, the number of nodes and child nodes, the text density in nodes, the connection density, etc. This embodiment does not limit this.

[0028] The list page information may be, for example, the Hyper Text Markup Language (HTML) content of the list page, but this embodiment does not limit this.

[0029] Specifically, when the bidding source task to be collected is stored in the message queue, based on the browser instance that has been created in advance according to the browser instance pool / browser context reuse, the bidding source task stored in the message queue is used to extract multiple nodes, child nodes, statistical tag types of nodes, number of nodes and child nodes, text density in nodes, connection density, etc., to determine the list page information.

[0030] In one specific embodiment, before obtaining the target bidding source task to be collected from the message queue according to the browser instance, the method further includes: determining the task priority of the bidding source task to be collected; and writing the bidding source task to be collected into the message queue according to the task priority.

[0031] In this step, the task priority of the bidding source task can be determined according to the task collection time, task urgency, task attributes, etc. The task priority can include high priority, medium priority and low priority, etc., but this embodiment does not limit it.

[0032] Specifically, after obtaining the bidding source tasks to be collected, the task priority is determined based on the collection time, urgency, and attributes of the bidding source tasks. Then, the bidding source tasks to be collected are sorted according to their corresponding task priorities. Finally, they are written into the message queue in descending order of task priority. This embodiment does not limit this process.

[0033] In one specific embodiment, before writing the tasks to be collected into the message queue according to task priority, the method further includes: electing a master node based on the distributed lock / lease and node capability score of the Redis (Remote Dictionary Server, an open-source high-performance in-memory key-value database) task, determining the master node, and after determining the master node, further writing the tasks to be collected into the message queue according to task priority based on the master node.

[0034] Specifically, before writing the tasks to be collected from the bidding sources into the message queue according to task priority, a master node election is first performed. Based on a distributed lock / lease mechanism, a master node is elected from all nodes in the bidding source tasks. When the lock is available, the candidate node with the highest node capability score among the bidding source tasks (which include at least two candidate nodes) becomes the master node. If the master node's heartbeat times out, a re-election is automatically triggered. After the master node is determined, the tasks to be collected are written into the message queue according to task priority based on the master node.

[0035] The advantage of this setup is that it allows for fine-grained task scheduling by writing tasks to be collected into a message queue based on task priority. This solves the problem of insufficient fine-grained management of task scheduling in existing technologies, where important tasks may be blocked by ordinary tasks, resulting in high overall latency.

[0036] In one specific embodiment, the method further includes: obtaining the queue occupancy rate and resource utilization rate of the message queue; and adjusting the number of browser instances based on the queue occupancy rate and resource utilization rate.

[0037] In this step, queue occupancy rate measures how many tasks are waiting in the message queue at this time, while resource utilization rate indicates the busyness of the message queue at this time. This embodiment does not limit these aspects.

[0038] Specifically, obtain the queue occupancy and resource utilization of the message queue, and then adjust the number of browser instances based on the queue occupancy and resource utilization.

[0039] For example, when the queue occupancy rate is continuously greater than or equal to the queue expansion threshold (the queue expansion threshold can be, for example, 70% of the total usage of the message queue, which is not limited in this embodiment), expansion information is determined, and the number of browser instances is increased based on the expansion signal; when the queue occupancy rate is continuously less than the queue shrinkage threshold (the queue shrinkage threshold can be, for example, 30% of the total usage of the message queue, which is not limited in this embodiment), shrinkage signal is determined, and the number of browser instances is reduced based on the shrinkage signal.

[0040] The advantage of this setup is that it enables flexible adjustment of the number of browser instances, optimizes resource usage in the browser instance cluster, and solves the problems of uncontrolled browser instance resource consumption, memory leaks, process crashes, and insufficient stability in existing technologies.

[0041] Step 102: Generate the initial path expression based on the list page information.

[0042] In this step, the initial path expression can be, for example, a candidate Extensible Markup Language Path Language (XPath).

[0043] In one specific embodiment, generating an initial path expression based on list page information includes: cleaning the list page information to obtain candidate list page information; and performing structural statistics on the candidate list page information to generate an initial path expression.

[0044] Specifically, after determining the list page information, the HTML of the list page is cleaned using the Document Object Model (DOM) to obtain a DOM tree. Then, structural statistics are performed on the cleaned results, which involves traversing the DOM tree hierarchically and calculating the tag type, depth, number of child nodes, repeating structures, text density, and link density of each node. The "list container score" of each node is calculated, and an initial path expression is generated based on the "list container score".

[0045] The advantage of this setup is that it solves the problems of high cost of manual XPath configuration in existing technologies, the need for reconfiguration after page structure changes, and poor adaptability. This invention generates initial path expressions based on DOM statistics and adaptive XPath generation mechanisms, reducing reliance on manual configuration and adapting to changes in page structure.

[0046] Step 103: If the initial path expression is verified to be valid, determine the target details page based on the initial path expression.

[0047] In this step, validating the initial path expression mainly involves extracting list items from the initial path expression to obtain target list items. The target list items are N list items extracted from the initial path expression, where N is a positive integer greater than or equal to 1. Then, it is verified whether the N list items are within a reasonable range, whether they contain title / link / time fields, and whether the links are accessible. If each list item is within a reasonable range, contains title / link / time fields, and the links are accessible, the initial path expression is determined to be valid; otherwise, the initial path expression is determined to be invalid. This embodiment does not impose any limitations on this.

[0048] Specifically, after generating the initial path expression, its validity is verified by extracting list items from it. The target list items are N list items extracted from the initial path expression, where N is a positive integer greater than or equal to 1. Then, it is verified whether the N list items are within a reasonable range, whether they contain title / link / time fields, and whether the links are accessible. If all list items are within a reasonable range, contain title / link / time fields, and the links are accessible, the initial path expression is considered valid; otherwise, it is considered invalid. If the initial path expression is valid, the target detail page is determined based on it. If the initial path expression is verified to be valid, the time field of each list item in the initial path expression is obtained. It is then determined whether the time field is greater than or equal to a preset time threshold (i.e., whether the time field is later than the preset time threshold). If the time field is less than the preset time threshold (i.e., the time field is earlier than the preset time threshold), outdated data is truncated from each list item in the initial path expression, and pagination for that list item is stopped. If the time field is greater than or equal to the preset time threshold (i.e., the time field is later than the preset time threshold), pagination elements are searched for in each list item of the initial path expression. Candidate nodes are determined based on these pagination elements, and each candidate node in each list item is scored. The scoring result for each candidate node is determined, and then a target path expression is generated based on the candidate nodes whose scoring results are greater than the preset scoring threshold. Multiple asynchronous tasks are created in the event loop based on the target path expression (the event loop creates multiple asynchronous tasks to determine the details page tasks from the target bidding source tasks in the message queue) for concurrent details page collection, thereby determining the target details page.

[0049] In this step, the pagination element may be a candidate node in the initial path expression that includes text such as "next page" or "more", or has a specific class name, identity document (ID), etc. This embodiment does not limit this.

[0050] The stale data can be, for example, data that has been stored for more than 30 days or 40 days. This embodiment does not limit this.

[0051] In a specific embodiment, determining the target detail page according to the candidate path expression includes: obtaining the time fields of each list item in the candidate path expression; when it is determined that the time field is greater than or equal to a preset time threshold, scoring the candidate nodes in each list item to determine the scoring result; generating a target path expression according to the scoring result; and determining the target detail page according to the target path expression.

[0052] In this step, the preset time threshold is a threshold preset for judging the time field. The format of the time field can be, for example, "YYYY-MM-DD", "YYYY year MM month DD day", etc. This embodiment does not limit this.

[0053] Specifically, when it is verified that the candidate path expression is valid, obtain the time fields of each list item in the candidate path expression; determine whether the time field is greater than or equal to the preset time threshold (that is, whether the time field is later than the preset time threshold). When it is determined that the time field is less than the preset time threshold (the time field is earlier than the preset time threshold), truncate the stale data in each list item of the candidate path expression and stop paging the list item; when it is determined that the time field is greater than or equal to the preset time threshold (the time field is later than the preset time threshold), search for the paging element in each list item of the candidate path expression, determine the candidate node according to the paging element, score the candidate nodes in each list item to determine the scoring result corresponding to each candidate node, and then generate a target path expression according to the candidate nodes corresponding to the scoring result greater than the preset scoring threshold. Create multiple asynchronous tasks for concurrent detail page collection according to the target path expression to determine the target detail page.

[0054] In this step, the paging element can be, for example, text such as "Next Page", "More", etc. included in the candidate nodes in the candidate path expression, or having a specific class name, identity document number (ID), etc. This embodiment does not limit this.

[0055] The stale data can be, for example, data that has been stored for more than 30 days or 40 days. This embodiment does not limit this.

[0056] In one specific embodiment, the method further includes: if the initial path expression is found to be invalid, re-parse the list page information to generate a candidate path expression and verify whether the candidate path expression is valid; if the candidate path expression is verified to be valid, collect the target details page of the target bidding source task according to the candidate path expression; if the candidate path expression is verified to be invalid, continue to execute the step of re-parse the list page information.

[0057] Specifically, if the initial path expression is found to be invalid, BeautifulSoup's structure parsing is called to re-parse the list page information based on the already obtained list page information, generating candidate path expressions and verifying their validity. If the candidate path expressions are valid, the target details page of the target bidding source task is collected according to the candidate path expressions. If the candidate path expressions are invalid, the process of re-parseing the list page information continues.

[0058] Similarly, if the candidate path expression is found to be invalid, the step of re-parses the list page information continues, and this embodiment does not limit this step.

[0059] In one specific embodiment, Figure 2 This is the second flowchart illustrating the method for collecting details from the tender source provided by this invention. Figure 2 As shown, the method for collecting details from the tender source includes steps 201, 202, 203, 204, 205, 206, 207, and 208.

[0060] Step 201: Determine the list page information.

[0061] Specifically, when the target bidding source task to be collected is obtained from the message queue based on the browser instance, the list page information is determined based on the target bidding source task to be collected.

[0062] Step 202: Generate the initial path expression based on the list page information.

[0063] Specifically, after determining the list page information, the HTML of the list page is cleaned using the Document Object Model (DOM) to obtain a DOM tree. Then, structural statistics are performed on the cleaned results, which involves traversing the DOM tree hierarchically and calculating the tag type, depth, number of child nodes, repeating structures, text density, and link density of each node. The "list container score" of each node is calculated, and an initial path expression is generated based on the "list container score".

[0064] Step 203: Verify that the initial path expression is valid.

[0065] Specifically, the initial path expression is used to extract list items to obtain target list items. The target list items are N list items extracted from the initial path expression, where N is a positive integer greater than or equal to 1. Then, it is verified whether the N list items are within a reasonable range, whether they contain title / link / time fields, and whether the links are accessible. If each list item is within a reasonable range, contains title / link / time fields, and the links are accessible, the initial path expression is verified as valid, and step 206 is executed; otherwise, the initial path expression is verified as invalid, and step 204 is executed.

[0066] Step 204: Re-parse the list page information to generate candidate path expressions.

[0067] Specifically, if the initial path expression verification fails, the list page information is re-parsed to generate candidate path expressions.

[0068] Step 205: Verify whether the candidate path expression is valid.

[0069] Specifically, the method for verifying the candidate path expression is the same as that for the initial path expression, and will not be repeated here. If the candidate path expression is invalid, proceed to step 204; if the candidate path expression is valid, proceed to step 206.

[0070] Step 206: Obtain the time field for each list item.

[0071] Specifically, after verifying the validity of the initial path expression, the time field of each list item in the initial path expression is obtained. It is determined whether the time field is greater than or equal to a preset time threshold (i.e., whether the time field is later than the preset time threshold). If the time field is less than the preset time threshold (i.e., the time field is earlier than the preset time threshold), stale data is truncated for each list item in the initial path expression, and pagination for the list item is stopped. If the time field is greater than or equal to the preset time threshold (i.e., the time field is later than the preset time threshold), step 207 is continued. Alternatively, after verifying the validity of the candidate path expression, the time field of each list item in the candidate path expression is obtained. It is determined whether the time field is greater than or equal to a preset time threshold (i.e., whether the time field is later than the preset time threshold). If the time field is less than the preset time threshold (i.e., the time field is earlier than the preset time threshold), stale data is truncated for each list item in the candidate path expression, and pagination for the list item is stopped. If the time field is greater than or equal to the preset time threshold (i.e., the time field is later than the preset time threshold), step 207 is continued.

[0072] Step 207: Score the candidate nodes in each list item, determine the scoring results, and determine the target path expression based on the scoring results.

[0073] Specifically, if the time field is greater than or equal to a preset time threshold (or if the time field is later than the preset time threshold), a pagination element is searched in each list item of the initial path expression. Candidate nodes are determined based on these pagination elements, and each candidate node in each list item is scored to determine its score. The target path expression is generated based on candidate nodes whose scores are greater than the preset scoring threshold. Alternatively, if the time field is greater than or equal to a preset time threshold (or if the time field is later than the preset time threshold), a pagination element is searched in each list item of the candidate path expression. Candidate nodes are determined based on these pagination elements, and each candidate node in each list item is scored to determine its score. The target path expression is generated based on candidate nodes whose scores are greater than the preset scoring threshold.

[0074] Step 208: Determine the target details page based on the target path expression.

[0075] Specifically, multiple asynchronous tasks are created in the event loop based on the target path expression to collect details concurrently, thereby determining the target details page.

[0076] In one specific embodiment, the implementation of the method for collecting details pages of bidding sources can be carried out using a details page collection system for bidding sources. The details page collection system for bidding sources includes: a data storage layer, a node collaboration layer, a task scheduling layer, a process management layer, a browser control layer, a page processing layer, a details page processing layer, and a monitoring and elasticity layer.

[0077] The data storage layer includes a Redis task queue / status queue, a MySQL database, and an attachment storage unit; the node coordination layer includes a node coordination unit and a master node / load component unit; the task scheduling layer includes a task distribution unit and a task priority storage unit; the process management layer includes a process management unit and a child process / heartbeat unit; the browser control layer includes a browser management unit and a resource interception unit; the page processing layer includes a list page processing unit, a pagination / load more recognition unit, and a time pattern recognition unit; the details page processing layer includes a details page processing unit, an information extraction unit, and an attachment extraction unit; the monitoring and elasticity layer includes a monitoring and elasticity unit, a queue occupancy judgment unit, and a scaling control unit, which are not limited in this embodiment.

[0078] Specifically, to determine the target details page, firstly, when the browser instance of the browser management unit in the browser control layer obtains the target bidding source task from the message queue, it determines the list page information based on the target bidding source task and passes the list page information to the list page processing unit in the page processing layer. Then, the list page processing unit performs DOM cleaning and structural statistics on the list page information, analyzing the tag type, number of child nodes, and text density of each level of nodes, and generates an initial path expression based on these parameters. Finally, the validity of the initial path expression is verified. If the initial path expression is invalid, the BeautifulSoup-based structure parsing is invoked to re-parse the list page information and verify it again. If the initial path expression is valid, the time field in the list page information is identified, and the time pattern recognition unit of the page processing layer is invoked to normalize the time field and determine the target time field. If the target time field is earlier than the preset time threshold, outdated data is truncated for each list item in the initial path expression, and pagination for the list item is stopped. If the time field is later than the preset time threshold, pagination elements are searched for in each list item of the initial path expression, and candidate nodes are determined based on the pagination elements. Each candidate node in the list item is scored, and the scoring result for each candidate node is determined. Then, candidate path expressions are generated based on the candidate nodes whose scores are greater than the preset scoring threshold. Multiple asynchronous tasks are created in the event loop based on the candidate path expressions to perform concurrent detail page collection, thereby determining the target detail page.

[0079] In one specific embodiment, the method further includes: before obtaining the target bidding source task from the message queue based on the browser instance, the master node / load component unit of the node coordination layer is mainly used to implement the election of the master node and load distribution within the node coordination unit of the node coordination layer. Specifically, the heartbeat and capability parameters of each node included in the bidding source task are periodically written to the Redis task queue / status queue of the data storage layer (the heartbeat and capability parameters may include, for example, the central processing unit (CPU) / memory, concurrency limit, failure rate, etc., which are not limited in this embodiment). The master node is elected from all nodes in the bidding source task based on the distributed lock / lease mechanism. When the lock is available, the candidate node with the highest node capability score in the bidding source task (which includes at least two candidate nodes) becomes the master node. If the master node heartbeat times out, a re-election is automatically triggered. The node coordination unit is mainly used for registering node information of each node in the task to be collected from the bidding sources and maintaining node heartbeats; the master node / load component unit is mainly used for electing the master node based on Redis distributed locks / leases and node capability scores; after the master node is determined, the task to be collected from the bidding sources is written to the Redis task queue / status queue according to task priority (task priority includes carrying high-value tasks, regular tasks and low-frequency tasks, which is not limited in this embodiment), and the task status of the task to be collected from the bidding sources is recorded synchronously.

[0080] The advantage of this setup is that, after the master node is determined, the master node dynamically allocates the tasks to be collected and adjusts their weights based on the node scores of each candidate node and the current queue length of the tasks to be collected, thereby achieving load balancing.

[0081] In one specific embodiment, the system further includes: a task priority storage unit in the task scheduling layer, which is mainly used for hierarchical storage and dequeue control of the tasks to be collected according to their task priorities before the target bidding source task is obtained from the message queue based on the browser instance. The task priorities include carrying high-value tasks, regular tasks, and low-frequency tasks, which are not limited in this embodiment. A task distribution unit is mainly used to determine the task priority of the bidding source task to be collected. Specifically, it writes the task to be collected into the message queue based on dimensions such as task urgency, update frequency, and site response speed. The subprocess selects tasks from the three queues (high-value tasks, regular tasks, and low-frequency tasks) according to a "weighted round-robin + timeout backoff" strategy to avoid blocking important tasks. Weighted round-robin is a traffic allocation algorithm that combines the weight values ​​of the backend servers (for example, the priority weight ratio of the bidding source task to be collected could be high priority: medium priority: low priority = 5:3:2, which is not limited in this embodiment) to determine the distribution of node requests in the bidding source task. Servers with higher weights receive more node requests for the bidding source task. Timeout backoff is a strategy that delays retrying when a node request for a task to collect bidding source data fails. The retry interval is increased progressively; for example, waiting 1 second after the first failure, 2 seconds after the second failure, and so on (the number of backoff retries for different task priorities is not specifically limited; for example, it could be 3 times for high priority, 2 times for medium priority, and 1 time for low priority; this embodiment does not impose such a limitation). Combining these two approaches can improve the system's load balancing capability and fault tolerance; this embodiment does not impose such a limitation either.

[0082] In one specific embodiment, the process management unit of the process management layer is mainly used to restart or reclaim the collection subprocess according to the expansion / shrinkage signal of the expansion / shrinkage control unit. The subprocess / heartbeat unit is mainly used to report indicators such as subprocess heartbeat, current task identifier, processing time, number of failures and resource consumption to the Redis task queue / status queue while the collection subprocess is executing the target bidding source task. The advantage of this setting is that it allows the master node / monitoring and elasticity unit to judge the process health and expansion / shrinkage decisions of the message queue in real time.

[0083] In a specific embodiment, before obtaining the target tender source task to be collected from the message queue according to the browser instance, the browser management unit of the browser control layer is mainly used to create a browser instance according to the "instance pool / context reuse" strategy: first initialize the browser process, then create an independent browser context and page object, load the necessary Cookie / proxy configuration, and enable the resource interception unit at the same time. It is also used to continuously perform a health check on the browser instance during the process of collecting the tender source task to be collected: when there are situations such as page loading timeout, continuous network errors / hypertext transfer protocol (HTTP) exception codes, browser process crashes, memory occupancy exceeding the preset threshold, or the error rate of the same site continuously rising, it is determined that the browser is abnormal and the error count is accumulated; when the error count reaches the preset threshold, a restart is triggered, and after the restart, the above list page information and target detail page processing processes are continued from the current task checkpoint (re-initialize the context and re-obtain the task if necessary) to ensure the continuous operation of the collection. The resource interception unit is mainly used to filter unnecessary resources such as pictures and fonts in the browser instance, and then take out the tender source task to be collected from the task priority storage unit and execute it. The advantage of such a setting is to reduce resource consumption and improve system stability.

[0084] In a specific embodiment, after obtaining the list page information, the list page processing unit of the page processing layer is mainly used to first perform DOM cleaning (removing scripts / styles / invalid nodes) on the list page information, then perform structure statistics, traverse the node hierarchy, and count indicators such as tag type, number of child nodes, repetition pattern, text density, and link density, calculate the list score, and generate an initial path expression (candidate extensible markup language path language (XPath)) based on the list score. Subsequently, sample extraction verification is performed on the initial path expression (such as extracting the first N list items and verifying the number range, whether it contains fields such as title / link / time, and whether the link is accessible, etc.), where N is a positive integer greater than or equal to 1. After the verification is valid, the detail link is extracted based on the initial path expression, and the paging / loading more recognition unit and time pattern recognition unit are called to recognize the paging element and the stale data truncation condition, and finally the detail page is obtained. Time pattern recognition is mainly used to recognize date / time fields (such as "YYYY-MM-DD", "YYYY year MM month DD day", etc.) in the initial path expression, normalize them into a unified time format, and compare them with a preset time threshold (for example, it can be 30 days, 60 days, etc., and this embodiment does not limit this). If it is detected that the time of the list item in the initial path expression is earlier than the preset time threshold, a "truncation" instruction is sent to the list page processing unit to stop subsequent paging, thus reducing invalid crawling.

[0085] In one specific embodiment, during the process of determining the target detail page, the detail page processing unit of the detail page processing layer is mainly used to concurrently collect detail pages in an asynchronous manner. That is, based on the candidate path expression, multiple asynchronous tasks are created based on the event loop and the concurrency is controlled by semaphores to obtain the final target detail page.

[0086] In one specific embodiment, after determining the target details page, the information extraction unit is mainly used to extract the main text and tables from the target details page using a "structured parsing + rule fallback" method. Then, the extracted main text and tables are parsed into two-dimensional data according to the column structure. After noise reduction and field normalization of the two-dimensional data, the target two-dimensional data is determined and written into a MySQL database. The MySQL database is mainly used to store the target two-dimensional data. The attachment extraction unit is mainly used to identify attachment links / buttons (such as suffixes like ".pdf / .doc / .xls" or download interfaces) in the target details page after determining it, extract the attachment Uniform Resource Locator (URL), filename, and type, call the download and verification process (retry, size / hash verification), and finally save all extracted attachment-related information to the attachment storage unit and establish associated records in the MySQL database. The attachment storage unit is used to store attachment-related information.

[0087] In one specific embodiment, during the entire process of collecting details from the bidding source's page, the monitoring and elasticity unit of the monitoring and elasticity layer is mainly used to continuously collect indicators such as queue length, queue occupancy rate, enqueue rate, dequeue rate, and task waiting time of the message queue. The queue occupancy judgment unit is mainly used to determine whether the message queue is congested based on queue length, queue occupancy rate, enqueue rate, dequeue rate, and task waiting time. When the queue occupancy rate is continuously greater than or equal to the queue expansion threshold, it sends expansion information to the expansion / shrinkage control unit; when the queue occupancy rate is continuously less than the queue shrinkage threshold, it sends a shrinkage signal to the expansion / shrinkage control unit. The advantage of this setup is that it enables elastic adjustment. The expansion / shrinkage control unit is mainly used to expand or shrink the message queue. When it receives a shrinkage signal, it shrinks the message queue; when it receives an expansion signal, it expands the message queue.

[0088] Specifically, the node coordination unit writes the tender source tasks to be collected into a message queue (such as a Redis task queue / status queue) according to priority. Then, the browser management unit of the browser control layer determines the number of browsers. Based on the number of browsers, the browser instance retrieves the target tender source task from all the tender source tasks to be collected in the message queue. Based on the target tender source task, the list page information is determined and passed to the list page processing unit. After processing the obtained list page information, the list page processing unit performs Document Object Model (DOM) cleaning on the list page HTML to obtain a DOM tree. Then, structural statistics are performed on the structural cleaning result, that is, the hierarchical traversal of the DOM tree is performed to count the tag type, depth, number of child nodes, repeating structure, text density and link density of each node in the DOM tree, calculate the "list container score" of each node, and then generate the initial path expression based on the "list container score". After generating the initial path expression, the list page processing unit also calls the time pattern recognition unit to identify the time field of the initial path expression. After normalizing the time field, the list item time is obtained. It is then determined whether the list item time is greater than or equal to a preset time threshold. If the list item time is greater than or equal to the preset time threshold, stale data is truncated from the generated initial path expression, and pagination is stopped. If the list item time is less than the preset time threshold, the pagination / load more recognition unit is called to extract the detail links and identify the pagination elements of the initial path expression. Based on the detail links and pagination elements, the target detail page corresponding to the task to be collected from the bidding source is obtained.

[0089] This invention provides a method for collecting details pages of bidding sources. When a target bidding source task is obtained from a message queue based on a browser instance, list page information is determined according to the target bidding source task. An initial path expression is generated based on the list page information. If the initial path expression is verified to be valid, the target details page is determined based on the initial path expression. This invention addresses the shortcomings of existing technologies where data extraction typically relies on manually pre-configuring path expressions (XPath expressions) to locate target data. Manual XPath configuration is costly to maintain and lacks adaptability to dynamic changes in webpage structure. This invention achieves the extraction of the target details page by adaptively generating the initial path expression, reducing reliance on manual XPath configuration and maintaining a high success rate in collecting the target details page even when the webpage structure changes.

[0090] The following describes the tender source details page acquisition device provided by the present invention. The tender source details page acquisition device described below and the tender source details page acquisition method described above can be referred to in correspondence.

[0091] Figure 3 This is a schematic diagram of the structure of the tender source detail page acquisition device provided by the present invention, with reference to... Figure 3 As shown, the tender source details page acquisition device 300 includes: an information acquisition module 301, an expression generation module 302, and a details page acquisition module 303; wherein, The information collection module 301 is used to determine the list page information based on the target bidding source task to be collected when the target bidding source task is obtained from the message queue according to the browser instance.

[0092] The expression generation module 302 is used to generate an initial path expression based on the list page information.

[0093] The detail page acquisition module 303 is used to determine the target detail page based on the initial path expression, provided that the initial path expression is valid.

[0094] In one example embodiment, the expression generation module 302 is specifically used to: clean the list page information to obtain candidate list page information; and perform structural statistics on the candidate list page information to generate an initial path expression.

[0095] In one example embodiment, the apparatus further includes a re-parse module. The re-parse module is configured to: re-parse the list page information to generate candidate path expressions if the initial path expression is found to be invalid, and verify whether the candidate path expressions are valid; If the candidate path expression is verified to be valid, the target details page of the task to be collected from the bidding source is collected based on the candidate path expression. If the candidate path expression is found to be invalid, continue with the step of re-parsed the list page information.

[0096] In one example embodiment, the apparatus further includes a task writing module. The task writing module is configured to: determine the task priority of the target tender source task before retrieving it from the message queue based on the browser instance; and write the tender source task to be collected into the message queue according to the task priority.

[0097] In one example embodiment, the apparatus further includes an instance number adjustment module. The instance number adjustment module is configured to: obtain the queue occupancy rate and resource utilization rate of the message queue; and adjust the number of browser instances based on the queue occupancy rate and resource utilization rate.

[0098] In one example embodiment, the details page acquisition module 303 is specifically used to: obtain the time field of each list item in the candidate path expression; if the time field is determined to be greater than or equal to a preset time threshold, score the candidate nodes in each list item and determine the scoring result; generate a target path expression based on the scoring result; and determine the target details page based on the target path expression.

[0099] The apparatus of this embodiment can be used to execute the method of any embodiment in the side embodiment of the method for collecting details of bidding sources. Its specific implementation process and technical effects are similar to those in the side embodiment of the method for collecting details of bidding sources. For details, please refer to the detailed description in the side embodiment of the method for collecting details of bidding sources, which will not be repeated here.

[0100] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a method for collecting details of a tender source. This method includes: if a target tender source task to be collected is obtained from a message queue based on a browser instance, determining list page information based on the target tender source task; generating an initial path expression based on the list page information; and, if the initial path expression is verified to be valid, determining the target details page based on the initial path expression.

[0101] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the tender source detail page collection method provided by the above methods. The method includes: when a target tender source task to be collected is obtained from a message queue based on a browser instance, determining list page information based on the target tender source task; generating an initial path expression based on the list page information; and determining the target detail page based on the initial path expression if the initial path expression is verified to be valid.

[0103] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements a method for collecting details pages of bidding sources provided by the methods described above. The method includes: determining list page information based on the target bidding source task when a target bidding source task is obtained from a message queue based on a browser instance; generating an initial path expression based on the list page information; and determining a target details page based on the initial path expression when the initial path expression is verified to be valid.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for collecting details from a tender source's page, characterized in that, include: If a target tender source task to be collected is obtained from the message queue based on the browser instance, the list page information is determined based on the target tender source task to be collected. Generate an initial path expression based on the list page information; If the initial path expression is verified to be valid, the target details page is determined based on the initial path expression.

2. The method for collecting details from a tender source according to claim 1, characterized in that, The step of generating the initial path expression based on the list page information includes: The list page information is cleaned to obtain candidate list page information; The candidate list page information is subjected to structural statistics to generate the initial path expression.

3. The method for collecting details from a tender source according to claim 1, characterized in that, Also includes: If the initial path expression is found to be invalid, the list page information is re-parsed to generate candidate path expressions, and the validity of the candidate path expressions is verified. If the candidate path expression is verified to be valid, the target details page is determined based on the candidate path expression; If the candidate path expression is found to be invalid, the step of re-parses the list page information continues.

4. The method for collecting details of a tender source according to any one of claims 1-3, characterized in that, Before retrieving the target tender source task from the message queue based on the browser instance, the process also includes: Determine the task priority of the task to be collected from the tender sources; The task to be collected from bidding sources is written into the message queue according to the task priority.

5. The method for collecting details from a tender source according to claim 4, characterized in that, Also includes: Obtain the queue occupancy rate and resource utilization rate of the message queue; The number of browser instances is adjusted based on the queue occupancy rate and the resource utilization rate.

6. The method for collecting details from a tender source according to claim 3, characterized in that, Determining the target details page based on the candidate path expression includes: Obtain the time field of each list item in the candidate path expression; If the time field is determined to be greater than or equal to a preset time threshold, the candidate nodes in each of the list items are scored to determine the scoring result. Generate a target path expression based on the scoring results; The target details page is determined based on the target path expression.

7. A device for collecting details of a tender source, characterized in that, include: The information collection module is used to determine the list page information based on the target bidding source task to be collected when the target bidding source task is obtained from the message queue according to the browser instance; The expression generation module is used to generate an initial path expression based on the list page information; The details page acquisition module is used to determine the target details page based on the initial path expression, provided that the initial path expression is valid.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method for collecting the details page of the tender source as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for collecting the details page of the tender source as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for collecting the details page of the tender source as described in any one of claims 1 to 6.

Images