A webpage data collection and dynamic monitoring method, device and equipment

Abstract

This invention discloses a method, apparatus, and device for webpage data acquisition and dynamic monitoring. The method includes responding to a data monitoring task, initializing a simulated browser according to the task cycle corresponding to the data monitoring task, and accessing the webpages on the list to be monitored specified by the data monitoring task; traversing the webpages on the list to be monitored and extracting webpage element data; if the webpage element data is plain text, extracting the text corresponding to the plain text element; if the webpage element data contains image elements, performing optical character recognition on the images corresponding to the image elements to generate image recognition results; using the text and image recognition results, constructing a blacklist and performing fuzzy matching with the list to be compared, and pushing early warning information according to the fuzzy matching results. This method, combined with the simulated browser, performs categorized identification and simulated matching of webpage elements, improving identification accuracy while ensuring monitoring efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, and device for web page data acquisition and dynamic monitoring. Background Technology

[0002] With the advancement of digitalization in import and export trade, dynamic compliance management based on enterprise web page data has become an important part of the core risk control capabilities of import and export enterprises. However, enterprise web page data is usually published in a mixed form of unstructured and semi-structured data, and is relatively difficult to obtain and parse.

[0003] The industry commonly uses browser automation simulation technology to access the target platform, extract table text data through page element selectors, and call the Optical Character Recognition (OCR) engine for text recognition.

[0004] When the aforementioned optical character recognition engine has insufficient accuracy in recognizing Chinese characters or characters containing special characters, it is prone to errors and omissions, leading to failure of subsequent comparisons. Summary of the Invention

[0005] This invention provides a method, apparatus, and device for web page data acquisition and dynamic monitoring, which solves the technical problem that optical character recognition engines have insufficient accuracy in recognizing Chinese or special characters in enterprise web page data, which easily leads to errors and omissions, resulting in subsequent comparison failures.

[0006] The first aspect of this invention provides a method for web page data collection and dynamic monitoring, comprising: In response to the data monitoring task, initialize the simulated browser according to the task cycle corresponding to the data monitoring task, and access the webpage of the list to be monitored specified by the data monitoring task; Iterate through the web pages in the list to be monitored and extract the web page element data; If the webpage element data is a plain text element, then extract the text corresponding to the plain text element; If the webpage element data contains image elements, optical character recognition is performed on the image to be determined corresponding to the image element to generate an image recognition result. Using the text and image recognition results, a blacklist is constructed and fuzzy-matched with the list to be compared, and warning information is pushed according to the fuzzy-matching results.

[0007] Optionally, the step of responding to the data monitoring task by initializing a simulated browser according to the task cycle corresponding to the data monitoring task and accessing the webpages of the list to be monitored specified by the data monitoring task includes: In response to a data monitoring task, extract the task period and webpage URL corresponding to the data monitoring task; According to the stated task cycle, target agents are selected from a preset list of user agents and injected into a simulated browser; Disable the automation features of the simulated browser to complete the initialization of the simulated browser; The simulated browser is invoked to access the webpage corresponding to the webpage URL, which is part of the list of pages to be monitored.

[0008] Optionally, the method further includes: When the simulated browser fails to load the webpage in the monitored list multiple times, or when the number of failures to extract the webpage element data exceeds a preset failure threshold, the proxy IP of the target proxy is switched according to the preset proxy IP table.

[0009] Optionally, the step of traversing the web pages in the list to be monitored and extracting web page element data includes: Once the webpage containing the list to be monitored is successfully loaded, locate and trigger the pagination control within the webpage. When the table main element in each page of the webpage to be monitored is successfully loaded, traverse each element row of the table main element and extract webpage element data from the cell of each element row respectively; If the current page is not the last page, then proceed to the step of locating and triggering the pagination control within the webpage of the list to be monitored, until the current page becomes the last page.

[0010] Optionally, if the webpage element data contains image elements, the step of performing optical character recognition on the image to be determined corresponding to the image element and generating an image recognition result includes: If the webpage element data contains image elements, and the proportion of the image elements is greater than a preset proportion threshold, then the image to be determined corresponding to the image elements is extracted. The image to be determined is preprocessed to obtain a preprocessed image; The preset optical character recognition engine is invoked to identify image strings from the preprocessed image; If the image string is a date field, then regular expression matching is performed according to the preset regularization rules to obtain the regular expression matching result; If the image string is a name field, then determine whether the name field meets at least one preset field condition; If the regular expression matching result is a match failure or the name field does not meet any of the field conditions, then the confidence level of the image string is determined to be lower than a preset threshold, and the process jumps to the step of calling the preset optical character recognition engine to identify the image string from the preprocessed image; If the regular expression matching result is successful and the name field meets all the field conditions, then the image string is cleaned to generate an image recognition result.

[0011] Optionally, it also includes: If the webpage element data contains image elements, and the proportion of the image elements is not greater than a preset proportion threshold, then the text elements within the webpage element data are located. Extract the text corresponding to the text element.

[0012] Optionally, the step of constructing a blacklist using the text and image recognition results and performing a fuzzy match with the list to be compared, and pushing early warning information according to the fuzzy match results, includes: Based on the extraction locations of the text and image recognition results, the table is reconstructed using the text and image recognition results in a preset field format to obtain the list of dishonest persons; Traverse the existing list corresponding to the data monitoring task and filter out fields that meet the preset comparison-free conditions to obtain the comparison list; The list to be compared is compared with the list of dishonest persons to be generated by fuzzy matching. Create and push an alert message for each matching field in the fuzzy matching results.

[0013] Optionally, the step of performing fuzzy matching between the list to be compared and the list of dishonest persons to generate fuzzy matching results includes: Iterate through all the first fields in the list to be compared, and calculate the field similarity between each field and each second field in the list of dishonest persons; If the similarity of the fields is greater than the preset similarity threshold, then the first field is determined as the matching field and added to the fuzzy matching result; If the similarity of the fields is not greater than the preset similarity threshold, the matching is deemed to have failed.

[0014] A second aspect of the present invention provides a web page data acquisition and dynamic monitoring device, comprising: The webpage access initialization module is used to respond to data monitoring tasks, initialize a simulated browser according to the task cycle corresponding to the data monitoring task, and access the webpages of the list to be monitored specified by the data monitoring task. The data extraction module is used to traverse the web pages of the list to be monitored and extract web page element data; The text extraction module is used to extract the text corresponding to the plain text element if the webpage element data is a plain text element; The image recognition module is used to perform optical character recognition on the image corresponding to the image element if the web page element data contains an image element, and generate an image recognition result. The fuzzy matching and early warning module is used to construct a blacklist of untrustworthy individuals using the text and image recognition results, perform fuzzy matching with the list to be compared, and push early warning information according to the fuzzy matching results.

[0015] A third aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the web page data acquisition and dynamic monitoring method as described in any one of the first aspects of the present invention.

[0016] As can be seen from the above technical solutions, the present invention has the following advantages: This invention responds to data monitoring tasks by initializing a simulated browser according to the task cycle corresponding to the data monitoring task and accessing the web pages on the list to be monitored specified by the data monitoring task; it traverses the web pages on the list to be monitored and extracts web page element data; if the web page element data is plain text, it extracts the text corresponding to the plain text element; if the web page element data contains image elements, it performs optical character recognition on the images corresponding to the image elements to generate image recognition results; using the text and image recognition results, it constructs a blacklist and performs fuzzy matching with the list to be compared, and pushes early warning information according to the fuzzy matching results. Thus, by combining the simulated browser to perform categorized identification and simulated matching of web page elements, it improves identification accuracy while ensuring monitoring efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the steps of a webpage data collection and dynamic monitoring method provided in this embodiment of the invention; Figure 2 This is a structural block diagram of a web page data acquisition and dynamic monitoring device provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] This invention provides a method, apparatus, and device for web page data acquisition and dynamic monitoring, which addresses the technical problem that optical character recognition engines have insufficient accuracy in recognizing Chinese characters or special characters in enterprise web page data, easily leading to errors and omissions, and causing subsequent comparison failures.

[0020] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of a webpage data collection and dynamic monitoring method provided in an embodiment of the present invention.

[0022] This invention provides a method for web page data collection and dynamic monitoring, comprising: Step 101: Respond to the data monitoring task, initialize the simulated browser according to the task cycle corresponding to the data monitoring task, and access the webpage of the list to be monitored specified by the data monitoring task. A data monitoring task refers to a predefined unit of work that is executed once or periodically, including the task triggering cycle (such as the 1st of each month or Monday of each week) and the web address that needs to be accessed.

[0023] A simulated browser refers to a browser instance that is controlled by a program and has full rendering capabilities, such as DrissionPage or Selenium-driven browsers based on the Chromium kernel, which can execute JavaScript, load CSS, and simulate real user operations.

[0024] The webpage to be monitored refers to the target website page that publishes credit information disclosure pages, which may include, but is not limited to, a list of dishonest enterprises, unified social credit codes, and AEO (Authorized Economic Operator) ratings.

[0025] In this embodiment, upon receiving a data monitoring task, the task cycle parameters are first parsed. When the system clock matches this cycle, a simulated browser instance is automatically launched. During initialization, anti-spoofing configurations are loaded, such as hiding automation traces. Subsequently, the browser sends an HTTP request to the webpage of the list to be monitored specified by the task and waits for the page to fully load. By binding the data collection behavior to the business cycle, enterprises can periodically obtain the latest blacklist without manual intervention, meeting the timeliness requirements of AEO certification for dynamic compliance review.

[0026] Furthermore, the task cycle can be dynamically adjusted based on the target webpage's update frequency history. For example, if continuous monitoring reveals that a website only updates once a month, the daily data collection will be automatically reduced to monthly collection to minimize invalid access. When multiple data monitoring tasks exist simultaneously, each task is assigned an independent simulated browser instance and network channel to prevent anti-scraping blocking of one task from affecting other tasks.

[0027] In one example of this application, step 101 may include the following sub-steps: In response to data monitoring tasks, extract the corresponding task period and web page URL; According to the task cycle, target proxies are selected from a preset list of user agents and injected into a simulated browser; Disable the automated features of the emulator browser to complete the initialization of the emulator browser; Call a simulated browser to access the webpage corresponding to the monitored list at the specified URL.

[0028] A user agent list refers to a collection of multiple browser identity profiles, each recording information such as the user agent (User-Agent), screen dimensions, and language. A user agent is a term used in computer and network communication to refer to client software or programs (such as browsers, web crawlers, and instant messaging clients) that represent a user in initiating requests and accessing network resources. In the HTTP protocol, the User-Agent request header field identifies the client type, version, operating system, and other information that issued the request.

[0029] Automation features refer to detectable attributes that a browser exposes to web page scripts when it is under program control. For example, navigator.webdriver is a boolean flag that indicates whether the browser is driven by automation software.

[0030] A simulated browser refers to a browser instance that is software-driven and capable of fully rendering web pages, such as a browser based on the Chromium kernel.

[0031] In this embodiment, upon receiving the input data monitoring task, the task period and webpage URL are parsed. Based on the task period (e.g., the 1st of each month), a target agent is selected from a pre-stored list of user agents in a predetermined order, such as through polling. The target agent's language, screen resolution, and other parameters are injected into the simulated browser instance via a startup command. Next, during the browser startup phase, a script is executed to force automated features such as navigator.webdriver to false values, while simultaneously completing other potentially exposed interfaces and initialization. Finally, the simulated browser is driven to access the specified list of webpages to be monitored. By binding the period to the agent, a different browser identity is used for each data collection, and with the masking of automated features, a stable data acquisition channel is maintained over the long term.

[0032] It should be noted that the task cycle can be set using APScheduler or a system Cron task, and the simulated browser instance can be implemented using a DrissionPage browser object. Furthermore, after disabling automation features, a test page can be automatically loaded to check if any traces of automation remain in the sent request headers. If an anomaly is detected, the disabling process is re-executed, forming a closed-loop verification.

[0033] In another example of this application, step 101 may also include the following sub-steps: When the simulated browser fails to load the webpage in the monitored list multiple times, or when the number of failures to extract webpage element data exceeds the preset failure threshold, the proxy IP of the target proxy is switched according to the preset proxy IP table.

[0034] In this embodiment, when a simulated browser accesses a webpage in the monitored list, if multiple consecutive requests fail to receive a valid response (e.g., TCP connection timeout or server error status code), the system accumulates a loading failure count. Similarly, if the webpage loads successfully but fails to locate the expected table or pagination control multiple times during the extraction of webpage element data due to page structure changes, network fluctuations, or other reasons, the system also accumulates an extraction failure count. An internal failure counter is set, incrementing each time any of the above types of failures occur. When the counter reaches a preset failure threshold, the system automatically selects a new proxy IP from a preset proxy IP table, either sequentially or randomly, and dynamically changes the network exit address of the current simulated browser via browser startup parameters or the Developer Tools Protocol (CDP). Simultaneously, the failure counter is reset to zero, and the system re-attempts to access the webpage. If failures continue after the switch, the system continues to rotate through the next entry in the proxy IP table. This enables the data collection process to self-repair when faced with temporary blocking of the target website or network instability, allowing it to change identities and continue performing tasks without manual intervention.

[0035] In addition, different thresholds can be set for loading failures and extraction failures. For example, the proxy can be switched after three consecutive loading failures, while it can be switched after five consecutive extraction failures. This is because extraction failures are more likely due to page redesigns than IP blocking, thus reducing unnecessary proxy changes.

[0036] Step 102: Traverse the web pages in the list to be monitored and extract the web page element data; Web page element data refers to the information extracted from the DOM (Document Object Model) nodes of a web page, including the element's tag name, attributes, text content, and nested child element structure.

[0037] In this embodiment, after the webpage containing the list to be monitored is loaded, the system locates the container (such as a table) on the page. Then, it proceeds layer by layer in pagination, row, and cell order, caching the DOM node and its attributes corresponding to each cell in memory. During traversal, it synchronously checks whether the last page has been reached and relocates the table container after pagination. This process simulates manual page-by-page viewing, but is automated by the program, ensuring the comprehensiveness and repeatability of the data collection.

[0038] It should be noted that after clicking on the pagination, the page is ready by monitoring changes in the number of table rows or the appearance of a certain indicator element, thus shortening the waiting time.

[0039] In one example of this application, step 102 may include the following sub-steps: Once the webpage containing the list to be monitored loads successfully, locate and trigger the pagination control within the webpage. When the main table element in each page of the webpage to be monitored loads successfully, iterate through each row of the main table element and extract the webpage element data from the cell of each row. If the current page is not the last page, then proceed to the step of locating and triggering the pagination control within the webpage containing the list to be monitored, until the current page becomes the last page.

[0040] Pagination controls are interactive components on a webpage used to switch between pages of a data list, typically appearing as a "next page" button, a page number link, or a drop-down menu.

[0041] The main element of a table refers to the HTML container that carries the data of the list of dishonest enterprises; it is generally a `` tag or its parent element.

[0042]

[0043]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] Figure 2 Figure 2

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113]

[0114]

[0115]

[0116] Figure 3 Figure 3 Figure 3

[0117]

[0118] Tags, internally organized by row. An element row refers to each row in the main table element, corresponding to one piece of credit data, typically using... Labels indicate that a cell is the basic unit of data within a row of elements, and can be represented by labels. or The label indicates that it contains specific field values ​​(such as company name, listing date).

[0044] In this embodiment, after the webpage of the monitoring list loads successfully, the system first locates the pagination control (e.g., a button with the attribute class="next") using XPath or CSS selectors, and simulates a mouse click to trigger the loading of the first page. It then waits for the table or other key elements to appear. Once the main table element appears on the current page, the system iterates through each row under that main element. Then, for all cells in each row ( Extract webpage element data sequentially, including but not limited to DOM node types and internal content.

[0045] After extracting all rows from the current page, the system checks if the pagination control is still operable, such as whether the "Next Page" button is not grayed out. If it is not the last page, the pagination control is triggered again, and the system waits for the table body of the new page to load, repeating the above traversal action until it detects that the pagination control is disabled or the current page is the last page. By processing each page sequentially, browser memory overflow caused by loading a large amount of data at once is avoided. Extraction is only performed after verifying that the table body of each page has loaded successfully, preventing data from being missed due to network latency.

[0046] Step 103: If the webpage element data is a plain text element, then extract the text corresponding to the plain text element; Plain text elements are DOM nodes whose content consists entirely of text characters and do not contain images, canvases, or rich media tags, such as a regular text element. The interior contains only text and nothing else. Sub-tags.

[0047] Text refers to readable strings obtained directly from plain text elements, such as "Guangzhou Supply Chain Management Co., Ltd." or "2026-05-14".

[0048] In this embodiment, for each cell's webpage element data within a pagination, the system first determines whether it contains image tags. If no image tags are found... , <canvas>or <svg>If a node is identified as a plain text element, its text is extracted directly by calling DOM properties (such as innerText or textContent). Since no network requests or image processing are required, this operation can be completed in milliseconds, avoiding the uncertainties introduced by subsequent OCR.

[0049] It should be noted that for some web pages, a line of text is split into multiple adjacent nodes (e.g., In cases where all text nodes are recursively collected and concatenated according to DOM order, complete information can be obtained. Simultaneously, the extracted text can be checked for HTML entities (such as '&'), automatically decoding and restoring the original characters to avoid matching errors caused by escaping.

[0050] Step 104: If the webpage element data contains image elements, perform optical character recognition on the image to be identified corresponding to the image element and generate the image recognition result. Image elements refer to elements on a webpage. The tag, whose src attribute points to an image, typically contains text rendered graphically (such as company name, penalty date).

[0051] "Undetermined image" refers to the raw image data downloaded from image elements into memory, which has not yet undergone any processing.

[0052] Image recognition results refer to the strings that the OCR engine recognizes and outputs from the image, which may contain text information such as company name and date.

[0053] In this embodiment, when the webpage element data of a cell contains an image element, the system first locates the image to be identified, obtains its binary data through the browser interface, and stores it in memory. The image is then preprocessed, such as converted to grayscale, binarized, and denoised. Finally, the processed image is sent to the OCR engine for recognition, outputting a string as the image recognition result. Since the company names and dates on the customs website are presented as images, and text cannot be directly extracted, this step solves the problem of poor machine readability, allowing key information originally locked in the image to be transformed into structured data.

[0054] In addition, for the same image, it is recognized once each at the original size, magnified by 1.2 times, and reduced by 0.8 times. The longest common substring in the recognition results is taken as the final output, which can eliminate the recognition error caused by the font being too small or too large.

[0055] In one example of this application, step 104 may include the following sub-steps: If the webpage element data contains image elements, and the proportion of image elements is greater than a preset proportion threshold, then the corresponding undetermined image will be extracted. The image to be processed is preprocessed to obtain a preprocessed image; The preset optical character recognition engine is invoked to identify image strings from the pre-processed image; If the image string is a date field, then regular expression matching will be performed according to the preset regularization rules to obtain the regular expression matching result; If the image string is a name field, then determine whether the name field meets at least one preset field condition; If the regular expression matching result is a failure or the name field does not meet any field condition, the confidence of the image string is determined to be lower than the preset threshold, and the process jumps to the step of calling the preset optical character recognition engine to identify the image string from the preprocessed image. If the regular expression matching result is successful and the name field meets all field conditions, then the image string is cleaned to generate the image recognition result.

[0056] The percentage threshold refers to a percentage limit of the area of ​​the image region relative to the total area of ​​the cell, used to determine whether the cell uses an image as the primary information carrier.

[0057] An image to be determined refers to the raw binary data of an image downloaded to the local machine from image elements, which has not yet undergone any transformation.

[0058] Optical Character Recognition (OCR) refers to a text recognition model based on deep neural networks. This solution uses the ddddocr library, which can map text regions in an image into a string output.

[0059] Image strings refer to the text content that the OCR engine recognizes from a pre-processed image, such as "Guangzhou XX Import & Export Co., Ltd." or "2026-05-14".

[0060] Regularization rules refer to preset string patterns for date fields, such as "four digits + separator + one or two digits + separator + one or two digits", where the separator can be a hyphen, a forward slash, or Chinese year and month characters.

[0061] Field conditions refer to the pre-defined reasonableness checks for the name field, such as Chinese characters accounting for no less than 70%, the string length being close to the number of characters estimated from the image width, and not containing consecutively repeated abnormal symbols.

[0062] Confidence level refers to the degree of trustworthiness of the system in the correctness of the OCR recognition result. In this solution, it is indirectly determined by the results of regular expression matching or field condition validation. If the validation fails, it is considered as low confidence level.

[0063] In this embodiment, when the system detects an image element in a cell, for example, by locating the image element using DrissionPage, it first calculates the area ratio of the image in the cell. If the ratio exceeds a preset threshold, such as 10%, it indicates that the cell mainly relies on the image to convey information. In this case, the binary data of the image is extracted as a candidate image by simulating a browser interface. For example, the image content can be obtained and saved to memory by calling the .src() method.

[0064] If the proportion does not exceed the threshold, it indicates that the image element is only a decoration or accessory to the cell and is not used to convey information. In this case, text extraction should still be the standard.

[0065] After extracting the image to be determined, preprocessing is performed on it, namely, converting the image to grayscale, adaptive thresholding (such as cv2.adaptiveThreshold), and median filtering for noise reduction, to obtain a preprocessed image with clear black and white contrast.

[0066] Subsequently, a preset optical character recognition engine, such as the ddddocr engine, is invoked to recognize the preprocessed image and output an image string. The ocr_type='general' setting allows for general recognition of Chinese, English, and numbers.

[0067] After recognizing the image string, the system determines the field type corresponding to the string: if it belongs to a date field, its format is validated using regularization rules, such as \d{4}-\d{1,2}-\d{1,2}. If it belongs to a name field, it checks field conditions such as the proportion of Chinese characters and the degree of garbled characters. If the regular expression match fails or any field condition is not met, the system considers the confidence level to be below the threshold and automatically jumps back to the previous step, that is, it calls the OCR engine again for recognition. The preprocessing parameters of the OCR engine can be fine-tuned, such as changing the size of the binarization block, and it will retry a maximum of two times.

[0068] Only when the regular expression match is successful and all field conditions are met does the data cleaning stage begin. This stage standardizes the strings to a standard format, removing completely duplicate records (i.e., records with identical company names and unified social credit codes), standardizing dates to YYYY-MM-DD, and converting full-width characters in company names to half-width characters. Finally, the image recognition result is generated. This process, through a combination of post-validation and retries, effectively filters out low-quality recognition results, ensuring high availability of the output data.

[0069] It should be noted that after obtaining the image recognition results, they can be stored in the local MySQL database table `customs_blacklist`. The proportion threshold can be dynamically adjusted based on the historical statistics of the target website. For example, if the image area ratio in a cell is insufficient but still constitutes the main content (with adjacent text empty) multiple times consecutively, the system automatically lowers the threshold parameter for that website to avoid missed sampling. For the enterprise name field, the first six digits (administrative division code) of the successfully recognized unified social credit code in the same row are used as a reference to check whether the name contains the corresponding regional term. If there is no match, a retry is triggered or it is marked for manual review, forming a field-level logical constraint.

[0070] In another example of this application, step 104 may also include the following sub-steps: If the webpage element data contains image elements, and the proportion of the image elements is not greater than the preset proportion threshold, then locate the text elements within the webpage element data. Extract the text corresponding to the text element.

[0071] In this embodiment, when the system detects an image element within a cell, but the calculated image area does not exceed a preset threshold proportion of the cell's total area, it indicates that the image is likely just a decorative icon (such as a padlock or arrow) rather than text carrying key information. In this case, the system abandons OCR processing of the image and instead uses DOM positioning to search for text elements within the current cell. Specifically, the system uses XPath expressions (e.g., `. / text()` or `. / span`) to obtain all text nodes within the cell, selects the longest non-empty one as the valid text, and directly extracts its `textContent` attribute value. This branch avoids unnecessary image downloads and recognition calculations while ensuring the complete acquisition of visible text within the cell, making it suitable for common scenarios in websites where "icons + text" are mixed.

[0072] Furthermore, if no text element can be directly located within the current cell, the system automatically searches upwards or towards adjacent sibling nodes, for example, from... < / svg> < / canvas> Rise to The system searches for text descriptions in other columns to address situations where some web pages place text in the alt or title attributes of images. Step 105 uses the text and image recognition results to construct a blacklist and perform a fuzzy match with the comparison list, pushing alerts based on the fuzzy match results. The blacklist refers to a collection of records assembled from data from the two sources according to the field order of the original table (e.g., company name, credit code, inclusion date), with each record corresponding to a blacklisted company. The comparison list refers to a list of business partners maintained locally by the company, containing multiple first fields, which can be the partner name and its customs certification level. Fuzzy matching refers to using a string similarity algorithm (such as the token_set_ratio function in the fuzzywuzzy library) to calculate the degree of matching between two names, outputting a score between 0 and 100, with higher scores indicating greater similarity. An alert is a notification generated by the system when a field in the comparison list successfully matches a company in the blacklist, including the partner name, the name of the matched blacklisted company, the inclusion date, and the blacklisted behavior. In this embodiment, after obtaining the text and image recognition results, they are aligned according to the column order of the page table and pieced together to form a complete list of companies with poor credit records. Then, the internal comparison list of the company is read, for example, exported from the supplier database. For each first field in the comparison list, it is first checked whether it meets the comparison exemption conditions, such as AEO advanced certification. If not, its first field is fuzzily matched with the names of all companies in the list of companies with poor credit records one by one. When the similarity exceeds a preset threshold, it is determined that the partner has entered the list of companies with poor credit records, and an early warning message is immediately generated. All early warning messages are summarized and pushed to compliance personnel via WeChat or email. This step directly associates the collected raw data with the company's actual business objects, and outputs risk warnings that can be directly used for decision-making, rather than a list that requires secondary manual comparison, forming a closed loop from data collection to risk warning. In addition, based on string similarity, a thesaurus can be introduced, such as treating "Limited Company" and "Limited Liability Company" as equivalent, and then matching the normalized names again to reduce the underreporting caused by differences in the terms used in company registration types. In one example of this application, step 105 may include the following sub-steps S11-S14: S11, according to the extraction positions of the text and image recognition results, reconstruct the table using the text and image recognition results in a preset field format to obtain the list of dishonest entities; the extraction position refers to the row and column coordinates of the text or image recognition result in the original webpage table, such as the 3rd row and 2nd column, used to determine which field in the list of dishonest entities the data corresponds to. The field format refers to a predefined data structure of the list of dishonest entities, such as a fixed structure containing four fields: "Company Name", "Unified Social Credit Code", "Inclusion Date", and "Dishonest Behavior" and their order. In this embodiment, after obtaining the text and image recognition results, they are aligned according to their original row and column positions in the webpage table.For example, if the text in column 2 corresponds to "Company Name" and the image recognition result in column 4 corresponds to "Inclusion Date", the system will fill the data from different positions in the same row into the corresponding field columns of the blacklist according to the preset field format, and then stitch them together row by row to form a complete structured blacklist. Through location-based reconstruction, even if the column order of the webpage table is adjusted, as long as the column mapping relationship is configured in advance, the system can still correctly assemble the data, ensuring the consistency of the fields in the collection results. S12, Traverse the existing list corresponding to the data monitoring task and filter out fields that meet the preset exemption conditions to obtain the list to be compared; the existing list refers to the business partner dataset maintained locally by the enterprise, usually from the ERP system or supplier management module. Each record contains attributes such as partner name, unified social credit code, and AEO level. The exemption conditions refer to the filtering rules set according to the business rules. For example, partners with AEO levels of "Advanced Certified Enterprise" or "General Certified Enterprise" do not need to participate in the blacklist comparison and are directly regarded as compliant. In this embodiment, the existing list file or database table configured in the data monitoring task is read, and each partner record is traversed one by one. For each record, its AEO level field is extracted and compared with a preset list of exemption criteria. If the level matches any value in the list (such as "Advanced Certification"), the record is removed from the subsequent comparison process; otherwise, it is retained in the list to be compared. The removal operation skips the partner directly and does not enter the fuzzy matching stage. S13: Fuzzy matching is performed between the list to be compared and the list of dishonest entities to generate fuzzy matching results; further, S13 may include the following sub-steps: traverse all first fields in the list to be compared and calculate the field similarity between each second field in the list of dishonest entities; if the field similarity is greater than a preset similarity threshold, the first field is determined as the matching field and added to the fuzzy matching result; if the field similarity is not greater than the preset similarity threshold, the matching is determined to be unsuccessful. The first field refers to the enterprise name field of each business partner record in the list to be compared. The second field refers to the enterprise name field of each dishonest enterprise record in the list of dishonest entities. Field similarity refers to the degree of matching between two names calculated using a string comparison algorithm, typically represented by a score between 0 and 1. A higher score indicates a greater likelihood that the two names refer to the same entity. The similarity threshold is a pre-defined score limit; only when the field similarity exceeds this limit are the two names considered a successful match. The matching field refers to the first field identified as corresponding to a certain second field after comparison. It is a paired record in the fuzzy matching result and includes associated information such as the partner name, the name of the matched dishonest enterprise, and the listing date. The fuzzy matching result refers to the set of all successfully matched pairs (first field, second field).In this embodiment, the system iterates through each first field in the list to be compared, such as the name of each business partner of the enterprise's local partners, the unified social credit code, and the AEO rating. For the current first field, it sequentially calculates the field similarity with each second field in the blacklist, such as the name of the blacklisted party and the unified social credit code. This calculation can use a bag-of-words-based Jaccard similarity algorithm or an edit distance algorithm, for example, splitting the two names into character sets and calculating the intersection and union ratio. When the calculated similarity is greater than a preset similarity threshold, it is determined that the first field and the current second field are successfully matched, and the first field and its corresponding second field are added to the fuzzy matching result. If the similarity does not exceed the threshold, the system continues to compare the next second field until the blacklist is completely traversed. If the similarity with all second fields does not exceed the threshold, the first field is determined to be a failed match and is no longer recorded. By setting a reasonable similarity threshold, the system can automatically identify inconsistencies such as "A City XX Trading Co., Ltd." and "AXX Trading Co., Ltd." caused by differences in administrative division terminology, thus avoiding missed reports while controlling the false report rate. It should be noted that if a partner's first field matches multiple names of dishonest enterprises simultaneously, multiple matching pairs are recorded separately. Furthermore, for the similarity threshold setting, two thresholds can be set (e.g., a high threshold of 0.9 and a low threshold of 0.7). Similarity scores higher than the high threshold are directly considered a match; those between the high and low thresholds are further judged using additional business rules (such as whether the unified social credit codes are the same); similarity scores lower than the low threshold are directly judged as a mismatch. This hierarchical structure reduces false judgments in critical situations. S14. Create and push an early warning message for each matching field in the fuzzy matching results. In this embodiment, each matching field in the simulated matching results is traversed, extracting the partner name, the matched name of the dishonest enterprise, the inclusion date, and the description of the dishonest behavior associated with the list of dishonest entities. This information is then assembled into a complete early warning message according to a preset early warning template. After all early warning messages are summarized, they are pushed to the designated receiving end of the compliance department via a WeChat Work robot webhook or SMTP email. Simultaneously, the system writes each alert message to the alert history table `alert_history` in the local database, recording the push time and content hash, facilitating subsequent auditing and preventing duplicate pushes. This step transforms the matching results into directly readable risk alerts, achieving a closed-loop business process from data collection to proactive alerting. It should be noted that the alert history table can also be queried before pushing an alert. If the same partner has already pushed the same alert to the same defaulting company within the past 30 days, no further alert will be pushed this time; only the "repetition count" field of the existing record will be updated, avoiding information overload. It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows.Unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Furthermore, at least some steps in the flowcharts involved in the above embodiments may include multiple steps or stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. The webpage data acquisition and dynamic monitoring device provided in the embodiments of this application is described below. The webpage data acquisition and dynamic monitoring device described below can be referred to in correspondence with the webpage data acquisition and dynamic monitoring method described above. Please refer to the diagram below, which illustrates a structural block diagram of a webpage data acquisition and dynamic monitoring device according to an embodiment of the present invention. The device includes: a webpage access initialization module 201, used to respond to a data monitoring task, initialize a simulated browser according to the task cycle corresponding to the data monitoring task, and access the webpages in the monitoring list specified by the data monitoring task; a data extraction module 202, used to traverse the webpages in the monitoring list and extract webpage element data; a text extraction module 203, used to extract the text corresponding to the plain text elements if the webpage element data is plain text; an image recognition module 204, used to perform optical character recognition on the images corresponding to the image elements if the webpage element data contains image elements, and generate image recognition results; and a fuzzy matching and early warning module 205, used to construct a blacklist using the text and image recognition results and perform fuzzy matching with the comparison list, and push early warning information according to the fuzzy matching results. Optionally, the webpage access initialization module 201 is specifically used for: responding to data monitoring tasks, extracting the task period and webpage URL corresponding to the data monitoring task; selecting a target agent from a preset user agent list according to the task period and injecting it into a simulated browser; disabling the automation features of the simulated browser to complete the initialization of the simulated browser; and calling the simulated browser to access the webpage to be monitored corresponding to the webpage URL. Optionally, the webpage access initialization module 201 is also specifically used for: when the simulated browser fails to load the webpage to be monitored multiple times, or when the number of failures to extract webpage element data exceeds a preset failure threshold, switching the proxy IP of the target agent according to a preset proxy IP table. Optionally, the data extraction module 202 is specifically used for: after the webpage to be monitored is successfully loaded, locating and triggering the pagination control within the webpage to be monitored; when the table body element within each page of the webpage to be monitored is successfully loaded, traversing each element row of the table body element and extracting webpage element data from the cells of each element row; if the current page is not the last page, then jumping to execute the step of locating and triggering the pagination control within the webpage to be monitored until the current page is the last page.Optionally, the image recognition module 204 is specifically used for: if the webpage element data contains image elements and the proportion of image elements is greater than a preset proportion threshold, then extracting the image to be determined corresponding to the image element; preprocessing the image to be determined to obtain a preprocessed image; calling a preset optical character recognition engine to recognize the image string from the preprocessed image; if the image string is a date field, then performing regular expression matching according to preset regularization rules to obtain the regular expression matching result; if the image string is a name field, then determining whether the name field meets at least one preset field condition; if the regular expression matching result is a matching failure or the name field does not meet any field condition, then determining that the confidence of the image string is lower than a preset threshold, and jumping to execute the step of calling the preset optical character recognition engine to recognize the image string from the preprocessed image; if the regular expression matching result is a successful match and the name field meets all field conditions, then performing data cleaning on the image string to generate an image recognition result. Optionally, the image recognition module 204 is also specifically used for: if the webpage element data contains image elements and the proportion of image elements is not greater than a preset proportion threshold, then locating the text elements within the webpage element data; extracting the text corresponding to the text elements. Optionally, the fuzzy matching and early warning module 205 includes: a blacklist reconstruction submodule, used to reconstruct a table using the extracted positions of text and image recognition results in a preset field format to obtain a blacklist; a list initial screening submodule, used to traverse the existing lists corresponding to the data monitoring task and filter out fields that meet preset comparison exemption conditions to obtain a list to be compared; a fuzzy matching submodule, used to perform fuzzy matching between the list to be compared and the blacklist to generate fuzzy matching results; and an early warning push submodule, used to create and push an early warning message for each matching field in the fuzzy matching result. Optionally, the fuzzy matching submodule is specifically used to: traverse all first fields in the list to be compared and calculate the field similarity between each second field in the blacklist; if the field similarity is greater than a preset similarity threshold, the first field is determined as a matching field and added to the fuzzy matching result; if the field similarity is not greater than the preset similarity threshold, the matching is determined to be unsuccessful. This invention provides a computer device including a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the web page data acquisition and dynamic monitoring method as described in any embodiment of this invention. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described devices and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways.For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or modules may be electrical, mechanical, or other forms. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Additionally, the functional modules in the various embodiments of this invention can be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules. If the integrated modules are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Schematic, as shown, is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. This computer device 300 can be provided as a server. Referring to this, the computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by a memory 301 for storing instructions executable by the processing component 302, such as application programs. The application programs stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the text recognition method of any of the above embodiments. The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305.Computer device 300 can operate based on storage. The above description and embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for webpage data collection and dynamic monitoring, characterized in that, include: In response to the data monitoring task, initialize the simulated browser according to the task cycle corresponding to the data monitoring task, and access the webpage of the list to be monitored specified by the data monitoring task; Iterate through the web pages in the list to be monitored and extract the web page element data; If the webpage element data is a plain text element, then extract the text corresponding to the plain text element; If the webpage element data contains image elements, optical character recognition is performed on the image to be determined corresponding to the image element to generate an image recognition result. Using the text and image recognition results, a blacklist is constructed and fuzzy-matched with the list to be compared, and warning information is pushed according to the fuzzy-matching results.

2. The webpage data acquisition and dynamic monitoring method according to claim 1, characterized in that, The steps of initializing a simulated browser according to the task cycle corresponding to the data monitoring task and accessing the webpages of the list to be monitored specified by the data monitoring task include: In response to a data monitoring task, extract the task period and webpage URL corresponding to the data monitoring task; According to the stated task cycle, target agents are selected from a preset list of user agents and injected into a simulated browser; Disable the automation features of the simulated browser to complete the initialization of the simulated browser; The simulated browser is invoked to access the webpage corresponding to the webpage URL, which is part of the list of pages to be monitored.

3. The webpage data acquisition and dynamic monitoring method according to claim 1 or 2, characterized in that, The method further includes: When the simulated browser fails to load the webpage in the monitored list multiple times, or when the number of failures to extract the webpage element data exceeds a preset failure threshold, the proxy IP of the target proxy is switched according to the preset proxy IP table.

4. The webpage data acquisition and dynamic monitoring method according to claim 1, characterized in that, The step of traversing the web pages in the list to be monitored and extracting web page element data includes: Once the webpage containing the list to be monitored is successfully loaded, locate and trigger the pagination control within the webpage. When the table main element in each page of the webpage to be monitored is successfully loaded, traverse each element row of the table main element and extract webpage element data from the cell of each element row respectively; If the current page is not the last page, then proceed to the step of locating and triggering the pagination control within the webpage of the list to be monitored, until the current page becomes the last page.

5. The webpage data acquisition and dynamic monitoring method according to claim 1, characterized in that, The step of performing optical character recognition on the image corresponding to the webpage element data and generating an image recognition result if the webpage element data contains image elements includes: If the webpage element data contains image elements, and the proportion of the image elements is greater than a preset proportion threshold, then the image to be determined corresponding to the image elements is extracted. The image to be determined is preprocessed to obtain a preprocessed image; The preset optical character recognition engine is invoked to identify image strings from the preprocessed image; If the image string is a date field, then regular expression matching is performed according to the preset regularization rules to obtain the regular expression matching result; If the image string is a name field, then determine whether the name field meets at least one preset field condition; If the regular expression matching result is a match failure or the name field does not meet any of the field conditions, then the confidence level of the image string is determined to be lower than a preset threshold, and the process jumps to the step of calling the preset optical character recognition engine to identify the image string from the preprocessed image; If the regular expression matching result is successful and the name field meets all the field conditions, then the image string is cleaned to generate an image recognition result.

6. The webpage data acquisition and dynamic monitoring method according to claim 5, characterized in that, Also includes: If the webpage element data contains image elements, and the proportion of the image elements is not greater than a preset proportion threshold, then the text elements within the webpage element data are located. Extract the text corresponding to the text element.

7. The webpage data acquisition and dynamic monitoring method according to claim 1, characterized in that, The steps of constructing a blacklist using the text and image recognition results and performing fuzzy matching with the list to be compared, and pushing early warning information according to the fuzzy matching results, include: Based on the extraction locations of the text and image recognition results, the table is reconstructed using the text and image recognition results in a preset field format to obtain the list of dishonest persons; Traverse the existing list corresponding to the data monitoring task and filter out fields that meet the preset comparison-free conditions to obtain the comparison list; The list to be compared is compared with the list of dishonest persons to be generated by fuzzy matching. Create and push an alert message for each matching field in the fuzzy matching results.

8. The webpage data acquisition and dynamic monitoring method according to claim 7, characterized in that, The step of performing fuzzy matching between the list to be compared and the list of dishonest persons to generate fuzzy matching results includes: Iterate through all the first fields in the list to be compared, and calculate the field similarity between each field and each second field in the list of dishonest persons; If the similarity of the fields is greater than the preset similarity threshold, then the first field is determined as the matching field and added to the fuzzy matching result; If the similarity of the fields is not greater than the preset similarity threshold, the matching is deemed to have failed.

9. A webpage data acquisition and dynamic monitoring device, characterized in that, include: The webpage access initialization module is used to respond to data monitoring tasks, initialize a simulated browser according to the task cycle corresponding to the data monitoring task, and access the webpages of the list to be monitored specified by the data monitoring task. The data extraction module is used to traverse the web pages of the list to be monitored and extract web page element data; The text extraction module is used to extract the text corresponding to the plain text element if the webpage element data is a plain text element; The image recognition module is used to perform optical character recognition on the image corresponding to the image element if the web page element data contains an image element, and generate an image recognition result. The fuzzy matching and early warning module is used to construct a blacklist of untrustworthy individuals using the text and image recognition results, perform fuzzy matching with the list to be compared, and push early warning information according to the fuzzy matching results.

10. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the web page data acquisition and dynamic monitoring method as described in any one of claims 1-8.

Images